ENH add CAP curve #28972

JosephBARBIERDARNAL · 2024-05-07T19:43:52Z

This PR is a dupplicate from #28752. I'm recreating a PR because I accidentally deleted my fork with the old one.

Reference Issue

What does this implement/fix?

creation of a CumulativeAccuracyDisplay class for plots

"The CAP of a model represents the cumulative number of positive outcomes along the y-axis versus the corresponding cumulative number of a classifying parameter along the x-axis. The output is called a CAP curve.[1] The CAP is distinct from the receiver operating characteristic (ROC) curve, which plots the true-positive rate against the false-positive rate." (wikipedia definition)

It's mainly inspired from the RocCurveDisplay class.

other

It's currently a work in progress.

TODO

Binary classification

raise a ValueError in from_estimator if the estimator is not fitted or is a classifier that was fitted with more than 3 classes;
fix pos_label handling when the positive class;
add/update a test to check that we have the same result for response_method="decision_function" and response_method="predict_proba" for a LogisticRegression classifier fit with string labels and for all 3 possible values of pos_label;
- not that this is different from what is already tested in test_display_from_estimator_and_from_prediction;
- CAP curves should be invariant under order preserving transformations of the predictions. This is way plotting CAP from the unormalized logits or the logistic sigmoid scaled probabilistic predictions should result in the same curve: the logistic sigmoid is strictly monotonic, hence order preserving.
  - joseph --> should be good
add an option (enabled by default) to draw CAP curve of the "perfect"/"oracle" model;
update the tests to check that the model curves always lie between the "chance level" and "perfect"/"oracle" curves.
add a test to check that the display array attributes y_true_cumulative and cumulative_total have the same dtype as y_pred in the test about from_predictions. We can test for y_pred passed either as np.float32 or np.float64.
- joseph: should be good
test that CAPCurveDisplay.from_estimator(LinearSVC().fit(X, y), ...) works (even if it does not have a predict_proba method. This should cover one of the line reported as uncovered by codecov.
leverage test_common_curve_display.py to reuse some generic tests on CAPCurveDisplay and maybe remove redundant tests on invalid inputs from test_cap_curve_display.py if any;
- joseph: should be good
add despine argument?
- olivier: I would rather not do that specifically for that PR but maybe consider a cross-display PR that does that for all *Display classes in scikit-learn. Feel free to open an issue to discuss this with screen shots e.g. on ROC or PR curves and your analysis of pros and cons.
- joseph: sure. I suggested it here because the ROC and PR curves already have it (see ENH despine keyword for ROC and PR curves #26367). I'm not sure it makes much sense for ConfusionMatrixDisplay (?). I'll open an issue (when this PR will be merged) for CAPCurveDisplay, PredictionErrorDisplay and DetCurveDisplay because I think they're the only ones that don't have this option.
- should be good --> I added the argument, as per suggested below in the discussion

Regression

update the docstrings to make it explicit that either regressors (with positive outcomes) or binary classifiers are accepted and anything else is rejected.
- joseph: I made a first pass, but I guess it can be improved
raise a ValueError with an informative error message if y_true has negative values;
- joseph: it is now tested here
raise a ValueError if all y_true are zeros (the plot would be degenerate and would raise a low level divide by zero warning whith normalize_scale=True);
- In theory, this should not happen, since if all the y_true are zeros, it will be considered a case of classification
add tests with continuous outputs on positive observation data (e.g. using PoissonRegressor) and check that the regressor curve lie between the "chance level" and "perfect" curves;
- joseph: it is now tested here
update the insurance model examples (examples/linear_model/plot_tweedie_regression_insurance_claims.py and examples/linear_model/plot_poisson_regression_non_normal_loss.py) to use the CAPCurveDisplay class instead of manually plotting the Lorenz curves.

Other

document the new feature with a new entry under doc/whats_new/upcoming_changes/
- done here
update the user guide (doc/visualization) to reference this new tool.
- done here.
- joseph: One thing I find strange is that the CAP curve is the only one where all the letters of the acronym are in capitals. Is this explicitly intended? There's DetCurveDisplay, RocCurveDisplay and then CAPCurveDisplay, which seems inconsistent.

Nice to have

add a test to check the sample weight/repetition equivalence property: ENH add CAP curve #28972 (comment)

github-actions · 2024-05-07T19:45:08Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 9cb6124. Link to the linter CI: here}

lorentzenchr · 2024-05-21T06:53:20Z

@JosephBARBIERDARNAL Could you please address all the comments from #28752?

JosephBARBIERDARNAL · 2024-06-25T18:08:37Z

Yes, I'm ready to do it, but I won't be able to get back to it quickly (2 to 3 months) unfortunately. Hope that's okay

JosephBARBIERDARNAL · 2024-10-06T13:53:27Z

Working on this PR again. Here are the current updates:

Focus on classifiers only (as per the decision made, see: FEA add CumulativeAccuracyDisplay #28752 (comment))
Correct most invalid names and complete docstrings
Improved code coverage (though it may still be incomplete)

Some (likely non-exhaustive) issues:

Should the figure be squared when normalize=False? Currently, it does self.ax_.set([...], aspect="equal") in both cases, so the figure dimensions are "random" (i.e., unpredictable)

import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

from sklearn.metrics import CapCurveDisplay

X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
y_scores = clf.decision_function(X_test)

fig, ax = plt.subplots(ncols=2, dpi=300, figsize=(12,12))

display = CapCurveDisplay.from_predictions(
    ax=ax[0],
    y_true=y_test,
    y_pred=y_scores,
    name='normalize_scale=False',
    normalize_scale=False,
    plot_chance_level=True
)

display = CapCurveDisplay.from_predictions(
    ax=ax[1],
    y_true=y_test,
    y_pred=y_scores,
    name='normalize_scale=True',
    normalize_scale=True,
    plot_chance_level=True
)

An error related to both RocCurveDisplay and CapCurveDisplay. I opened a dedicated issue since it's outside the scope of this PR: chance_level_kw in RocCurveDisplay raises an error when using valid matplotlib args #30015 (fixed with FIX handle aliases in displays when used as default and provided by user #30023)

glemaitre · 2024-11-03T10:24:35Z

I'll come back on this PR after the release. This will be one of my priority to be merged for 1.7.

JosephBARBIERDARNAL · 2024-11-03T11:52:09Z

I'll come back on this PR after the release. This will be one of my priority to be merged for 1.7.

Ok cool. There's still a few things I need to do anyway before a review (cov test and adding the #30023 check).

ogrisel · 2024-11-25T17:37:18Z

sklearn/metrics/_plot/cap_curve.py

+
+        # compute cumulative sums for true positives and all cases
+        y_true_cumulative = np.cumsum(y_true_sorted * sample_weight_sorted)
+        cumulative_total = np.cumsum(sample_weight_sorted)


For information, there was a concurrent PR to fix the lack of cumsum of the sample_weight to define the x-axis of the Lorenz curves in one of our regression examples. To check that this was the correct fix, we ran a quick check on synthetic data with integer valued sample weights to check that there is an exact equivalence between repeated data points and reweighting them by exposure:

#30198 (comment)

Maybe this PR could be expanded to test for this property also holds for CapCurveDisplay.from_predictions with non-constant, integer-valued sample_weight.

EDIT: I am not entirely sure how to write such a test, but possibly we could numpy.interp1d the CAP curve computed on the repeated for the x-axis location of the points of the weighted CAP curve and check that the two curves match up to a small eps at those locations.

ogrisel

There are several review comments in #28752 that have not yet been addressed.

I would like to make sure that we do not forget about them when iterating on this PR.

@JosephBARBIERDARNAL to sort things out, please reply in the threads of the review of #28752 and make it explicit which comments have been addressed in #28972 and how, and then mark those threads as resolved.

Also, we need some test and update an existing example, where we compare ROC, DET and CAP curves on the same classifier. I suppose this example is the best candidate:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_det.html

sklearn/metrics/_plot/cap_curve.py

JosephBARBIERDARNAL · 2024-11-27T13:03:12Z

@ogrisel That sounds good to me. I won't be able to work on it immediately, but I’ll definitely be able to get to it within the next few weeks.

I'll ping you and/or @glemaitre for review

lorentzenchr · 2024-11-30T09:40:31Z

2 things are important for me:

Even if this PR focuses on binary classification, it should be implemented in a way such that (positive) regression can be added easily, in particular
- without API hassle (note that it is predict_proba for classification but predict for regression)
- with the same class/function
The plot should contain the lines for "perfect" and "random" as in the screenshot from the paper in FEA add CumulativeAccuracyDisplay #28752 (comment).

JosephBARBIERDARNAL · 2025-01-11T16:42:09Z

The plot should contain the lines for "perfect" and "random" as in the screenshot from the paper in

Will this actually be visible? I'm not entirely sure how broad the possible shape of such a line could be. In my mind, it would resemble the "perfect" line in ROC curves, which suggests that this line would almost always span the spines of the Matplotlib Axes (except perhaps in cases with very low sample sizes?). However, we could adjust the x/y axis limits to accommodate this scenario.
If we decide to implement it, what should we name it, and what should its default style be?

JosephBARBIERDARNAL · 2025-01-11T16:45:36Z

@JosephBARBIERDARNAL to sort things out, please reply in the threads of the review of #28752 and make it explicit which comments have been addressed in #28972 and how, and then mark those threads as resolved.

I resolved most of the conversations there. I didn't touch some of them if I wasn't sure. Feel free to ping me if any changes are needed.

I'm just not sure what "make it explicit which comments have been addressed in https://github.com/scikit-learn/scikit-learn/pull/28972 and how" means

JosephBARBIERDARNAL · 2025-04-28T11:55:38Z

A few questions:

Should we set the same default behavior as in other displays for chance_level (not plotting it). The same goes for plot_perfect.
Should we add gini/AR to the plot labels instead of requesting users to call auc and calculate it themselves?

JosephBARBIERDARNAL · 2025-04-28T13:00:24Z

@ogrisel You can review. See my comments in the TODO.

ogrisel · 2025-04-29T09:25:25Z

Should we set the same default behavior as in other displays for chance_level (not plotting it). The same goes for plot_perfect.

No strong opinion, but I think I would rather update the other to plot the baseline curve by default instead (to be done later).

ogrisel · 2025-04-29T09:26:21Z

Should we add gini/AR to the plot labels instead of requesting users to call auc and calculate it themselves?

I would rather do that in a follow-up PR that introduces an official Gini scoring function instead.

ogrisel

Thanks @JosephBARBIERDARNAL. Here is another batch of feedback.

examples/linear_model/plot_poisson_regression_non_normal_loss.py

sklearn/metrics/_plot/cap_curve.py

sklearn/metrics/_plot/tests/test_cap_curve_display.py

ogrisel · 2025-04-30T14:11:01Z

sklearn/metrics/_plot/cap_curve.py

+            self.cumulative_total = self.cumulative_total / x_max
+            self.y_true_cumulative = self.y_true_cumulative / y_max


We need to be robust to edge-cases.

Suggested change

self.cumulative_total = self.cumulative_total / x_max

self.y_true_cumulative = self.y_true_cumulative / y_max

if x_max != 0:

self.cumulative_total = self.cumulative_total / x_max

if y_max != 0:

self.y_true_cumulative = self.y_true_cumulative / y_max

and please add tests to check that no low level warnings are raised for edge cases like:

CAPCurveClassifier.from_predictions(np.zeros(3), np.asarray([0.1, 0.3, 1.]))

CAPCurveClassifier.from_predictions(np.asarray([0.1, 0.3, 1.]), np.zeros(3))

CAPCurveClassifier.from_predictions(np.zeros(3), np.zeros(3))

Please also test when the classification interpretation forced by setting an explicit pos_label != None:

CAPCurveClassifier.from_predictions( np.zeros(3), np.asarray([0.1, 0.3, 1.]), pos_label=0 )

CAPCurveClassifier.from_predictions( np.zeros(3), np.asarray([0.1, 0.3, 1.]), pos_label=1 )

I am not 100% sure but maybe some of those edge cases should better trigger high level ValueError with an explicit message instead of plotting a meaningless chart.

It's now tested with test_edge_cases_from_predictions()

sklearn/metrics/_plot/cap_curve.py

sklearn/metrics/_plot/tests/test_cap_curve_display.py

examples/model_selection/plot_det.py

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

glemaitre

Just a really quick review. I'll continue tomorrow.

sklearn/metrics/_plot/cap_curve.py

glemaitre · 2025-05-05T16:48:38Z

sklearn/metrics/_plot/cap_curve.py

+            if sample_weight is None:
+                sample_weight = np.ones_like(y_true, dtype=np.float64)
+
+            sorted_indices = np.argsort(y_pred)


I could not see this comment in the GitHub frontend but it seems important to be addressed.

sklearn/metrics/_plot/cap_curve.py

ArturoAmorQ

Thanks for the PR @JosephBARBIERDARNAL. This new feature is definitely very valuable. Here's a batch of comments regarding the modified example.

ArturoAmorQ · 2025-05-07T14:35:12Z

examples/model_selection/plot_det.py

@@ -4,17 +4,22 @@
 ====================================


I don't think this name for the example would be pertinent anymore. Maybe we can go with something like "Compare ROC, DET and CAP curves". Then we would have to rename the section a bit further below.

Sounds good to me! Let me know when this is approved by others.

examples/model_selection/plot_det.py

ArturoAmorQ · 2025-05-07T14:47:56Z

examples/model_selection/plot_det.py

 DET curves are a variation of ROC curves where False Negative Rate (FNR) is
 plotted on the y-axis instead of the TPR. In this case the origin (bottom left
-corner) is the "ideal" point.
+corner) is the "ideal" point. Furthermore, the axes use a normal deviate scale


The use of a normal deviate scale is already mentioned below. But if you really think it's worth keeping here, let's rather mention it before the discussion of the "ideal" point, to have a more consistent flow of ideas.

It's ok for me if we remove it and keep only the mention below.

ArturoAmorQ · 2025-05-07T14:52:02Z

examples/model_selection/plot_det.py

+        ax=ax_roc,
+        name=name,
+        pos_label=pos_label,
+        plot_chance_level=is_last,


As we already have the DummyClassifier, this line is overlapping. I think back in the day we decided that we better avoid the mention to "chance level" if possible

Suggested change

plot_chance_level=is_last,

see my comment

examples/model_selection/plot_det.py

ArturoAmorQ · 2025-05-07T14:57:13Z

examples/model_selection/plot_det.py

+# The diagonal black-dotted lines named "chance level" in the plots above
+# correspond to a the expected value of a non-informative classifier on an
+# infinite evaluation set.


I we go with avoiding the term "chance level" I would rather remove this whole paragraph and only mention in the paragraph about the DummyClassifier below that non-informative classifiers can be used as baseline.

Co-authored-by: Arturo Amor <86408019+ArturoAmorQ@users.noreply.github.com>

JosephBARBIERDARNAL · 2025-05-07T15:43:11Z

A few things that I think need to be addressed:

One thing I find strange is that the CAP curve is the only one where all the letters of the acronym are in capitals. There's DetCurveDisplay, RocCurveDisplay and then CAPCurveDisplay, which seems inconsistent.
I'm not entirely sure I understand what is intended for chance_level naming in scikit-learn in general. I've seen a few threads mentioning avoiding using it or renaming it. If that's the case, wouldn't now be a good time to not use that name in this display?

add cap curve main class and tests

dc058d9

github-actions bot added the module:metrics label May 7, 2024

JosephBARBIERDARNAL mentioned this pull request May 7, 2024

FEA add CumulativeAccuracyDisplay #28752

Closed

JosephBARBIERDARNAL and others added 2 commits May 11, 2024 15:09

linting issue

4e73049

Merge branch 'main' into cap-curve

342de96

lorentzenchr changed the title ~~add cap curve main class and tests~~ ENH add CAP curve May 11, 2024

JosephBARBIERDARNAL added 2 commits June 3, 2024 17:13

Merge branch 'main' into cap-curve

716d812

Merge branch 'main' into cap-curve

d03c666

JosephBARBIERDARNAL and others added 3 commits September 29, 2024 11:29

Merge branch 'main' into cap-curve

3453696

Merge branch 'scikit-learn:main' into cap-curve

0e0073a

fix cap curve name/description

0a83ca2

JosephBARBIERDARNAL mentioned this pull request Oct 6, 2024

chance_level_kw in RocCurveDisplay raises an error when using valid matplotlib args #30015

Closed

cap curve tests + docstrings + small updates

36f7ffa

glemaitre self-requested a review November 3, 2024 10:24

glemaitre added this to the 1.7 milestone Nov 3, 2024

ogrisel reviewed Nov 25, 2024

View reviewed changes

ogrisel reviewed Nov 27, 2024

View reviewed changes

sklearn/metrics/_plot/cap_curve.py Outdated Show resolved Hide resolved

JosephBARBIERDARNAL and others added 2 commits January 11, 2025 11:04

Merge branch 'main' into cap-curve

192bc8b

various capcurve improvements + tests

72da696

remove gini function in examples

ea7b322

ogrisel reviewed Apr 30, 2025

View reviewed changes

examples/model_selection/plot_det.py Outdated Show resolved Hide resolved

JosephBARBIERDARNAL and others added 12 commits May 3, 2025 12:24

Merge branch 'main' into cap-curve

5789286

Update examples/linear_model/plot_poisson_regression_non_normal_loss.py

343e740

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

Update examples/linear_model/plot_poisson_regression_non_normal_loss.py

ed76d2a

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

Update examples/model_selection/plot_det.py

a8547c6

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

handle explicit classification prob

8371c06

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

test despine

bfed568

use consistent variable names

148043c

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

fix match in tests

40d0ec3

handle x_max/y_max equal 0

d5a4e99

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

fix typo error test

91bbc6d

add tests for name

8d97373

refacto test attributes and add edge cases

290d162

glemaitre reviewed May 5, 2025

View reviewed changes

ArturoAmorQ reviewed May 7, 2025

View reviewed changes

JosephBARBIERDARNAL and others added 8 commits May 7, 2025 17:08

Merge branch 'main' into cap-curve

ec7ffc2

set consistent kwargs naming

f8cf77c

add mention of regressors in y_pred doc

19e61cd

update pos_label doc

2a55a4b

Merge branch 'main' into cap-curve

b992762

reorder paragraph in example

e215fd1

reorder paragraph in example

b8ca8cd

remove unnecessary argument

9cb6124

Co-authored-by: Arturo Amor <86408019+ArturoAmorQ@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH add CAP curve #28972

ENH add CAP curve #28972

JosephBARBIERDARNAL commented May 7, 2024 •

edited

Loading

github-actions bot commented May 7, 2024 •

edited

Loading

lorentzenchr commented May 21, 2024

JosephBARBIERDARNAL commented Jun 25, 2024

JosephBARBIERDARNAL commented Oct 6, 2024 •

edited

Loading

glemaitre commented Nov 3, 2024

JosephBARBIERDARNAL commented Nov 3, 2024

ogrisel Nov 25, 2024 •

edited

Loading

ogrisel left a comment

JosephBARBIERDARNAL commented Nov 27, 2024 •

edited

Loading

lorentzenchr commented Nov 30, 2024

JosephBARBIERDARNAL commented Jan 11, 2025

JosephBARBIERDARNAL commented Jan 11, 2025 •

edited

Loading

JosephBARBIERDARNAL commented Apr 28, 2025 •

edited

Loading

JosephBARBIERDARNAL commented Apr 28, 2025

ogrisel commented Apr 29, 2025

ogrisel commented Apr 29, 2025

ogrisel left a comment

ogrisel Apr 30, 2025 •

edited

Loading

JosephBARBIERDARNAL May 7, 2025 •

edited

Loading

glemaitre left a comment

glemaitre May 5, 2025

ArturoAmorQ left a comment

ArturoAmorQ May 7, 2025 •

edited

Loading

JosephBARBIERDARNAL May 7, 2025

ArturoAmorQ May 7, 2025

JosephBARBIERDARNAL May 7, 2025

ArturoAmorQ May 7, 2025

JosephBARBIERDARNAL May 7, 2025

ArturoAmorQ May 7, 2025

JosephBARBIERDARNAL commented May 7, 2025 •

edited

Loading

		self.cumulative_total = self.cumulative_total / x_max
		self.y_true_cumulative = self.y_true_cumulative / y_max

ENH add CAP curve #28972

Are you sure you want to change the base?

ENH add CAP curve #28972

Conversation

JosephBARBIERDARNAL commented May 7, 2024 • edited Loading

Reference Issue

What does this implement/fix?

other

TODO

Binary classification

Regression

Other

Nice to have

github-actions bot commented May 7, 2024 • edited Loading

✔️ Linting Passed

lorentzenchr commented May 21, 2024

JosephBARBIERDARNAL commented Jun 25, 2024

JosephBARBIERDARNAL commented Oct 6, 2024 • edited Loading

glemaitre commented Nov 3, 2024

JosephBARBIERDARNAL commented Nov 3, 2024

ogrisel Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

ogrisel left a comment

Choose a reason for hiding this comment

JosephBARBIERDARNAL commented Nov 27, 2024 • edited Loading

lorentzenchr commented Nov 30, 2024

JosephBARBIERDARNAL commented Jan 11, 2025

JosephBARBIERDARNAL commented Jan 11, 2025 • edited Loading

JosephBARBIERDARNAL commented Apr 28, 2025 • edited Loading

JosephBARBIERDARNAL commented Apr 28, 2025

ogrisel commented Apr 29, 2025

ogrisel commented Apr 29, 2025

ogrisel left a comment

Choose a reason for hiding this comment

ogrisel Apr 30, 2025 • edited Loading

Choose a reason for hiding this comment

JosephBARBIERDARNAL May 7, 2025 • edited Loading

Choose a reason for hiding this comment

glemaitre left a comment

Choose a reason for hiding this comment

glemaitre May 5, 2025

Choose a reason for hiding this comment

ArturoAmorQ left a comment

Choose a reason for hiding this comment

ArturoAmorQ May 7, 2025 • edited Loading

Choose a reason for hiding this comment

JosephBARBIERDARNAL May 7, 2025

Choose a reason for hiding this comment

ArturoAmorQ May 7, 2025

Choose a reason for hiding this comment

JosephBARBIERDARNAL May 7, 2025

Choose a reason for hiding this comment

ArturoAmorQ May 7, 2025

Choose a reason for hiding this comment

JosephBARBIERDARNAL May 7, 2025

Choose a reason for hiding this comment

ArturoAmorQ May 7, 2025

Choose a reason for hiding this comment

JosephBARBIERDARNAL commented May 7, 2025 • edited Loading

JosephBARBIERDARNAL commented May 7, 2024 •

edited

Loading

github-actions bot commented May 7, 2024 •

edited

Loading

JosephBARBIERDARNAL commented Oct 6, 2024 •

edited

Loading

ogrisel Nov 25, 2024 •

edited

Loading

JosephBARBIERDARNAL commented Nov 27, 2024 •

edited

Loading

JosephBARBIERDARNAL commented Jan 11, 2025 •

edited

Loading

JosephBARBIERDARNAL commented Apr 28, 2025 •

edited

Loading

ogrisel Apr 30, 2025 •

edited

Loading

JosephBARBIERDARNAL May 7, 2025 •

edited

Loading

ArturoAmorQ May 7, 2025 •

edited

Loading

JosephBARBIERDARNAL commented May 7, 2025 •

edited

Loading