This PR is a dupplicate from #28752. I'm recreating a PR because I accidentally deleted my fork with the old one.

Reference Issue

fix for #10003

What does this implement/fix?

creation of a CumulativeAccuracyDisplay class for plots

"The CAP of a model represents the cumulative number of positive outcomes along the y-axis versus the corresponding cumulative number of a classifying parameter along the x-axis. The output is called a CAP curve.[1] The CAP is distinct from the receiver operating characteristic (ROC) curve, which plots the true-positive rate against the false-positive rate." (wikipedia definition)

It's mainly inspired from the RocCurveDisplay class.

other

It's currently a work in progress.

TODO

Binary classification

• raise a ValueError in from_estimator if the estimator is not fitted or is a classifier that was fitted with more than 3 classes;
• fix pos_label handling when the positive class;
• add/update a test to check that we have the same result for response_method="decision_function" and response_method="predict_proba" for a LogisticRegression classifier fit with string labels and for all 3 possible values of pos_label;
  • not that this is different from what is already tested in test_display_from_estimator_and_from_prediction;
  • CAP curves should be invariant under order preserving transformations of the predictions. This is way plotting CAP from the unormalized logits or the logistic sigmoid scaled probabilistic predictions should result in the same curve: the logistic sigmoid is strictly monotonic, hence order preserving.
    • joseph --> should be good
• add an option (enabled by default) to draw CAP curve of the "perfect"/"oracle" model;
• update the tests to check that the model curves always lie between the "chance level" and "perfect"/"oracle" curves.
• add a test to check that the display array attributes y_true_cumulative and cumulative_total have the same dtype as y_pred in the test about from_predictions. We can test for y_pred passed either as np.float32 or np.float64.
  • joseph: should be good
• test that CAPCurveDisplay.from_estimator(LinearSVC().fit(X, y), ...) works (even if it does not have a predict_proba method. This should cover one of the line reported as uncovered by codecov.
• leverage test_common_curve_display.py to reuse some generic tests on CAPCurveDisplay and maybe remove redundant tests on invalid inputs from test_cap_curve_display.py if any;
  • joseph: should be good
• add despine argument?
  • olivier: I would rather not do that specifically for that PR but maybe consider a cross-display PR that does that for all *Display classes in scikit-learn. Feel free to open an issue to discuss this with screen shots e.g. on ROC or PR curves and your analysis of pros and cons.
  • joseph: sure. I suggested it here because the ROC and PR curves already have it (see ENH despine keyword for ROC and PR curves #26367). I'm not sure it makes much sense for ConfusionMatrixDisplay (?). I'll open an issue (when this PR will be merged) for CAPCurveDisplay, PredictionErrorDisplay and DetCurveDisplay because I think they're the only ones that don't have this option.
  • should be good --> I added the argument, as per suggested below in the discussion

Regression

• update the docstrings to make it explicit that either regressors (with positive outcomes) or binary classifiers are accepted and anything else is rejected.
  • joseph: I made a first pass, but I guess it can be improved
• raise a ValueError with an informative error message if y_true has negative values;
  • joseph: it is now tested here
• raise a ValueError if all y_true are zeros (the plot would be degenerate and would raise a low level divide by zero warning whith normalize_scale=True);
  • In theory, this should not happen, since if all the y_true are zeros, it will be considered a case of classification
• add tests with continuous outputs on positive observation data (e.g. using PoissonRegressor) and check that the regressor curve lie between the "chance level" and "perfect" curves;
  • joseph: it is now tested here
• update the insurance model examples (examples/linear_model/plot_tweedie_regression_insurance_claims.py and examples/linear_model/plot_poisson_regression_non_normal_loss.py) to use the CAPCurveDisplay class instead of manually plotting the Lorenz curves.

Other

• document the new feature with a new entry under doc/whats_new/upcoming_changes/
  • done here
• update the user guide (doc/visualization) to reference this new tool.
  • done here.
  • joseph: One thing I find strange is that the CAP curve is the only one where all the letters of the acronym are in capitals. Is this explicitly intended? There's DetCurveDisplay, RocCurveDisplay and then CAPCurveDisplay, which seems inconsistent.

Nice to have

• add a test to check the sample weight/repetition equivalence property: ENH add CAP curve #28972 (comment)