Any recommendations/comments. The lift metric seems to be a desired feature in previous PRs.
@jnothman, @rth, @GuillemGSubies I am ating you as you had a look at the previous PRs related #18479 and #10003
Thanks!

Thanks @nawarhalabi this looks quite nice. However, given that the major bottleneck in scikit-learn is in review time, I'd suggest you break it into smaller contributions, starting with the lift_curve function and its tests. For lift_score I'd be interested in seeing references to its use in practice for classification evaluation or diagnostics.

Thanks @jnothman

• Regarding splitting the contribution: Sure! I will limit this PR to lift_curve and its tests. I will create a new branch off this branch for the lift_score and create a separate PR for that later. Does this sound good?
• Regarding references: I will add references to the docstring of the lift_score function in the future PR. Here are the ones I would add:
  • https://www.kdnuggets.com/2016/03/lift-analysis-data-scientist-secret-weapon.html
  • https://en.wikipedia.org/wiki/Lift_(data_mining)
  • https://howtolearnmachinelearning.com/articles/the-lift-curve-in-machine-learning/

Just to clarify expectations: While splitting this PR will help in the review process, it is not yet decided to include the proposed functionality. In #21718, we are trying to figure out some principles for model evaluation tools.

@lorentzenchr do you think these days we'd be including this in the codebase?

Or is this something we'd be happier to have in skrub @GaelVaroquaux @ogrisel

or maybe scikit-lego? (@koaning)

I'd be open to adding it to scikit-lego. But at first glance it does feel general enough that it could also live here. If the conclusion is that it's not a great fit for sklearn or skrub then I'll gladly consider it for sklego.

I think an "ML Gini index" as well as an accompanying graph "Cumulative Accuracy Profile" (CAP)¹ for non-negative regression (of which binary classification is an example) would complete our tools for measuring ranking (discriminative) power of models and generalize the existing AUC and ROC.

Note that there is a great confusion about terms. In Gini Index and Friends, my coauthors and me summarized existing literature and gave tutorial like examples. This is now my main reference if I look up terms like "Gini index" (which one?!?!?).

¹ aka gain curve and (cumulative) lift curve, it is almost the inverted Lorenz curve of the empirical distribution generated by the model predictions.

@nawarhalabi would you be able to give this PR an update?

For sure we should prioritize this action when it comes to the inspection.

For sure we should prioritize this action when it comes to the inspection.

What do you mean by action?

Wrong word, my brain did not work properly. By "action" I meant that we should have a display for this type of curve.