Skip to content

Add sklearn.metrics.cumulative_gain_curve and sklearn.metrics.lift_curve #10003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
reiinakano opened this issue Oct 25, 2017 · 8 comments · May be fixed by #18479 or #15176
Open

Add sklearn.metrics.cumulative_gain_curve and sklearn.metrics.lift_curve #10003

reiinakano opened this issue Oct 25, 2017 · 8 comments · May be fixed by #18479 or #15176

Comments

@reiinakano
Copy link
Contributor

reiinakano commented Oct 25, 2017

Description

I recently added plot_cumulative_gain and plot_lift_curve methods to https://github.com/reiinakano/scikit-plot. To do this, I built an adhoc version of cumulative_gain_curve closely following the sklearn.metrics.roc_curve interface at https://github.com/reiinakano/scikit-plot/blob/master/scikitplot/helpers.py#L157. Let me know if sklearn.metrics.cumulative_gain_curve is something you'd be interested in adding into scikit-learn. I could add example docs for plotting gain and lift curves as well.

Reference I followed for lift and gain: https://www.ibm.com/support/knowledgecenter/en/SSLVMB_23.0.0/spss/tutorials/mlp_bankloan_outputtype_02.html

plot_cumulative_gain

plot_lift_curve

@jnothman
Copy link
Member

jnothman commented Nov 4, 2017

This is the first time I'm seeing these kinds of plots. I like CG curve at a glance, as an alternative to looking at gradients in an ROC curve, I suppose. Using more familiar terms, we're just plotting recall@k (for each class). Perhaps we should name it as such. Recall@k is something we should probably have implemented for the fixed k case...

@jnothman
Copy link
Member

jnothman commented Nov 4, 2017

I've not yet understood the utility of lift.

@RNAer
Copy link

RNAer commented Jun 12, 2018

the mixed usage of lift and gain in different papers is very confusing. The original paper for lift is here:

Ling C, Li C (1998). “Data Mining for Direct Marketing: Problems and solutions.”
In “Proceedings of the Fourth International Conference on Knowledge
Discovery and Data Mining,” pp. 73–79.

It is also very well explained in Kuhn's book of Applied Predictive Modeling

@GuillemGSubies
Copy link
Contributor

Any progress here? An intuitive explanation of the lift curve can be found here:

http://www2.cs.uregina.ca/~dbd/cs831/notes/lift_chart/lift_chart.html

It is like "how much better than the random model I am doing at each percentile"

@adrinjalali
Copy link
Member

@jnothman , what @GuillemGSubies provides is convincing to me. Given the utility of these metrics and its usage, which seems to be used, I'm leaning towards accepting this feature.

@lorentzenchr
Copy link
Member

From Ling & Li 1998

For any point (p, q) where p < q on the ROC curve, the corresponding point on the lift curve would be (q*X% + p*(1-X%), q), where X is the percent of positive cases in the dataset. Thus, the lift curve is slightly under the ROC curve when p < q. Therefore, the lift index we propose is also closely related to the area under the curve (AUC) index of the ROC curves.

From wikipedia:

The lift curve can also be considered a variation on the receiver operating characteristic (ROC) curve, and is also known in econometrics as the Lorenz or power curve.[1]

@lorentzenchr
Copy link
Member

lorentzenchr commented Oct 7, 2022

TLDR

+1 for inclusion of the gain curve/CAP. Naming should reflect different strands of literature: cumulative accuracy profile (CAP) [2][4], concentration curve [3], cumulative lift curve [5]. It should work for binary classification as well as regression (models for the expectation).

Some more background

The cumulative gains curve is the same as the Cumulative Accuracy Profile (CAP), see [1] and [4].
From [2]

Moody’s uses Cumulative Accuracy Profiles (CAP), to make visual, qualitative assessments of model performance. While similar tools exist under a variety of different names (lift-curves, dubbed-curves, receiver-operator curves, power curves, etc.).

References:
[1] Tasche 2006 "Validation of internal rating systems and PD estimates"
https://arxiv.org/pdf/physics/0606071.pdf
[2] Soběhart, J.R., Keenan, S.C., & Stein, R.M. (2000). "Benchmarking Quantitative Default Risk Models: A Validation Methodology"
[3] Denuit, M., Trufin, J. (2021). "Lorenz curve, Gini coefficient, and Tweedie dominance for autocalibrated predictors"
https://dial.uclouvain.be/pr/boreal/object/boreal%3A254535/datastream/PDF_01/view
[4] https://www.listendata.com/2019/09/gini-cumulative-accuracy-profile-auc.html
[5] Ling C, Li C (1998). "Data Mining for Direct Marketing: Problems and solutions."
In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, pp. 73–79.

@ogrisel
Copy link
Member

ogrisel commented Apr 4, 2024

The following reference seems also very relevant:

https://arxiv.org/abs/2207.14372

In particular:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Discussion
9 participants