Skip to content

[MRG] ROCCH calibration method #4822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[MRG] ROCCH calibration method #4822

wants to merge 5 commits into from
+254 −23

Conversation

albahnsen
Copy link
Contributor

@albahnsen albahnsen commented Jun 5, 2015 •
edited by ogrisel
Loading

Implementation of the ROCCH calibration method, as described in http://www.jmlr.org/papers/volume13/hernandez-orallo12a/hernandez-orallo12a.pdf and http://link.springer.com/article/10.1007/s10994-007-5011-0

It is implemented as an other method within the CalibratedClassifierCV class. Moreover, the documentation is expanded to include the results.

The method performs similarly to Isotonic calibration when measured by Brier loss, F1 Score or AUC. But with an speedup of up to 1.3X

Here an example

In[2]: %paste

import pandas as pd
from sklearn import datasets
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV
from sklearn.cross_validation import train_test_split
from sklearn.metrics import brier_score_loss, f1_score, roc_auc_score

X, y = datasets.make_classification(n_samples=100000, n_features=20,
                                    n_informative=2, n_redundant=10,
                                    random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,
                                                    random_state=42)

def test_times(method, est):
    clf =  CalibratedClassifierCV(est, cv=5, method=method)
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    y_prob = clf.predict_proba(X_test)[:,1]
    brier = brier_score_loss(y_test, y_prob, pos_label=y.max())
    f1 = f1_score(y_test, y_pred)
    auc = roc_auc_score(y_test, y_prob)
    return brier, f1, auc

res = pd.DataFrame(columns=['brier_loss','f1_score','auc_score'])
## -- End pasted text --
Backend TkAgg is interactive backend. Turning interactive mode on.
In[3]: %timeit res.loc['GNB+isotonic'] = test_times(method='isotonic', est=GaussianNB())
1 loops, best of 3: 823 ms per loop
In[4]: %timeit res.loc['GNB+sigmoid'] = test_times(method='sigmoid', est=GaussianNB())
1 loops, best of 3: 776 ms per loop
In[5]: %timeit res.loc['GNB+rocch'] = test_times(method='rocch', est=GaussianNB())
1 loops, best of 3: 752 ms per loop
In[6]: %timeit res.loc['LSVC+isotonic'] = test_times(method='isotonic', est=LinearSVC())
1 loops, best of 3: 416 ms per loop
In[7]: %timeit res.loc['LSVC+sigmoid'] = test_times(method='sigmoid', est=LinearSVC())
1 loops, best of 3: 367 ms per loop
In[8]: %timeit res.loc['LSVC+rocch'] = test_times(method='rocch', est=LinearSVC())
1 loops, best of 3: 324 ms per loop
In[9]: res['time'] = [823, 776, 752, 416, 367, 324]
In[10]: print res
brier_loss f1_score auc_score time
GNB+isotonic 0.098487 0.854058 0.939305
GNB+sigmoid 0.108937 0.866259 0.937025
GNB+rocch 0.098752 0.851963 0.939027
LSVC+isotonic 0.099401 0.864936 0.936852
LSVC+sigmoid 0.098603 0.862041 0.937657
LSVC+rocch 0.099584 0.864046 0.936733

@agramfort
Copy link
Member

what is the practical benefit of this method? speed improvement is marginal. Can you expect much better performance on some setups. If so when?

thanks

@amueller
Copy link
Member

amueller commented Jun 5, 2015

This is a relatively common method, right?
We discussed finding different threshold here: #4813.
That would target more the predict not the predict_proba, though.

@albahnsen
Copy link
Contributor Author

@agramfort the ROCCH calibration is design to maximize the AUC (At least during training) therefore it is a better method when the estimated probabilities will be used afterwards and not just the prediction, i.e. in cost-sensitive classification.
Both the ROCCH and Isotonic methods fix the monotonicity of the estimated probabilities, moreover is this paper (http://link.springer.com/article/10.1007/s10994-007-5011-0) it is discussed the practical relation between them.
The speedup is gained because the ROCCH does not fit an additional model as the Isotonic method does. However, the ROCCH method does need to find a convex hull of the ROC curve, which, on special cases, may be very time consuming.

@albahnsen
Copy link
Contributor Author

@amueller I do find some relation between this and #4813 as the calibration of probabilities is changing the threshold of the classifier indirectly. Nevertheless, I dont think this is what @rahuldave had in mind when he open that issue.

@albahnsen
Copy link
Contributor Author

Regarding the Travis CI build. This code needs SciPy >=0.12, while the Travis test is being executed with Scipy 0.9. Initially, I had my own convex hull function, but it is slower than the one SciPy has.

@amueller
Copy link
Member

amueller commented Jun 8, 2015

We need to support scipy 0.9. Usually the way around this is to backport the function to scikit-learn. I'm not sure you should do the work before we agree on inclusion, though.

@amueller
Copy link
Member

amueller commented Jun 8, 2015

What do you mean by not fitting an additional model? You mean the isotonic model?
It maximizes the ROC on the hold-out validation set, right, not on the training set?

@amueller
Copy link
Member

amueller commented Jun 8, 2015

I'm not sure I have time to look into the papers right now. Do they use a hold-out method for the threshold or do they use the training set?
The R package caret does something similar (by default I think), and I was wondering if they use training or validation, but I didn't get a reply from them yet.

@albahnsen
Copy link
Contributor Author

The support for scipy 0.9 its ok then. I will wait for the moment.

the ROC is maximized on the hold-out set, sorry my confusion. On the papers the hold-out method is also the one used.

I'm not familiar enough with the caret package.

@amueller
Copy link
Member

amueller commented Jun 8, 2015

I'm +1 on adding this as I feel it is a very general method.

@agramfort
Copy link
Member

agramfort commented Jun 10, 2015 via email

@albahnsen
Copy link
Contributor Author

@amueller I did the backport to scipy ConvexHull function
It is not the same code as the one in scipy rely on the Qhull libary which implies several c files that I dont need for this particular function. Instead, when scipy.__version__ <0.12 I use the convexhull code from the Python Cookbook 2005.

@amueller
Copy link
Member

Thanks, that seems like the way to go.

@albahnsen
Copy link
Contributor Author

Guys, this PR has been open for a while. Not sure which are the next steps

@amueller amueller changed the title [WIP] ROCCH calibration method [MRG] ROCCH calibration method Oct 10, 2016
@amueller
Copy link
Member

Hm because the PR was name [WIP] I guess it didn't get any reviews. Also maybe because it's hard to get reviewer's attention. Can you maybe rebase? Thanks!

@PGryllos PGryllos mentioned this pull request Oct 23, 2017
Closed
@amueller
Copy link
Member

wow still no reviews after two years :-/ sorry....

Base automatically changed from master to main January 22, 2021 10:48
@lucyleeow
Copy link
Member

Is there still interest in including this? cc @amueller ?

@adrinjalali
Copy link
Member

Unfortunately there hasn't been much support from the maintainer community here. So I think we can close until interest comes up again.

@ogrisel ogrisel mentioned this pull request Jun 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
module:utils
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@albahnsen @agramfort @amueller @lucyleeow @adrinjalali @cmarmo