Isotonic calibration changes rank-based test metrics values #16321
Description
[As discussed with @ogrisel]
Describe the bug
Using isotonic calibration changes metrics values. This is because it is a non-strictly monotonic calibration. Sigmoid calibration being strictly monotonic doesn't suffer from this.
Steps/Code to Reproduce
Here is quick example where we split in three train/calibration/test sets and compare ROC AUC on the test set before and after calibrating for both isotonic and sigmoid..
import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import roc_auc_score
X, y = datasets.make_classification(n_samples=100000, n_features=20,
n_informative=18, n_redundant=0,
random_state=42)
X_train, X_, y_train, y_ = train_test_split(X, y, test_size=0.5,
random_state=42)
X_test, X_calib, y_test, y_calib = train_test_split(X_, y_, test_size = 0.5,
random_state = 42)
clf = LogisticRegression(C=1.)
clf.fit(X_train, y_train)
y_pred = clf.predict_proba(X_test)
print(roc_auc_score(y_test, y_pred[:,1]))The ROC AUC is then: 0.88368
isotonic = CalibratedClassifierCV(clf, method='isotonic', cv='prefit')
isotonic.fit(X_calib, y_calib)
y_pred_calib = isotonic.predict_proba(X_test)
print(roc_auc_score(y_test, y_pred_calib[:,1]))After isotonic calibration, the ROC AUC becomes: 0.88338
isotonic = CalibratedClassifierCV(clf, method='sigmoid', cv='prefit')
isotonic.fit(X_calib, y_calib)
y_pred_calib = isotonic.predict_proba(X_test)
print(roc_auc_score(y_test, y_pred_calib[:,1]))As expected for sigmoid calibration, the ROC AUC is constant. 0.88368
Versions
System:
python: 3.8.1 | packaged by conda-forge | (default, Jan 29 2020, 15:06:10) [Clang 9.0.1 ]
executable: /Users/leodreyfusschmidt/opt/miniconda2/envs/isotonic/bin/python
machine: macOS-10.13.4-x86_64-i386-64bit
Python dependencies:
pip: 20.0.2
setuptools: 45.1.0.post20200119
sklearn: 0.22.1
numpy: 1.16.5
scipy: 1.4.1
Cython: None
pandas: None
matplotlib: 3.1.2
joblib: 0.14.1
Built with OpenMP: True