@aijordan For your information.

It's possible that using Centered Isotonic Regression (#21454) would make the reliability diagram look even better but might break the theoretical results of the paper you linked above.

@ogrisel As pointed out in #21454, centered isotonic regression seems to be an invalid option because it is not calibrated itself which is a crucial property of (standard) isotonic regression, in particular important for assessing (auto-) calibration of a model as is done in a reliability diagram.

Posted by @ogrisel in #23767 (comment)

My main concern with the CORP reliability diagrams is that they are very square-looking on finite size test sets, see for instance:

Those diagrams would probably qualitatively look very different in the asymptotic regime of large test sets.

To avoid this finite test sample artifact, we might also what to consider methods such as implemented in:

https://github.com/apple/ml-calibration

Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing
Jarosław Błasiok, Preetum Nakkiran
https://arxiv.org/abs/2309.12236

@ogrisel

My main concern with the CORP reliability diagrams is that they are very square-looking on finite size

As Prof Simon Woods says:

Statistics is the honest interpretation of data

The reliability diagram is a statistical diagnostics/verification tool and not something that needs to be pleasant for the eye, but easy to interpret.
The zigzag is a mere consequence of the underlying (small sample) uncertainty in the estimation of $E[Y|prediction]$.

BTW, I never understood why the PAV (=CORP) approach is good enough for calibration of classifiers (modifies actual model predictions), but not good enough in a reliability diagram (diagnostics) - within scikit-learn 🤨

The reliability diagram is a statistical diagnostics/verification tool and not something that needs to be pleasant for the eye, but easy to interpret.

It's not a matter of being pleasing to the eye but a matter about being misleading about the shape of the asymptotic curve. The asymptotic curve will be smooth most of the time, and the CORP finite sample size estimate can induce the reader into thinking otherwise, which I find misleading and a potential source of confusion for our users.

BTW, I never understood why the PAV (=CORP) approach is good enough for calibration of classifiers (modifies actual model predictions), but not good enough in a reliability diagram (diagnostics) - within scikit-learn

I actually have the exact same concern with isotonic regression as a post hoc calibrator. I would much rather use the centered isotonic regression as post hoc classifier which is mostly strictly monotonic (and as a result would not introduce an unexpected change to pure ranking metrics such as ROC AUC / Gini index) and converges to the same solution as the step-wise constant calibrator in the large sample size.

The asymptotic curve will be smooth most of the time

That‘s not correct. For instance, tree based models or GLMs with categorical features are not smooth in the predictions.

That‘s not correct. For instance, tree based models or GLMs with categorical features are not smooth in the predictions.

Indeed that might be the case. Although depending on the size of the training set, I suspect that they are still be much smoother than what the CORP reliability diagram suggests. To settle this debate we will need some experiments with a few large datasets we we can subsample both the training set and the test set used to estimate the reliability curve and compare the small test sample CORP/smoothed reliability curves to the CORP curve on the full test set.

We could also have a reliability diagram with a user settable option to decide what strategy they want to use (fixed binning as we do now, CORP induced bin edges and some smooth estimate). Still comparing the methods on a few canonical datasets would help us make informed recommendations in the docstring of that parameter.