More details about the existing OvR case: we found out that a naive softmax normalization of the raw decision function of the OvR LinearSVC was often competitive with Platt / isotonic calibration of the binary classifier followed by a simple probs.sum(axis=1) normalization in terms of logloss. Maybe the choice of the logloss is the source of the problem (see below) but by re-reading the main paper we reference for multiclass classification Transforming Classifier Scores into Accurate Multiclass Probability Estimates by Bianca Zadrozny and CharlesElkan I have the feeling that the subject of multiclass calibration has not been properly investigated. In particular there are only 2 toy experiments in the paper (pendigits and 20 newsgroups) only with Naive Bayes and Boosted NB as the base classifier. Furthermore the theoretical justification for binary calibration (OvR or other) followed by probs.sum(axis=1) normalization seems very weak.

I also think using the logloss to evaluate multiclass calibration was not necessary a good idea: for not perfectly calibrated models, there is not guarantee that the model with the "best" but imperfect calibration has the lowest log loss. I changed the multiclass calibration test in deb75fc to use a multiclass version of the Brier loss which seems slightly more stable (e.g. try to change the seed) but it also seems like an imperfect calibration metric (see #10883). Maybe we should try to extend Expected Calibration Error (#11096) to the multiclass setting but I am not sure whether this is common practice or not.

Another baseline we could compare to: stacking the uncalibrated model with a multinomial logistic regression or multinomial gradient boosting, possibly with positivity constrained (for LR) or monotonicity constraints for GBRT.

If the later baseline proves to work in extensive benchmarks with various base classifiers on various multiclass classification datasets, it would be worth documenting it as an example and linking this strategy as an alternative to CalibratedClassifierCV in the user guide.

Mentioning @lucyleeow @dsleo @samronsin as you might be interested in this and maybe sharing your own insights.

We could even introduce an additional temperature hyperparemeter in the multinomial loss of LR / HGBRT. It would be set to 1. by default to get regular LR but could be grid searched with multiclass ECE when those models are used as second stage calibrators (instead of relying on a regularization alone).

Cross-referencing a survey of our community on twitter: https://twitter.com/ogrisel/status/1322119718334013443 with very relevant references in the replies.