scikit-learn · ogrisel · Oct 29, 2020 · Jun 24, 2020 · Jun 25, 2020 · Jun 30, 2020
diff --git a/doc/modules/calibration.rst b/doc/modules/calibration.rst
@@ -96,9 +96,9 @@ in [0, 1]. Denoting the output of the classifier for a given sample by :math:`f_
 the calibrator tries to predict :math:`p(y_i = 1 | f_i)`.
 
 The samples that are used to fit the calibrator should not be the same
-samples used to fit the classifier, as this would
-introduce bias. The classifier performance on its training data would be
-better than for novel data. Using the classifier output from training data
+samples used to fit the classifier, as this would introduce bias.
+This is because performance of the classifier on its training data would be
+better than for novel data. Using the classifier output of training data
 to fit the calibrator would thus result in a biased calibrator that maps to
 probabilities closer to 0 and 1 than it should.
 
@@ -107,22 +107,39 @@ Usage
 
 The :class:`CalibratedClassifierCV` class is used to calibrate a classifier.
 
-:class:`CalibratedClassifierCV` uses a cross-validation approach to fit both
-the classifier and the regressor. The data is split into k
-`(train_set, test_set)` couples (as determined by `cv`). The classifier
-(`base_estimator`) is trained on the train set, and its predictions on the
-test set are used to fit a regressor. This ensures that the data used to fit
-the classifier is always disjoint from the data used to fit the calibrator.
-After fitting, we end up with k
-`(classifier, regressor)` couples where each regressor maps the output of
-its corresponding classifier into [0, 1]. Each couple is exposed in the
-`calibrated_classifiers_` attribute, where each entry is a calibrated
+:class:`CalibratedClassifierCV` uses a cross-validation approach to ensure
+unbiased data is always used to fit the calibrator. The data is split into k
+`(train_set, test_set)` couples (as determined by `cv`). When `ensemble=True`
+(default), the following procedure is repeated independently for each
+cross-validation split: a clone of `base_estimator` is first trained on the
+train subset. Then its predictions on the test subset are used to fit a
+calibrator (either a sigmoid or isotonic regressor). This results in an
+ensemble of k `(classifier, calibrator)` couples where each calibrator maps
+the output of its corresponding classifier into [0, 1]. Each couple is exposed
+in the `calibrated_classifiers_` attribute, where each entry is a calibrated
 classifier with a :term:`predict_proba` method that outputs calibrated
 probabilities. The output of :term:`predict_proba` for the main
 :class:`CalibratedClassifierCV` instance corresponds to the average of the
-predicted probabilities of the `k` estimators in the
-`calibrated_classifiers_` list. The output of :term:`predict` is the class
-that has the highest probability.
+predicted probabilities of the `k` estimators in the `calibrated_classifiers_`
+list. The output of :term:`predict` is the class that has the highest
+probability.
+
+When `ensemble=False`, cross-validation is used to obtain 'unbiased'
+predictions for all the data, via
+:func:`~sklearn.model_selection.cross_val_predict`.
+These unbiased predictions are then used to train the calibrator. The attribute
+`calibrated_classifiers_` consists of only one `(classifier, calibrator)`
+couple where the classifier is the `base_estimator` trained on all the data.
+In this case the output of :term:`predict_proba` for
+:class:`CalibratedClassifierCV` is the predicted probabilities obtained
+from the single `(classifier, calibrator)` couple.
+
+The main advantage of `ensemble=True` is to benefit from the traditional
+ensembling effect (similar to :ref:`bagging`). The resulting ensemble should
+both be well calibrated and slightly more accurate than with `ensemble=False`.
+The main advantage of using `ensemble=False` is computational: it reduces the
+overall fit time by training only a single base classifier and calibrator
+pair, decreases the final model size and increases prediction speed.
 
 Alternatively an already fitted classifier can be calibrated by setting
 `cv="prefit"`. In this case, the data is not split and all of it is used to

diff --git a/doc/whats_new/v0.24.rst b/doc/whats_new/v0.24.rst
@@ -58,6 +58,14 @@ Changelog
   sparse matrix or dataframe at the start. :pr:`17546` by
   :user:`Lucy Liu <lucyleeow>`.
 
+- |Enhancement| Add `ensemble` parameter to
+  :class:`calibration.CalibratedClassifierCV`, which enables implementation
+  of calibration via an ensemble of calibrators (current method) or
+  just one calibrator using all the data (similar to the built-in feature of
+  :mod:`sklearn.svm` estimators with the `probabilities=True` parameter).
+  :pr:`17856` by :user:`Lucy Liu <lucyleeow>` and
+  :user:`Andrea Esuli <aesuli>`.
+
 :mod:`sklearn.cluster`
 ......................