Then the predictions from all the folds are concatenated in a single list, and calibration parameters for the base_estimator are determined using such list.

In other words, we should use cross_val_predict instead of a cross_val_score.

The procedure described in Platt99 is what one would expect from a proper application of a cross validation procedure, as the cross validation only determines the parameters of the calibration and does not fit the estimator.
It is also more efficient, as is does not store the estimators for each fold and requires a single predict at prediction time.

IMO, using the ensemble will be more stable and powerful than using a single learner. I assume that it should attenuate changes linked to model parameters found for different splits.

ping @GaelVaroquaux @agramfort you should be better aware than me for commenting.

@aesuli I thing I understand what you asking. Before suggesting any change to the way we do things 2 questions:

• you suggest that CalibratedClassifierCV should not change the base_estimator . If so how do you end a pipeline with such an estimator? where do you get your pretrain model from? can I still do this with fit(X, y) from scratch?
• in libsvm https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2099 I see a for loop over folds when I fit with probability=True which is what we tried to replicate offering more than platt scaling / sigmoid for calibration method. Are you suggesting that what we implement is not exactly what libsvm does? thanks

@glemaitre right! cross_val_predict can be the way to get CV predictions for the whole training set and then compute calibration from them.

@agramfort

• you suggest that CalibratedClassifierCV should not change the base_estimator . If so how do you end a pipeline with such an estimator? where do you get your pretrain model from? can I still do this with fit(X, y) from scratch?
  I meant that the CalibratedClassifierCV should not alter the form of the base_estimator.
  If the base estimator is a linear_svm it should fit a linear_svm on the whole training set, not fit an ensemble of linear_svm on folds.

The fit(X,y) of the CalibratedClassifierCV could be something like (rough pseudo code):

def fit(X,y):
  if not self.cv == 'prefit': #Platt99-style CV calibration
    self.base_estimator.fit(X,y)
    cv_predictions = cross_val_predict(self.base_estimator, X, y, cv=self.cv)
    self.calibration_params = self.estimate_calibration(cv_predictions)
  else: # prefit, assuming base_estimator is already fit on different data
    predictions = self.base_estimator.predict_proba(X)
    self.calibration_params = self.estimate_calibration(predictions)

def predict_proba(X):
    predictions = self.base_estimator.predict_proba(X)
    calibrated_predictions = self.apply_calibration_model(predictions, self.calibration_params)
    return calibrated_predictions

• in libsvm https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/svm/src/libsvm/svm.cpp#L2099 I see a for loop over folds when I fit with probability=True which is what we tried to replicate offering more than platt scaling / sigmoid for calibration method. Are you suggesting that what we implement is not exactly what libsvm does? thanks

Yes the current CalibratedClassiferCV implementation differs from libsvm.
svm.cpp performs CV only to get predictions that are all stored together in dec_values
dev_values is populated in various points of the code depending on the composition of the training set for a fold (see here and here).
At the end of each fold the model fitted for that fold is destroyed.
The calibration parameters are computed on all the predictions for the training set.

OK, I'll work on this in the next days.