Thanks @NicolasHug for the review.
I made the corrections you suggested.

Thanks for your patience @aesuli .

I understand the need for predictions_in_X, but this shows that the current _CalibratedClassifier design isn't ideal for this current PR. Changing the semantics of X is something we want to avoid. Since it's a private class, we should be able to modify it without trouble.

I think we need to decouple 2 different things which are currently mixed in _CalibratedClassifier:

1. the chaining of the classifier and its corresponding calibrator, for prediction
2. the training of the calibrator

For now, 1. is done in _CalibratedClassifier.predict_proba and 2. is done in _CalibratedClassifier.fit.

I think we should separate them. The training of the calibrator could just be a single function (I think). And we could have a new class that does 1., i.e. just chaining, not fitting. Here's a simplified example of what I have in mind:

class _CalibratedClassiferPipeline:
    # Simple pipeline chaining  classifier and its calibrator.

    #  the calibrated_classifiers_ attribute would be a list of
    #  _CalibratedClassifierPipeline now. We can remove _CalibratedClassifier.
    # That class has no fit method. Fitting of the classifier is done in
    # CalibratedClassifierCV, and fitting of the regressor is done in
    # _calibrate.

    def __init__(self, fitted_classifier, fitted_calibrator):
        pass

    def predict_proba(self, X):
        df = self.fitted_classifier.decision_function(X)  # or predict_proba if it doesn't exist
        return self.fitted_calibrator.predict_proba(df)
    
    def predict(self, X):
        pass  # similar stuff


def _fit_calibrator(fitted_classifier, method, y, X=None, df=None):
    # Take a fitted classifier as input, fit a calibrator, and return the
    # corresponding pipeline.
    # df is mutually exclusive with X
    # y should always be passed
    # This should handle the multiclass case too

    if df is None:
        df = fitted_classifier.decision_function(X)  # or predict_proba
    
    calibrator = _get_regressor(method)
    calibrator.fit(df, y)

    return _CalibratedClassiferPipeline(fitted_classifier, calibrator)

Sorry I couldn't get to this earlier. LMK what you think

@NicolasHug Ok I'll work on this.

Hi @aesuli, are you still working on this? Thanks!

@lucyleeow excuse me for my silence. I see that you correctly guessed that I cannot find time to work on this. Thank you for keeping this alive.

Ah no problem. I started with the refactoring suggested by @NicolasHug - I think you were right to abandon, it was really complicated! Please let me know if you want to take over/be involved. Otherwise I am happy to finish, will credit you if it eventually gets merged!