Looks like CalibratedClassifierCV doesn't support pipelines, see https://github.com/scikit-learn/scikit-learn/issues/19625 and https://github.com/scikit-learn/scikit-learn/discussions/19279 We should try to delegate the validation to the base estimator (https://github.com/scikit-learn/scikit-learn/discussions/19279#discussioncomment-313812)