Overall this looks okay. Try merging with the master branch and some of the tests should pass.

You can also parameterize and refactor the test.

It would also be nice if you could add a section to the user guide or add an example to the example gallery regarding the effect of this parameter.

Hi,
Thank you for your comments, I've merged my branch with sklearn's master branch and I've updated the user guide with this added feature in the calibration section. Please tell me if there is anything else to do.

@joshuacwnewton would you feel comfortable reviewing this one?

I have updated the PR with your comments, thank you @joshuacwnewton and @adrinjalali

@lucyleeow might be interested in reviewing?

Thanks to both the author and the reviewers for the PR. It looks very clean. However, it's not obvious to me as to why passing class_weight="balanced" to the calibrator is a good idea from a mathematical point of view. So before considering merging the I would really like to see an empirical study that shows that passing class_weight="balanced" to the calibrator (especially with cv="prefit") is actually improving the shape of a calibration curve (both with uniform and quantile binning) and the Brier score of a binary classifier it-self fitted with class_weight="balanced".

To make the study interesting the dataset should be large enough with a severe class imbalance (e.g. 1 to 10 or more between the positive and binary class) and not too easy. One way to generate such a dataset would be to use something like:

X, y = make_classification(
    n_samples=100_000,
    weights=[0.95, 0.05],  # ~20x more negative than positive samples
    n_features=10,
    n_informative=10,
    n_redundant=0,
    n_repeated=0,
)
X = X[:, :8]  # hide some informative features to make the problem more difficult

Then one could evaluate estimators such as RandomForestClassifier or make_pipeline(SplineTransformer(), LogisticRegression()).

Ideally, this study can be done with a notebook shared on gist.github.com.

If we fine situations where passing class_weight="balanced" to the calibrator improves the shape of the calibration curve or calibration-sensitive metrics such as the log loss or the Brier score (or threshold-sensitive "hard" metrics such as f1_score), then we can probably proceed to merge the PR as is, maybe after converting the notebook into an example.

If on the other hand with find out that passing class_weight="balanced" is always detrimental to calibration, I think I would still be in favor of accepting the PR but we should consider documenting in the docstring that using class_weight to calibrated classifier is rarely a good way to improve the calibration of classifiers for imbalanced classification problems.

In my experience class_weight="balanced" to the raw classifiers (without CalibratedClassifierCV) themselves does not seem to make a significant change in in the Brier score of such imbalanced datasets with the classifier proposed above (either it's in the CV sampling noise or it is actually detrimental!). So we are already misleading our users with other part of scikit-learn with this class_weight="balanced" option.

Somewhat related discussion: #13227 (in particular the example).

anybody working on the study? removing from the milestone for now, but happy to be pinged for reviews once the study is done.

@ogrisel could you please write why you think it might not be mathematically sane?

is this stalled PR available? would like to try and pick up the work , making the study described above and putting @amascia 's changes in the new codebase (ive already taken a look and it seems it can go in without many modifications)

Hi @daustria, please take a look at #17541 (comment), this PR is stalled but there is uncertainty about whether we add this feature, which needs to be decided first. Thank you.

yes, i saw the comment and i was thinking of trying my hand and making such a study. unless maybe it is not something for volunteers to do?

edit: no longer working on this, feel free to take over the work

Please go ahead, look forward to seeing your findings!