Skip to content

[WIP] Make it possible to pass an arbitrary regressor as method for CalibratedClassifierCV #21992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 13 commits into from
Closed

[WIP] Make it possible to pass an arbitrary regressor as method for CalibratedClassifierCV #21992

wants to merge 13 commits into from

Conversation

aperezlebel
Copy link
Contributor

Reference Issues/PRs

Fixes #21280.

What does this implement/fix? Explain your changes.

This is a draft of a proposed solution to #21280.

  • Implements a third calibration method using a custom scikit-learn regressor in CalibratedClassifierCV.
  • Update calibration tests with this new method.
  • Docstrings.
  • Docstring tests.
  • Find meaningful names for new variables/functions/classes.
  • Extend at least one calibration example to demonstrate this new capability.
  • Update the Probability calibration page in the user guide with a paragraph on this method.

@aperezlebel
Update tests with custom regressor and classifier
8167f18 
Test if CalibratedClassifierCV works with custom scikit-learn regressor, and if it raises an error when giving a classifier instead.
@aperezlebel
Update _fit_calibrator and create class to accept custom regressors
614007a
@aperezlebel
Fix edge case when score = 1
a54282e 
When proba=1, the bin n_bins+1 was given by np.digitize, which is out of bound
@aperezlebel
Update test_calibration_ensemble_false and fix errors
54bad90
@aperezlebel
Use _check_sample_weight to check sample_weight
1430649
@aperezlebel
Fix error when some weights are zero
faf4fca
@aperezlebel
Use 2D histogram instead of two histograms
b97de6e
@aperezlebel
Increase n_samples in test_calibration_multiclass
1284b91 
The calibrator with custom regressor needs more samples
@aperezlebel
Increase n_samples in data and in test_calibration
c13d088 
Calibration using a custom regressor needs more sample to reduce brier score
@aperezlebel
Remove useless raise
a4492b5
@aperezlebel
Rename scores
0f5b4a8
@aperezlebel
Add docstring
852572b
@aperezlebel
Use default for HistGradientBoosting in tests
b96ac0f
@ogrisel
Copy link
Member

ogrisel commented Dec 16, 2021

Thanks! By reading the PR, I realized that I made a mistake in the way I wrote #21280. What I had in mind it to allow to pass an arbitrary classifier with a predict_proba rather than an arbitrary regressor.

What you did with the binning reformulation is interesting but more complicated to what I had in mind. I edited #21280 to fix it. Sorry for the confusion.

@aperezlebel aperezlebel mentioned this pull request Dec 17, 2021
Draft
6 tasks
@aperezlebel
Copy link
Contributor Author

Following updates of issue #21280, I close this PR and created #22010 instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make it possible to pass an arbitrary probablistic classifier as method for CalibratedClassifierCV
2 participants
@aperezlebel @ogrisel