Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
3420a3d
wip
lucyleeow Jun 4, 2020
17dd94a
calib curve
lucyleeow Jun 4, 2020
2f34245
lint
lucyleeow Jun 4, 2020
56eda05
avoid circular import
lucyleeow Jun 4, 2020
f98169e
circ import
lucyleeow Jun 4, 2020
5d3ea22
circ import
lucyleeow Jun 4, 2020
549106b
add tests
lucyleeow Jun 4, 2020
12b6704
suggestions
lucyleeow Jun 4, 2020
102f826
lint, fix tests
lucyleeow Jun 4, 2020
cbc73ab
lint
lucyleeow Jun 4, 2020
8427385
add brier score
lucyleeow Jun 5, 2020
7beb99d
update example
lucyleeow Jun 5, 2020
e0fcff6
amend brier label
lucyleeow Jun 5, 2020
32efb16
typo
lucyleeow Jun 5, 2020
58caff9
brier var name
lucyleeow Jun 5, 2020
92a4387
lint
lucyleeow Jun 5, 2020
3267309
update comp cal
lucyleeow Jun 5, 2020
5f3660e
lint, add ref link
lucyleeow Jun 5, 2020
14e52ac
minor formatting
lucyleeow Jun 5, 2020
ceba3de
plt
lucyleeow Jun 5, 2020
1196d2a
add y prob att
lucyleeow Jun 5, 2020
dec7a6d
fix examples
lucyleeow Jun 5, 2020
4a34cf5
clean plots
lucyleeow Jun 5, 2020
07f3737
clean plots
lucyleeow Jun 5, 2020
2029fb4
explain scores, fix cal curv
lucyleeow Jun 5, 2020
622a14c
enable decision function
lucyleeow Jun 5, 2020
f1c5a03
lint
lucyleeow Jun 5, 2020
e202962
clarify logistic in example, minor wording
lucyleeow Jun 5, 2020
6ee5ec3
lint
lucyleeow Jun 5, 2020
12fcb92
fix test
lucyleeow Jun 5, 2020
b21355b
fix test
lucyleeow Jun 6, 2020
3394ee1
mv files to calibration
lucyleeow Jun 6, 2020
aa82f0d
revert import adj
lucyleeow Jun 6, 2020
e378286
remove from init
lucyleeow Jun 6, 2020
b07432f
fix import in ex, format docstring
lucyleeow Jun 7, 2020
8d172c2
fix import, param order
lucyleeow Jun 7, 2020
0393964
add example in docstring
lucyleeow Jun 7, 2020
08a1d22
use sep hist in ex
lucyleeow Jun 7, 2020
e9f2443
add grid
lucyleeow Jun 7, 2020
ee44d5a
fix color
lucyleeow Jun 7, 2020
ef240dd
fix subplot
lucyleeow Jun 7, 2020
018712f
plt
lucyleeow Jun 7, 2020
1447951
amend hist both ex
lucyleeow Jun 8, 2020
2093260
typo
lucyleeow Jun 8, 2020
cd48cec
wording of ex
lucyleeow Jun 8, 2020
03c2e17
fix rst
lucyleeow Jun 8, 2020
464c09c
fix docstrings
lucyleeow Jun 8, 2020
df0f317
amend docstring
lucyleeow Jun 8, 2020
d57fb91
mention in vis and cal docs
lucyleeow Jun 8, 2020
e93c26d
[empty] CI
lucyleeow Jun 8, 2020
28fb7d6
merge master
lucyleeow Jun 9, 2020
d663932
Merge branch 'master' into IS/8425
ogrisel Jun 10, 2020
5aec36b
use gridspec plot compare
lucyleeow Jun 12, 2020
7ef048b
Merge branch 'IS/8425' of github.com:lucyleeow/scikit-learn into IS/8425
lucyleeow Jun 12, 2020
3f5addc
use gridspec, df, suggestions
lucyleeow Jun 12, 2020
d5c04fb
wording
lucyleeow Jun 12, 2020
30230b5
typo
lucyleeow Jun 12, 2020
45e3658
typo
lucyleeow Jun 12, 2020
a2c433e
Merge branch 'master' into IS/8425
lucyleeow Jun 15, 2020
5b16224
wip
lucyleeow Jun 15, 2020
2dc7b8b
remove decision function
lucyleeow Jun 15, 2020
eaaf372
merge master
lucyleeow Jul 2, 2020
aeb9c61
remove brier score
lucyleeow Jul 2, 2020
b0342e2
amend examples to scale df
lucyleeow Jul 2, 2020
e8d0b91
lint
lucyleeow Jul 2, 2020
9f5e853
remove brier from tests
lucyleeow Jul 2, 2020
3dda24a
lint
lucyleeow Jul 2, 2020
70af4e8
merge master
lucyleeow Jul 28, 2020
fdc2826
Merge branch 'master' into IS/8425
lucyleeow Jul 29, 2020
7de3ec0
wip
lucyleeow Jul 30, 2020
7ed7d01
add logloss, fix df
lucyleeow Jul 31, 2020
cacf02b
suggestions
lucyleeow Jul 31, 2020
b41f969
lint
lucyleeow Jul 31, 2020
f4b7525
lint
lucyleeow Jul 31, 2020
75d6758
revert call pred first
lucyleeow Jul 31, 2020
1f2102e
fix scores
lucyleeow Jul 31, 2020
c71f6cf
fix y proba dim svc
lucyleeow Jul 31, 2020
1563b9f
fix naive svc for proba
lucyleeow Jul 31, 2020
3d02f57
user guide, est without pred proba
lucyleeow Jul 31, 2020
9e8e528
add reliability diagram
lucyleeow Aug 12, 2020
05e5059
suggestions 1
lucyleeow Aug 17, 2020
4294df2
add roc auc
lucyleeow Aug 19, 2020
faacbb1
fix docstring
lucyleeow Aug 19, 2020
13486d1
fix error msg in test
lucyleeow Aug 19, 2020
7ee101d
naive bayes
lucyleeow Aug 20, 2020
6961cd2
suggestions
lucyleeow Sep 10, 2020
1225731
Merge branch 'master' into IS/8425
lucyleeow Sep 16, 2020
8e81ce4
add svn details
lucyleeow Sep 16, 2020
c1fb248
merge master
lucyleeow Oct 9, 2020
7c535e1
change to 2 methods
lucyleeow Oct 10, 2020
1c0c1a7
amend tests
lucyleeow Oct 11, 2020
7c52cee
merge main
lucyleeow Jan 26, 2021
91f0c79
remove docstring inject
lucyleeow Jan 26, 2021
edfc917
add example and see also docstring
lucyleeow Jan 26, 2021
dad4e1b
mv whats new to 1.0
lucyleeow Jan 26, 2021
c0908ed
update docs
lucyleeow Jan 26, 2021
433d7e4
lint
lucyleeow Jan 26, 2021
75dcb3e
lint
lucyleeow Jan 26, 2021
48aae81
lint
lucyleeow Jan 26, 2021
5596276
amend tests
lucyleeow Jan 26, 2021
396087d
fix params
lucyleeow Jan 26, 2021
3eaf27c
dont param mult calls test
lucyleeow Jan 26, 2021
5b70f9e
rm plot cal curve import
lucyleeow Jan 26, 2021
f9d0fea
test both meth in mult call
lucyleeow Jan 26, 2021
83ef0d6
add import
lucyleeow Jan 26, 2021
75a22d5
docstring format
lucyleeow Jan 26, 2021
f657a57
rm old estimator name attr
lucyleeow Jan 26, 2021
cd88a9a
fix tests
lucyleeow Jan 26, 2021
7780131
lint
lucyleeow Jan 26, 2021
eca1b97
fix docstring
lucyleeow Jan 26, 2021
de9e9ad
add name docstring
lucyleeow Jan 26, 2021
f00e624
rm name from disp class
lucyleeow Jan 28, 2021
8947cc4
fix name arg
lucyleeow Jan 29, 2021
d4f9f1d
add name arg to class, add test
lucyleeow Jan 29, 2021
37b05b5
fix caldisp docstring
lucyleeow Jan 29, 2021
6cb9fcb
add pyplot test
lucyleeow Jan 29, 2021
3aa9831
[epmty]
lucyleeow Jan 29, 2021
80976c7
merge master
May 24, 2021
b76380c
[empty]
May 25, 2021
33597c3
show display in CI
May 25, 2021
8566086
revert
May 29, 2021
1616900
Merge branch 'main' into IS/8425
May 29, 2021
405872d
try skip doctest
Jun 6, 2021
078279c
merge master
Jun 6, 2021
c2cf45f
try removing examples entirely
Jun 9, 2021
5126413
add examples back
Jun 11, 2021
44dc52b
merge main
Jun 19, 2021
8216333
add imports
Jun 19, 2021
062457c
black
Jun 19, 2021
b3d511f
move imports to cells
Jul 16, 2021
731c1e7
add reviewer suggestions
Jul 19, 2021
80a2677
lint
Jul 19, 2021
1de06b1
lint
Jul 19, 2021
f30b7bc
lint
Jul 19, 2021
acccd48
typo
Jul 19, 2021
a649a50
Merge branch 'main' into IS/8425
Jul 19, 2021
39b5cb7
add to example
Jul 19, 2021
89e652e
fix test error msg
Jul 21, 2021
6ea9950
fix docstring
Jul 21, 2021
0c25cf6
fix test
Jul 21, 2021
1367022
typo
Jul 21, 2021
8dc99a8
fix docstrings
Jul 21, 2021
e3797b6
typo in test
Jul 21, 2021
ecd47b9
fix error msg
Jul 22, 2021
e18419d
fix error msg
Jul 22, 2021
244a8fd
Merge remote-tracking branch 'origin/main' into IS/8425
ogrisel Aug 30, 2021
99f38ba
Merge branch 'main' into IS/8425
Aug 31, 2021
ec4764d
Merge branch 'IS/8425' of github.com:lucyleeow/scikit-learn into IS/8425
Aug 31, 2021
81d43d6
suggestions
Aug 31, 2021
66851d9
update msg
Aug 31, 2021
b25f03b
update msg
Aug 31, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions doc/modules/calibration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,25 @@ approximately 80% actually belong to the positive class.
Calibration curves
------------------

The following plot compares how well the probabilistic predictions of
different classifiers are calibrated, using :func:`calibration_curve`.
Calibration curves (also known as reliability diagrams) compare how well the
probabilistic predictions of a binary classifier are calibrated. It plots
the true frequency of the positive label against its predicted probability,
for binned predictions.
The x axis represents the average predicted probability in each bin. The
y axis is the *fraction of positives*, i.e. the proportion of samples whose
class is the positive class (in each bin).
class is the positive class (in each bin). The top calibration curve plot
is created with :func:`CalibrationDisplay.from_estimators`, which uses
:func:`calibration_curve` to calculate the per bin average predicted
probabilities and fraction of positives.
:func:`CalibrationDisplay.from_estimator`
takes as input a fitted classifier, which is used to calculate the predicted
probabilities. The classifier thus must have :term:`predict_proba` method. For
the few classifiers that do not have a :term:`predict_proba` method, it is
possible to use :class:`CalibratedClassifierCV` to calibrate the classifier
outputs to probabilities.

The bottom histogram gives some insight into the behavior of each classifier
by showing the number of samples in each predicted probability bin.

.. figure:: ../auto_examples/calibration/images/sphx_glr_plot_compare_calibration_001.png
:target: ../auto_examples/calibration/plot_compare_calibration.html
Expand Down Expand Up @@ -161,6 +175,8 @@ mean a better calibrated model.
:class:`CalibratedClassifierCV` supports the use of two 'calibration'
regressors: 'sigmoid' and 'isotonic'.

.. _sigmoid_regressor:

Sigmoid
^^^^^^^

Expand Down
2 changes: 1 addition & 1 deletion doc/modules/classes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1123,7 +1123,7 @@ See the :ref:`visualizations` section of the user guide for further details.
metrics.DetCurveDisplay
metrics.PrecisionRecallDisplay
metrics.RocCurveDisplay

calibration.CalibrationDisplay

.. _mixture_ref:

Expand Down
2 changes: 2 additions & 0 deletions doc/visualizations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ values of the curves.
* :ref:`sphx_glr_auto_examples_miscellaneous_plot_roc_curve_visualization_api.py`
* :ref:`sphx_glr_auto_examples_miscellaneous_plot_partial_dependence_visualization_api.py`
* :ref:`sphx_glr_auto_examples_miscellaneous_plot_display_object_visualization.py`
* :ref:`sphx_glr_auto_examples_calibration_plot_compare_calibration.py`

Available Plotting Utilities
============================
Expand All @@ -90,6 +91,7 @@ Display Objects

.. autosummary::

calibration.CalibrationDisplay
inspection.PartialDependenceDisplay
metrics.ConfusionMatrixDisplay
metrics.DetCurveDisplay
Expand Down
3 changes: 3 additions & 0 deletions doc/whats_new/v1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,9 @@ Changelog
:class:`calibration.CalibratedClassifierCV` can now properly be used on
prefitted pipelines. :pr:`19641` by :user:`Alek Lefebvre <AlekLefebvre>`.

- |Feature| :func:`calibration.CalibrationDisplay` added to plot
calibration curves. :pr:`17443` by :user:`Lucy Liu <lucyleeow>`.

- |Fix| Fixed an error when using a ::class:`ensemble.VotingClassifier`
as `base_estimator` in ::class:`calibration.CalibratedClassifierCV`.
:pr:`20087` by :user:`Clément Fauchereau <clement-f>`.
Expand Down
Loading