Skip to content

[MRG+2] referred reliability diagrams and added citations #8527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 5, 2017
Merged

[MRG+2] referred reliability diagrams and added citations #8527

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
20f9794
referred reliability diagrams and added citations
mehak-sachdeva Mar 4, 2017
ab18cea
formatting edits
mehak-sachdeva Mar 4, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions doc/modules/calibration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@ with different biases per method:
ensemble away from 0. We observe this effect most strongly with random
forests because the base-level trees trained with random forests have
relatively high variance due to feature subseting." As a result, the
calibration curve shows a characteristic sigmoid shape, indicating that the
classifier could trust its "intuition" more and return probabilties closer
to 0 or 1 typically.
calibration curve also referred to as the reliability diagram (Wilks 1995[5]) shows a
characteristic sigmoid shape, indicating that the classifier could trust its
"intuition" more and return probabilties closer to 0 or 1 typically.

.. currentmodule:: sklearn.svm

Expand Down Expand Up @@ -118,13 +118,14 @@ is evaluated with Brier score :func:`brier_score_loss`, reported in the legend
:align: center

One can observe here that logistic regression is well calibrated as its curve is
nearly diagonal. Linear SVC's calibration curve has a sigmoid curve, which is
typical for an under-confident classifier. In the case of LinearSVC, this is
caused by the margin property of the hinge loss, which lets the model focus on
hard samples that are close to the decision boundary (the support vectors). Both
kinds of calibration can fix this issue and yield nearly identical results.
The next figure shows the calibration curve of Gaussian naive Bayes on
the same data, with both kinds of calibration and also without calibration.
nearly diagonal. Linear SVC's calibration curve or reliability diagram has a
sigmoid curve, which is typical for an under-confident classifier. In the case of
LinearSVC, this is caused by the margin property of the hinge loss, which lets
the model focus on hard samples that are close to the decision boundary
(the support vectors). Both kinds of calibration can fix this issue and yield
nearly identical results. The next figure shows the calibration curve of
Gaussian naive Bayes on the same data, with both kinds of calibration and also
without calibration.

.. figure:: ../auto_examples/calibration/images/sphx_glr_plot_calibration_curve_001.png
:target: ../auto_examples/calibration/plot_calibration_curve.html
Expand Down Expand Up @@ -200,3 +201,7 @@ a similar decrease in log-loss.

.. [4] Predicting Good Probabilities with Supervised Learning,
A. Niculescu-Mizil & R. Caruana, ICML 2005

.. [5] On the combination of forecast probabilities for
consecutive precipitation periods. Wea. Forecasting, 5, 640–
650., Wilks, D. S., 1990a
2 changes: 2 additions & 0 deletions sklearn/calibration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,8 @@ def predict(self, T):
def calibration_curve(y_true, y_prob, normalize=False, n_bins=5):
"""Compute true and predicted probabilities for a calibration curve.

Calibration curves may also be referred to as reliability diagrams.

Read more in the :ref:`User Guide <calibration>`.

Parameters
Expand Down