Skip to content

[MRG+1] Add Brier_score_loss to model validation doc #6841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

[MRG+1] Add Brier_score_loss to model validation doc #6841

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
20ff083
Add Brier_score_loss to model validation doc
moutai May 29, 2016
46f8baa
Add more precise explanation of the metric, add two references and fi…
moutai Jun 4, 2016
9a5eb74
Remove mention of multi-class to match the supported implementation, …
moutai Jun 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions doc/modules/model_evaluation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1033,6 +1033,60 @@ set [0,1] has an error: ::
for an example of zero one loss usage to perform recursive feature
elimination with cross-validation.

.. _brier_score_loss:

Brier score loss
----------------

The :func:`brier_score_loss` function computes the
`Brier score <https://en.wikipedia.org/wiki/Brier_score>`_
for binary classes. Quoting Wikipedia:


"The Brier score is a proper score function that measures the accuracy of
probabilistic predictions. It is applicable to tasks in which predictions
must assign probabilities to a set of mutually exclusive discrete outcomes."

This function returns a score of the mean square difference between the actual outcome and
the predicted probability of the possible outcome. The actual outcome has to be 1 or 0
(true or false), while the predicted probability of the actual outcome can be
a value between 0 and 1.
The brier score loss is also between 0 to 1 and the lower the score (the mean square difference is smaller),
the more accurate the prediction is. It can be thought of as a measure of the "calibration"
of a set of probabilistic predictions.

Copy link
Member

@ogrisel ogrisel Jun 3, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph need to give some hint on the "why" of this score function. For instance:

The Brier score was proposed by Glenn W. Brier in 1950. It can be thought of as a measure of the "calibration" of a set of probabilistic predictions.

And please also add a reference to the original paper:

http://docs.lib.noaa.gov/rescue/mwr/078/mwr-078-01-0001.pdf

.. math::

BS = \frac{1}{N} \sum_{t=1}^{N}(f_t - o_t)^2

where : :math:`N` is the total number of predictions, :math:`f_t` is the predicted
probablity of the actual outcome :math:`o_t`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a detail, but I prefer if, before a code block, you add "::", which is the explicit way of defining a code-block, rather than relying on indentation, which is fragile.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I will do that change and squash/merge.


>>> import numpy as np
>>> from sklearn.metrics import brier_score_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
>>> y_prob = np.array([0.1, 0.9, 0.8, 0.4])
>>> y_pred = np.array([0, 1, 1, 0])
>>> brier_score_loss(y_true, y_prob)
0.055
>>> brier_score_loss(y_true, 1-y_prob, pos_label=0)
0.055
>>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
0.055
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be 0.0 as the prediction is perfect once thresholded. This means that the doctests are not executed for doc/modules/model_evaluation.rst. Can you please this and fix the Makefile to make sure that those docstests are run properly?

>>> brier_score_loss(y_true, y_prob > 0.5)
0.0


.. topic:: Example:

* See :ref:`example_calibration_plot_calibration.py`
for an example of Brier score loss usage to perform probability calibration of classifiers.

.. topic:: References:

* G. Brier, `Verification of forecasts expressed in terms of probability
<http://docs.lib.noaa.gov/rescue/mwr/078/mwr-078-01-0001.pdf>`_, Monthly weather review 78.1 (1950)

.. _multilabel_ranking_metrics:

Expand Down