Add Brier_score_loss to model validation doc

moutai · ogrisel · commit c6a457fe9953 · 2016-06-16T10:57:16.000+02:00
diff --git a/doc/modules/model_evaluation.rst b/doc/modules/model_evaluation.rst
@@ -1033,6 +1033,65 @@ set [0,1] has an error: ::
     for an example of zero one loss usage to perform recursive feature
     elimination with cross-validation.
 
+.. _brier_score_loss:
+
+Brier score loss
+----------------
+
+The :func:`brier_score_loss` function computes the
+`Brier score <https://en.wikipedia.org/wiki/Brier_score>`_
+for binary classes. Quoting Wikipedia:
+
+    "The Brier score is a proper score function that measures the accuracy of
+    probabilistic predictions. It is applicable to tasks in which predictions
+    must assign probabilities to a set of mutually exclusive discrete outcomes."
+
+This function returns a score of the mean square difference between the actual
+outcome and the predicted probability of the possible outcome. The actual
+outcome has to be 1 or 0 (true or false), while the predicted probability of
+the actual outcome can be a value between 0 and 1.
+
+The brier score loss is also between 0 to 1 and the lower the score (the mean
+square difference is smaller), the more accurate the prediction is. It can be
+thought of as a measure of the "calibration" of a set of probabilistic
+predictions.
+
+.. math::
+
+   BS = \frac{1}{N} \sum_{t=1}^{N}(f_t - o_t)^2
+
+where : :math:`N` is the total number of predictions, :math:`f_t` is the
+predicted probablity of the actual outcome :math:`o_t`.
+
+Here is a small example of usage of this function:::
+
+    >>> import numpy as np
+    >>> from sklearn.metrics import brier_score_loss
+    >>> y_true = np.array([0, 1, 1, 0])
+    >>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
+    >>> y_prob = np.array([0.1, 0.9, 0.8, 0.4])
+    >>> y_pred = np.array([0, 1, 1, 0])
+    >>> brier_score_loss(y_true, y_prob)
+    0.055
+    >>> brier_score_loss(y_true, 1-y_prob, pos_label=0)
+    0.055
+    >>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
+    0.055
+    >>> brier_score_loss(y_true, y_prob > 0.5)
+    0.0
+
+
+.. topic:: Example:
+
+  * See :ref:`example_calibration_plot_calibration.py`
+    for an example of Brier score loss usage to perform probability
+    calibration of classifiers.
+
+.. topic:: References:
+
+  * G. Brier, `Verification of forecasts expressed in terms of probability
+    <http://docs.lib.noaa.gov/rescue/mwr/078/mwr-078-01-0001.pdf>`_,
+    Monthly weather review 78.1 (1950)
 
 .. _multilabel_ranking_metrics:
 
