diff --git a/doc/modules/model_evaluation.rst b/doc/modules/model_evaluation.rst index 4d693b68cfb77..7d1722d5efaaf 100644 --- a/doc/modules/model_evaluation.rst +++ b/doc/modules/model_evaluation.rst @@ -1033,6 +1033,60 @@ set [0,1] has an error: :: for an example of zero one loss usage to perform recursive feature elimination with cross-validation. +.. _brier_score_loss: + +Brier score loss +---------------- + +The :func:`brier_score_loss` function computes the +`Brier score `_ +for binary classes. Quoting Wikipedia: + + + "The Brier score is a proper score function that measures the accuracy of + probabilistic predictions. It is applicable to tasks in which predictions + must assign probabilities to a set of mutually exclusive discrete outcomes." + +This function returns a score of the mean square difference between the actual outcome and +the predicted probability of the possible outcome. The actual outcome has to be 1 or 0 +(true or false), while the predicted probability of the actual outcome can be +a value between 0 and 1. +The brier score loss is also between 0 to 1 and the lower the score (the mean square difference is smaller), +the more accurate the prediction is. It can be thought of as a measure of the "calibration" +of a set of probabilistic predictions. + +.. math:: + + BS = \frac{1}{N} \sum_{t=1}^{N}(f_t - o_t)^2 + +where : :math:`N` is the total number of predictions, :math:`f_t` is the predicted +probablity of the actual outcome :math:`o_t` + + >>> import numpy as np + >>> from sklearn.metrics import brier_score_loss + >>> y_true = np.array([0, 1, 1, 0]) + >>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"]) + >>> y_prob = np.array([0.1, 0.9, 0.8, 0.4]) + >>> y_pred = np.array([0, 1, 1, 0]) + >>> brier_score_loss(y_true, y_prob) + 0.055 + >>> brier_score_loss(y_true, 1-y_prob, pos_label=0) + 0.055 + >>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham") + 0.055 + >>> brier_score_loss(y_true, y_prob > 0.5) + 0.0 + + +.. topic:: Example: + + * See :ref:`example_calibration_plot_calibration.py` + for an example of Brier score loss usage to perform probability calibration of classifiers. + +.. topic:: References: + + * G. Brier, `Verification of forecasts expressed in terms of probability + `_, Monthly weather review 78.1 (1950) .. _multilabel_ranking_metrics: