task list:

• implement fix
• add appropriate tests (regression)
• successful tests run / docs etc

Reference Issues/PRs

Fixes #30713

What does this implement/fix? Explain your changes.

This implementation fixes an exception that is being thrown in d2_log_loss_score function, when the user passes a labels variable (hence labels are not inferred) and the vector y_true does not contain one or more labels.

An example that reproduces the error:

from sklearn.metrics import d2_log_loss_score

y_true = [0, 1, 1]
y_pred = [[1, 0, 0], [1, 0, 0], [1, 0, 0]]
labels = [0, 1, 2]

d2_log_loss_score(y_true, y_pred, labels=labels)

Breakdown:

• inside d2_log_loss_score, two log losses are calculated, the log loss of the fitted model aka numerator and the log loss of the null model aka denominator.
• both losses are calculated with the log_loss function in which the labels variable is passed down to.
• For numerator, y_pred is equal to the y_pred passed by the user. y_pred has already the "correct" dimension (n, k=3) with respect to the number of labels -- 3 labels given, 3 columns are present in y_pred:

    # log loss of the fitted model
    numerator = log_loss(
        y_true=y_true,
        y_pred=y_pred,
        normalize=False,
        sample_weight=sample_weight,
        labels=labels,
    )

• The issue arises before the computation of denominator, where y_true is supposed to be transformed into a 2D array of dimension (n, k):

    _, y_value_indices = np.unique(y_true, return_inverse=True)
    counts = np.bincount(y_value_indices, weights=weights)
    y_prob = counts / weights.sum()
    y_pred_null = np.tile(y_prob, (len(y_true), 1))

    # log loss of the null model
    denominator = log_loss(
        y_true=y_true,
        y_pred=y_pred_null,
        normalize=False,
        sample_weight=sample_weight,
        labels=labels,
    )

• y_value_indices = [0, 1, 1] because the unique elements in y_true are 2: [0,1]
• counts = [1, 2] because there are two bins in y_value_indices. At this point counts should be [1,2,0] but labels are not taken into consideration. This is also why the error does not appear when all labels are present in y_true.
• y_prob has the same size as counts, i.e. 2
• y_pred_null is thefore an array of (n, 2) instead of (n, 3)
• exception is thrown inside log_loss because labels has size 3 and y_pred has 2 columns

Change explanation:

• Take into account the labels array (if passed) for the computation of counts

Any other comments?