Indeed, I can reproduce, looks like an oversight, but more investigation is needed.

Thanks for the report @aperezlebel. Are you interested in opening on a PR yourself?

I would be, but I am afraid I won't have time in the near future.

Hello, I think I have identified the issue here:

• inside d2_log_loss_score, two log losses are calculated, the log loss of the fitted model aka numerator and the log loss of the null model aka denominator.
• both losses are calculated with the log_loss function in which the labels variable is passed down to.
• For numerator, y_pred is equal to the y_pred passed by the user. y_pred has already the "correct" dimension (n, k=3) with respect to the number of labels -- 3 labels given, 3 columns are present in y_pred.
• The issue arises in the computation of denominator, where y_true is supposed to be transformed into a 2D array of dimension (n, k). I will try to break it down below:

   # >>> labels = [0, 1, 2]
   # >>> y_true = [0, 1, 1]

    _, y_value_indices = np.unique(y_true, return_inverse=True)
    # >>> y_value_indices = [0, 1, 1]
   
    counts = np.bincount(y_value_indices, weights=weights)
    # >>> counts = [1, 2] 
    # At this point counts should be [1, 2, 0] but labels are not taken into consideration
    # this is also why the error does not appear when all labels are present in y_true

    y_prob = counts / weights.sum()
    # y_prob has the same size as counts, i.e. 2

    y_pred_null = np.tile(y_prob, (len(y_true), 1))
    # y_pred_null is now an array of (n, 2) instead of (n, 3)

    # log loss of the null model
    denominator = log_loss(
        y_true=y_true,
        y_pred=y_pred_null,
        normalize=False,
        sample_weight=sample_weight,
        labels=labels,
    )
    # exception is thrown inside log_loss because labels has size 3 and y_pred has 2 columns

I think that computing counts by appropriately taking labels into consideration will fix the issue.

@ogrisel I will open a PR myself within the next days. Excuse me for my long response and if I did not follow protocol. This is my very first attempt at contributing to open source and I am still familiarizing with the workflow