In my previous comment I said:

If needed, we should replace needs_threshold=True by needs_proba=True but I do not see why this would be the case for ROC AUC.

But are you sure we really need to use proba for multiclass ROC AUC scores? Why couldn't we use unnormalized class assignment scores also in the multiclass case?

Replying to myself: when calling _multiclass_roc_auc_score on un-normalized decision thresholds, one gets the following message:

ValueError: Target scores need to be probabilities for multiclass roc_auc, i.e. they should sum up to 1.0 over classes

So indeed the fix implemented in this PR is correct.

@qinhanmin2014 any further comment?

Moved the mentioning of the scores to the entry about the introduction of the multiclass roc_auc metric.

I think there's still an annoying issue here.
In roc_auc_score, we have

if y_type == "multiclass" or (y_type == "binary" and
                              y_score.ndim == 2 and
                              y_score.shape[1] > 2):
    # multiclass case
else:
    # binary case and multilabel indicator case

which means that when y_type is binary, this can still be a multiclass problem, but in _ProbaScorer

if y_type == "binary":
    if y_pred.shape[1] == 2:
        y_pred = y_pred[:, 1]
    else:
        raise ValueError(...)

So we'll get a ValueError.

One possible solution is to remove input validation in Scorer. We can do input validation in the function where we calculate the score.

For reference, the weird condition on roc_auc_score is to support the following use case:

import numpy as np
from sklearn.metrics import roc_auc_score

y_true = np.array([0, 1, 0, 1])

y_score = np.array([[0.1 , 0.8 , 0.1 ],
                    [0.3 , 0.4 , 0.3 ],
                    [0.35, 0.5 , 0.15],
                    [0.  , 0.2 , 0.8 ]])

roc_auc_score(y_true, y_score, labels=[0, 1, 2], multi_class='ovo')

Without the labels this will fail. I have a feeling we need to think of a better API for this. (#12385)

Even if we remove the restriction on _ProbaScorer a user would need to create their own scorer for this to work:

scorer = make_scorer(roc_auc_score, multi_class='ovo',
                        labels=[0, 1, 2], needs_proba=True)
X, y = make_classification(n_classes=3, n_informative=3, n_samples=20,
                            random_state=0)
lr = LogisticRegression(multi_class="multinomial").fit(X, y)
scorer(lr, X, y == 0)

@thomasjpfan When defining built-in scorers, we often use default value of the parameters. If users do not want to use default value of the parameters, they'll need to define a scorer themselves through make_scorer. I think it's reasonable.

In multiclass roc_auc_score, we infer the labels by default. When y_true do not contain all the labels, it's impossible to imfer the labels, so it's reasonble to require users to define a scorer themselves.

Current issue is that when y_true only contains two labels, users can't define a scorer themselves because of the input validation in _ProbaScorer, so I think we need to remove that.

Updated the check in _ProbaScorer to check if y_pred looks multiclass before raising the error.

Edit: If it does not look multiclass, then it raises the ValueError.

thank, though I still think that multiclass roc_auc can accept the output of decision_function. Not all estimators in scikit-learn has predict_proba :)