I do not understand this. In both cases it returns the same message.

Also, what happens when support_multi_class and y_pred.shape[1] != 1?

It was what was intended in the past: https://github.com/scikit-learn/scikit-learn/pull/18114/files#diff-e907207584273858caf088b432e38d04L243-L247

Also, what happens when support_multi_class and y_pred.shape[1] != 1

It is a case where y_true is encoded as binary but that the y_pred` is multi-class. It is a case supported by ROC-AUC apparently and I needed to keep it like this for backward compatibility.

I am not sure that there is a better way of inferring the exact encoding in this case.

I added an example to be more specific. I think that we should investigate if we can pass labels to type_of_target which could be an optional argument given when we are using it in the metrics.

I still do not understand this condition. This comment refers to the metric function outside of the scorer API.

I assume that this is related to to roc_auc_ovo_scorer which is a _ProbaScorer instance and this condition is about raising a ValueError when y_pred.shape[1] == 1 for some reason but I really do not see how this relates to the example you give here as y_pred has 3 columns in this example so it does not match the case of the condition.

I think this is trying to keep the logic from:

which I added in #15274. This was added because we can have a binary y_true with an estimator trained on > 2 classes.

y_pred.shape[1] == 1 was use to mean y_pred came from a classifier with only one class. The check for the shape of y_pred was added here: 94db3d9

But in scikit-learn we cannot have a classifier train with a single class, isn't it?

It is strange, but it can happen:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_blobs
import numpy as np

X, y = make_blobs(random_state=0, centers=2)
clf = DecisionTreeClassifier()
clf.fit(X, np.zeros_like(y))

clf.classes_
# array([0])

Uhm. Then, I am confused with check_classifiers_one_label :) I have to check. Anyway, I think that the last change make everything more explicit.

Oh I see a must_pass in the raises. It would be much more explicit to have tag for that I think: accept_single_label for instance.