I'm not sure how this should be dealt with: either rely on lexical ordering and let the last label be the positive or expose the pos_label argument. Anyways, the error is misleading because the data is binary and the pos_label cannot be specified.

why can pos_label not be specified? shouldn't it be yes?

The greater-is-positive approach is used elsewhere, but is awkward (e.g.
hinge_loss iirc). And there's a PR to add pos_label here (#2616). Though I
suggest that in order to not make the mess of multiclass averages and
pos_label interaction that we consider something more like #2610, which
allows you to specify a set of labels to average over, such that binary is
no longer a special case (indeed, the lack of labels parameter in
multiclass roc_auc_score is already a bug, because it means a missing class
in one cv fold could give a wildly different macro-average).

On 7 January 2014 11:22, Andreas Mueller notifications@github.com wrote:

why can pos_label not be specified? shouldn't it be yes?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2723#issuecomment-31702170
.

related #2616

What I can understand is, roc_auc_score works for binary, 0, 1, does not understand 'yes' and 'no' as negative and positive class, which will make confusion for computation of true/false positive/negative, which to be treated what??

-----------try this instead ------This works-------
import numpy as np
np.random.seed(13)
#classes = np.array(['yes', 'no'])
#y = classes[np.random.randint(2, size=10)]
#t = classes[np.random.randint(2, size=10)]
y = np.random.randint(2, size=10)
t = np.random.randint(2, size=10)
from sklearn.metrics import roc_auc_score
from sklearn.metrics import accuracy_score
accuracy_score(t,y)
roc_auc_score(t, y)
-----------------end-----------------------------------------

We should not be able to provide pos_label to roc_auc_score: it is invariant to swapping classes as long as the score is similarly inverted (which I hope is tested but probably isn't...?). Rather we should make sure that roc_auc_score assumes that the greater of binary labels is the one for which the score is being reported.

@jnothman Just make sure that this will only work for binary y_true, not for multilabel-indicator y_true, right?

@jnothman
I'm still wondering how you want to solve the problem.
For example, if I get y_true = ['good','not-good', 'good', 'not-good'] and y_score = [0.9, 0.1, 0.2, 0.7], y_score represents the probability of 'good'. So you are expecting users to do something like roc_auc_score(y_true, 1-y_pred) or 1-roc_auc_score(y_true, y_pred) instead of roc_auc_score(y_true, y_pred, pos_label='good')? Thanks.

why does y_score represent the probability of good? If someone trains a LogisticRegression model with y=['good','not-good', 'good', 'not-good'], it will automatically map 'not-good' as the positive class. When the user provides the output of the LogisticRegression.predict_proba (or does so via the scorer) as y_score, a high value will correspond to a likely 'not-good' prediction.

@jnothman Thanks. I opened #9828 to address your solution. Please have a look when you have time :)