@Gael: the difference between the f1_score in multiclass settings is that whenever there is a binary classification problem, it will return only the positive f1 score by default. this will always return the average. clarified in the doc string now.

also we should work towards including all the metrics in the referenced doi under matthews_corrcoef. there are some generally useful ones for the multiclass case. we already have several of the mutual info metrics.

Maybe we should instead write separate functions for binary and multiclass f1 score such that calling the binary classification f1_score on multi-class input would raise a ValueError telling to use the multi class version instead? The binary classification could be named f1_score and the multi class version could be named either multiclass_f1_score or average_f1_score. WDYT?

Also I would rather avoid names such as avg_f1_score but rather use the explicit name average_f1_score.

@ogrisel: just a quick question. in the binary case the current f1_score returns the value for the positive label. that's purely convention right? there is no reason to think that the +1 label is any different from a -1 or a 0 label.

if that's the case, wouldn't it make sense to have a single function and always return np.average(f1, weights=support) and which would have pos_label=None. if pos_label is not None, then it would return np.average(f1[pos_label], weights=support). that would be a generic single function solution, but would change the api.

i'll change avg -> average. should corrcoef be changed to correlation_coefficient?

i'll change avg -> average. should corrcoef be changed to correlation_coefficient?

I don't think so: corrcoef is just a question of following numpy.

@satra: yes the choice of the positive label is a convention in binary classification. In information retrieval positive means "belong to the search results of the query" whereas all other document from the index are classified as "negative". Hence the unbalanced labeling between positive and negative.

As for the treatment of pos_label=None to tell apart the multiclass classification tasks from binary classification tasks, I would rather change change the existing method fbeta_score (and related methods such as precision_score and recall_score) to do:

y_true, y_pred = check_arrays(y_true, y_pred)

if labels is None:
    labels = unique_labels(y_true, y_pred)
else:
    labels = np.asarray(labels, dtype=np.int)
_, _, f, s = precision_recall_fscore_support(y_true, y_pred, beta=beta, labels=labels)
if pos_label is not None:
    if f.shape[0] != 2:
        raise ValueError("pos_label should be set to None for multiclass tasks got %d" % pos_label)
    if pos_label is not in labels:
        raise ValueError("pos_label=%d is not a valid label: %r" % pos_label, labels)
    pos_label_idx = list(labels).index(pos_label)
    return f[pos_label_idx]
else:
    return np.average(f, weights=s)

In that case we don't need to introduce a new function (we can reuse the existing ones). Of course we need to make it explicit that the binary classification tasks should provide an explicit pos_label otherwise the problem is treated as multiclass classification with support-weighted averaging of the scores.

@ogrisel: this sounds good to me. shall i post to the list to get some feedback before i go ahead and change?

Yes that would be great to ask for more comments.

This is great - it was actually on my todo list when I got some time!
I would also prefer ogrisel's method of doing both binary and average f1 in the same function.
If f.shape[0] > 2, should the value of pos_label just be ignored?

can't we simply do

np.average(f[labels], weights=s[labels])

where labels can be an array. we can do the appropriate checks and change pos_label to labels.

The default value being pos_label=None it's better to throw an exception when the user passes a none None value on a multiclass dataset rather than trying to silently fix the problem in the background without telling the caller that what was asked for is meaningless.

I didn't quite understand how the new metric for average f1 score differs from what we currently have (I'm a bit in a hurry so I guess I'm missing something). By the way, the way we average the f1 score (before this PR) is not standard. We need to add micro and macro averaging instead. See #83.

@mblondel: the difference with the new metric is that for even two classes it gives the weighted average, which is impossible to do with the current. i like the micro-macro-average idea. is there a reference i can look up?

there are a couple of tests in other sections that are failing. could somebody please review this to ensure that this is doing what was intended?

query about default behavior. so a bunch of tests are failing in test_sgd because currently we set defaults as:
pos_label=1, average=weighted, but if a multiclass value is sent it raises a ValueError. i don't want to touch the sgd tests, but the idea there would be to call f1_score with pos_label=None for multiclass input.

also along these lines the cross_validation routines are failing. is there a way to assign score_func=f1_score(pos_label=None)

Hum the cross validation stuff is very good remark.

Maybe we should split features and have explicit methods for multiclass and binaryclass cases after all. But also cross_val_score is restrictive in other cases for instance for using AUC as cross-validated score I have to subclass the estimator to override the estimator.score function because the cross_val_score API is not flexible enough.

Otherwise maybe as a short term solution we could use pos_label=None as default?

About the cross_val_score API issues, I had a chat with @agramfort in the bus from the Alhambra to Sierra Nevada and we came up to the same conclusion that we probably need a parameterized Scorer API in the scikit that would be suitable for the following use cases:

• cross validation with explicit integer predictions, transforms or AUC based on decision functions or
• grid search with cross validation (same as above)
• early stopping against a fixed validation set for online / large scale incremental estimators (SGD, MinibatchKMeans, MinibatchDictionaryLearning)

The gist would be to have the user provide one time constructor level parameters (like the averaging scheme for multiclass evaluation using the F1 score) or a fixed validation set to compute the score against that could be then treated as a curified evaluation function to be cold many times (on different cross val folds during grid search or every n_minibatches for early stopping).

We will need to think this idea further: I think we need a new experimental branch to devise the exact API by implementing the afore mentioned use cases.

I would also like to keep the *_score functions in parallel so as to make it easier for the user to compute one time evaluation of an estimator (e.g. in a IPython session). The scorer implementations could be based on those function implementations when it makes sense.

WDYT @satra and @mblondel?

@ogrisel

1. first i changed pos_label to None - this is an api change so it should be made ample clear to folks.
2. yes we need a new api for scoring in relation to any functionality that calls the scoring object (also add permutation test to your list above). something along the following lines?

class Scorer(object):
    def __init__(self, score_func, **score_func_args):
        self._score_func = score_func
        self._score_func_args = score_func_args
        #TODO: validate that score_func_args can be applied to score_func

    def score(self, y_true, y_pred):
        return self._score_func(y_true, y_pred, **self._score_func_args)

I presume a similar thing for loss_func.

@ogrisel could you clarify a little more on the validation set and the transforms aspect? seems like that should be different from the scoring api. perhaps you are thinking that score api can itself take a pipeline object?

@satra sorry for the late reply. I think @GaelVaroquaux has an alternative suggestion based on functools to address this issue.

@GaelVaroquaux could you please elaborate on your functools suggestions

I pushed a couple of commits to my metrics branch (https://github.com/mblondel/scikit-learn/tree/metrics). First, I reverted pos_label back to 1: pos_label is just to specify a label encoding and I think it's confusing to use it as a switch to enable / disable a behavior. Also now people who want the values of the two classes in the binary case (a feature that @satra wanted) need to pass an explicit average=False or average=None. I think my solution is simpler. The tests are passing. I hope I didn't misunderstand some of the things you were trying to do.

I think we can now merge @satra 's PR with my commits to master and work on the scorer API in another PR. This PR has been pending for too long (and I need averaged metrics for my research :p)

In the grid-search module, there are plenty of calls to predict but for AUC we need to use either decision_function or predict_proba. Currently, score_func must be a callable. I suggest we also allow strings so we can for example do GridSearchCV(score_func="auc"). This way we can easily switch to decision_function or predict_proba where it's necessary.

Sounds like a reasonable plan to advance incrementally on this issue.

@satra can you merge @mblondel's change into your branch so that we can merge this PR?

It could be convenient to have alias functions macro_precision, micro_precision, ..., micro_f1_score. Pro: useful for cross-validation. Con: 6 more functions. What do you think?

@mblondel i think allowing strings is what would solve the problem but @GaelVaroquaux had some alternate thoughts (?)

also i have no issue with adding helper functions. in fact what got this started was an additional helper function called average_f1_score!

@ogrisel : merged @mblondel's branch.

I am -0 for adding more public functions for these metrics.

If we add new public functions they should be suffixed with "_score" when positive and "_loss" when negative to be consistent with the rest of the module. In this case (precision and recall) they are both positive.

@satra : if you don't have any further change, please press the green button :)

We can add the helper functions later in master if we need them (for now, I put a few helpers in my user-land code).