For edge cases, I consider to use the strategy described in this question and answer

Thanks. We mark such pull requests WIP (work in progress).

Whenever you think, that your work is ready to be reviewed. You can prefix your pull request title with the MRG tag (instead of WIP) and tell it to us in the pull request discussion.

MRG means ready to be reviewed.

I think the binary part is ready to be reviewed, but we may need to discuss how to implement multi-class cases. So in such case should I change WIP to MRG (when there are still works need to be done)?

Coverage decreased (-0.0%) when pulling e7160a5 on lazywei:balanced_accuracy_score into 1b2833a on scikit-learn:master.

Coverage decreased (-0.0%) when pulling 80f1048 on lazywei:balanced_accuracy_score into 1b2833a on scikit-learn:master.

As inspired by @larsmans in #3506 (comment), I introduced a new parameter neg_class to let user indicate which class should be regarded as negative, and treat others as positive.
Based on this, I think we can also support multi-lable cases by letting user specify such "negative class". For binary or multiclass, the neg_class can only be integer, while for multi-label, we can allow neg_class to be an indicator array, e.g. neg_class=[0, 1, 0, 1, 1]. Any thought?

Coverage decreased (-0.0%) when pulling bcd5963 on lazywei:balanced_accuracy_score into 1b2833a on scikit-learn:master.

It's fine to have only a binary implementation. Averaging is something there are functions for already.

But you do need tests before it's worthy of MRG. A test for sanity and tricky cases for this particular metric implementation, plus a presence in the common metrics tests.

neg_class has heretofore been represented in pos_label (see e.g. precision_recall_fscore_support), and I have proposed in #2610 to get rid of that parameter, in favour of making the include-ignore distinction a function of the labels parameter. In many cases neg_class (or neg_label) is easier to state, but in the case of P/R/F, labels also controls the ordering of the per-class results in a multiclass classification context. I would really like to see this feature be both expressive and consistent across metrics.

It is also easier to provide a good default for pos_label (i.e. 1) than for neg_label which is frequently 0 or -1.

For averaging, see _average_binary_score

neg_class has heretofore been represented in pos_label (see e.g. precision_recall_fscore_support), and I have proposed in #2610 to get rid of that parameter, in favour of making the include-ignore distinction a function of the labels parameter

I could change from neg_class to pos_label. However, I don't really understand the label parameter. It should be an array of all possible labels? Why we need that?
Do I need to implement both pos_label and label parameters, or I should only implement the latter as we are going to deprecated pos_label?

labels is used where average=None in precision_recall_fscore_support to
define an order for the array of per-label scores. Because our definition
of macro-average can include labels not seen in a particular sample, it
also enables those unseen labels to be accounted for.

I would like to see it also used to exclude one or more negative classes in
the multiclass case (like pos_label but with multiple positive labels),
i.e. to indicate the or all positive labels, such that the default setting
would be labels=1.

On 3 August 2014 02:00, Bert Chang notifications@github.com wrote:

neg_class has heretofore been represented in pos_label (see e.g.
precision_recall_fscore_support), and I have proposed in #2610
#2610 to get rid of
that parameter, in favour of making the include-ignore distinction a
function of the labels parameter

I could change from neg_class to pos_label. However, I don't really
understand the label parameter. It should be an array of all possible
labels? Why we need that?

—
Reply to this email directly or view it on GitHub
#3511 (comment)
.

@jnothman Thanks. I replace the pos_label by labels, but I'm not sure if this is what you mean.
Please let me know if I misunderstands anything. Thanks.

Can you add tests for the new metric to ensure the correctness (see sklearn/metrics/tests/test_classification.py)? Some tests shared by all metrics are already written in sklearn/metrics/tests/test_common.py that you can readily re-used with minimal effort as explained in the file.

You also need to add a narrative documentation in doc/modules/model_evaluation.rst to highlight your work and showsto everybody your useful contribution. It should give the appropriate definition in more mathematical term and also gives insights where you want to use the balanced_accuracy_score or not. Lastly, could you add an entry for balanced_accuracy_score in doc/modules/classses.rst?

closed in favor of #5588