[MRG+2] ENH labels parameter in P/R/F may extend or reduce label set #4287

jnothman · 2015-02-24T10:43:50Z

This PR replaces #2610, making the labels parameter to precision_recall_fscore_support more functional (and better documented, tested), but in accordance with #4192 not deprecating pos_label, but instead being restricted to the average != 'binary' case.

A common use of the micro-average is to extend the notion of binary P/R/F to the case where there is a frequent "negative class" and multiple classes of interest. Following this PR, explicitly listing the labels of interest allows the negative class to be excluded from a multiclass problem. (The same result can be achieved by transforming a multiclass problem into a multilabel problem excluding one label, but in the model evaluation API that would necessitate a custom and tricky scoring object.)

jnothman · 2015-03-01T04:44:03Z

Ahh it seems rebase incorporated a failing test due to #4147. See https://github.com/scikit-learn/scikit-learn/pull/4147/files#r25564051

jnothman · 2015-03-01T04:47:14Z

I've changed that test, hopefully not weakening it substantially in the multiclass case.

jnothman · 2015-04-11T12:47:44Z

(Travis had failed due to a heisenbug and I hadn't noticed for a while...)

jnothman · 2015-04-11T12:59:40Z

Rebased.

arjoly · 2015-04-14T10:57:50Z

An unfortunate amount of code here is dedicated to continued support of the deprecated multilabel sequence of sequences format. I'll be glad to see it go.

This is now deprecated. We can dig that out of the codebase.

jnothman · 2015-04-14T12:48:43Z

This is now deprecated. We can dig that out of the codebase.

Thanks for the reminder. I've cleared this out, I think.

GaelVaroquaux · 2015-04-14T14:33:16Z

sklearn/metrics/classification.py

@@ -498,8 +498,13 @@ def f1_score(y_true, y_pred, labels=None, pos_label=1, average='binary',
    y_pred : 1d array-like, or label indicator array / sparse matrix
        Estimated targets as returned by a classifier.

-    labels : array
-        Integer array of labels.
+    labels : array-like, optional


It's more a list-like, isn't it?

I don't think "list-like" is used elsewhere in the codebase for lists of labels/classes... I'm fine for this to be list.

jnothman · 2015-06-06T13:54:52Z

Rebased

jnothman · 2015-06-06T13:56:30Z

Rebased and Gaël's doc comment addressed

amueller · 2015-06-06T17:32:11Z

sklearn/metrics/tests/test_classification.py

@@ -315,7 +382,7 @@ def test_precision_refcall_f1_score_multilabel_unordered_labels():
    y_pred = np.array([[0, 0, 1, 1]])
    for average in ['samples', 'micro', 'macro', 'weighted', None]:
        p, r, f, s = precision_recall_fscore_support(
-            y_true, y_pred, labels=[4, 1, 2, 3], warn_for=[], average=average)


huh the previous test seems odd...

amueller · 2015-06-06T17:35:01Z

LGTM apart from minor comments.

jnothman · 2015-06-06T22:18:50Z

Thanks for the review @amueller!

arjoly · 2015-06-08T07:55:25Z

sklearn/metrics/classification.py

+        excluded, for example to calculate a multiclass average ignoring a
+        majority negative class, while labels not present in the data will
+        result in 0 components in a macro average.  By default, all labels in
+        ``y_true`` and ``y_pred`` are used in sorted order.


Could you add a comment about the labels in multi-label classification?

Otherwise thanks, the docstring is a lot better!

arjoly · 2015-06-08T07:59:23Z

Can you add a test ensuring that specifying labels with more labels than y_true / y_pred in multilabel classificaiton raises an error?

edit: I mean for micro, weighted and sample averaging as macro is already covered by your test.

arjoly · 2015-06-08T08:01:40Z

sklearn/metrics/tests/test_classification.py

+        assert_array_equal([.5, 1.], recall_13(average=None))
+        assert_equal((.5 + 1.) / 2, recall_13(average='macro'))
+        assert_equal((.5 * 2 + 1. * 1) / 3, recall_13(average='weighted'))
+        assert_equal(2. / 3, recall_13(average='micro'))


assert_array_almost_equal and assert_almost_equal?

arjoly · 2015-06-08T08:06:04Z

Do we still support the custom label order?

jnothman · 2015-06-08T09:06:12Z

Thanks for the review. Is there a reason we shouldn't still support custom
label order? Docstring says "... their order if average is None".

Also not sure what you want me to say about multilabel in the document
given "Labels present in the data can be excluded" is already said. I don't
know of use-cases for multilabel data where labels is useful in a
non-diagnostic scoring setting. Can you think of one?

On 8 June 2015 at 18:06, Arnaud Joly notifications@github.com wrote:

Do we still support the custom label order?

—
Reply to this email directly or view it on GitHub
#4287 (comment)
.

arjoly · 2015-06-08T09:24:16Z

I was unclear. Could you specify what you should be specified if multilabel data is passed? I would add something like : "In multi-label classification, labels are the column indices."

jnothman · 2015-06-08T10:04:21Z

Can you add a test ensuring that specifying labels with more labels than y_true / y_pred in multilabel classificaiton raises an error?
edit: I mean for micro, weighted and sample averaging as macro is already covered by your test.

You realise that for micro, weighted and sample averaging, the result would be identical were labels added with no instances?

jnothman · 2015-06-08T10:36:46Z

Your comments have been addressed, @arjoly. Thanks

On 8 June 2015 at 19:24, Arnaud Joly notifications@github.com wrote:

I was unclear. Could you specify what you should be specified if
multilabel data is passed? I would add something like : "In multi-label
classification, labels are the column indices."

—
Reply to this email directly or view it on GitHub
#4287 (comment)
.

jnothman · 2015-06-08T10:52:49Z

Travis error appears unrelated.

arjoly · 2015-06-08T11:17:39Z

Can you add a test ensuring that specifying labels with more labels than y_true / y_pred in multilabel classificaiton raises an error?
edit: I mean for micro, weighted and sample averaging as macro is already covered by your test.
You realise that for micro, weighted and sample averaging, the result would be identical were labels added with no instances?

Yes, however I would expect that all labels are specified given the indicator matrix encoding.

arjoly · 2015-06-08T11:18:31Z

LGTM !

Thanks @jnothman !

jnothman · 2015-06-08T13:47:11Z

throwing in what's new.

jnothman · 2015-06-08T13:50:06Z

and squashed

jnothman · 2015-06-08T14:26:11Z

shall i merge, then?

arjoly · 2015-06-08T14:57:09Z

You can !! Thanks @jnothman

arjoly · 2015-06-08T15:25:48Z

Thanks @jnothman !

amueller · 2015-06-08T16:54:58Z

Thanks!
Btw, there are currently no known travis errors, I think. If you find an unrelated error, please ping me.

jnothman · 2015-06-09T03:25:23Z

You're welcome; I'm glad to have this support a meaningful multiclass micro-average.

jnothman added the Enhancement label Feb 24, 2015

jnothman mentioned this pull request Mar 1, 2015

[MRG + 1] Sort labels in precision_recall_fscore_support #4147

Merged

jnothman force-pushed the prf_labels branch from 75afeed to 1acaa21 Compare April 11, 2015 12:58

jnothman force-pushed the prf_labels branch from 1acaa21 to 65e95a7 Compare April 14, 2015 12:48

GaelVaroquaux reviewed Apr 14, 2015
View reviewed changes

jnothman force-pushed the prf_labels branch from 65e95a7 to 9ab40d6 Compare June 6, 2015 13:54

amueller reviewed Jun 6, 2015
View reviewed changes

jnothman changed the title ~~[MRG] ENH labels parameter in P/R/F may extend or reduce label set~~ [MRG+1] ENH labels parameter in P/R/F may extend or reduce label set Jun 6, 2015

arjoly reviewed Jun 8, 2015
View reviewed changes

arjoly changed the title ~~[MRG+1] ENH labels parameter in P/R/F may extend or reduce label set~~ [MRG+2] ENH labels parameter in P/R/F may extend or reduce label set Jun 8, 2015

jnothman force-pushed the prf_labels branch from e4df432 to c3b2169 Compare June 8, 2015 13:50

jnothman closed this Jun 8, 2015

jnothman force-pushed the prf_labels branch from c3b2169 to 019fc9b Compare June 8, 2015 15:02

jnothman merged commit 019fc9b into scikit-learn:master Jun 8, 2015

raghavrv mentioned this pull request Oct 7, 2015

labels argument of classification_report is not useful when y is a list of strings #3123

Closed

jnothman mentioned this pull request Aug 11, 2016

ENH P/R/F should be able to ignore a majority class in the multiclass case #1983

Closed

This was referenced Oct 14, 2016

Get room level accuracy using DRED dataset nurhidayat86/adem#85

Closed

Imbalance data in DRED room level occupancy nurhidayat86/adem#89

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG+2] ENH labels parameter in P/R/F may extend or reduce label set #4287

[MRG+2] ENH labels parameter in P/R/F may extend or reduce label set #4287

jnothman commented Feb 24, 2015

jnothman commented Mar 1, 2015

jnothman commented Mar 1, 2015

jnothman commented Apr 11, 2015

jnothman commented Apr 11, 2015

arjoly commented Apr 14, 2015

jnothman commented Apr 14, 2015

GaelVaroquaux Apr 14, 2015

jnothman Apr 14, 2015

jnothman commented Jun 6, 2015

jnothman commented Jun 6, 2015

amueller Jun 6, 2015

jnothman Jun 6, 2015

amueller commented Jun 6, 2015

jnothman commented Jun 6, 2015

arjoly Jun 8, 2015

arjoly commented Jun 8, 2015

arjoly Jun 8, 2015

arjoly commented Jun 8, 2015

jnothman commented Jun 8, 2015

arjoly commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

arjoly commented Jun 8, 2015

arjoly commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

arjoly commented Jun 8, 2015

arjoly commented Jun 8, 2015

amueller commented Jun 8, 2015

jnothman commented Jun 9, 2015

[MRG+2] ENH labels parameter in P/R/F may extend or reduce label set #4287

[MRG+2] ENH labels parameter in P/R/F may extend or reduce label set #4287

Conversation

jnothman commented Feb 24, 2015

jnothman commented Mar 1, 2015

jnothman commented Mar 1, 2015

jnothman commented Apr 11, 2015

jnothman commented Apr 11, 2015

arjoly commented Apr 14, 2015

jnothman commented Apr 14, 2015

GaelVaroquaux Apr 14, 2015

Choose a reason for hiding this comment

jnothman Apr 14, 2015

Choose a reason for hiding this comment

jnothman commented Jun 6, 2015

jnothman commented Jun 6, 2015

amueller Jun 6, 2015

Choose a reason for hiding this comment

jnothman Jun 6, 2015

Choose a reason for hiding this comment

amueller commented Jun 6, 2015

jnothman commented Jun 6, 2015

arjoly Jun 8, 2015

Choose a reason for hiding this comment

arjoly commented Jun 8, 2015

arjoly Jun 8, 2015

Choose a reason for hiding this comment

arjoly commented Jun 8, 2015

jnothman commented Jun 8, 2015

arjoly commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

arjoly commented Jun 8, 2015

arjoly commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

jnothman commented Jun 8, 2015

arjoly commented Jun 8, 2015

arjoly commented Jun 8, 2015

amueller commented Jun 8, 2015

jnothman commented Jun 9, 2015