[MRG] Added metrics support for multiclass-multioutput classification #3681

akshayah3 · 2014-09-20T09:12:53Z

Fix for #3453
Ping @arjoly . Added support for zero_one_loss and accuracy_score

akshayah3 · 2014-09-21T11:49:48Z

@MechCoder could you please help in figuring out the test failure?

jnothman · 2014-09-21T12:01:27Z

The errors look like somehow you're transforming metric outputs into integers...

jnothman · 2014-09-21T12:03:45Z

Your current code in _check_targets is reporting the type as multilabel-indicator when the input is multiclass-multioutput. I don't see why that's creating the current barrage of errors, but it can't possibly be correct behaviour.

coveralls · 2014-09-21T16:15:56Z

Coverage decreased (-0.04%) when pulling 392f18a on akshayah3:metrics into 9580431 on scikit-learn:master.

akshayah3 · 2014-09-21T16:18:47Z

@jnothman The issue was with the _check_targets method. I fixed it, could you review the code please?

jnothman · 2014-09-22T01:38:20Z

sklearn/metrics/classification.py

        if y_type == 'multilabel-sequences':
            labels = unique_labels(y_true, y_pred)
            binarizer = MultiLabelBinarizer(classes=labels, sparse_output=True)
            y_true = binarizer.fit_transform(y_true)
            y_pred = binarizer.fit_transform(y_pred)
-
+            y_type = 'multilabel-indicator'
+        if y_type == 'multiclass-multioutput':


This clause is redundant.

@jnothman Yes i will remove that, apart from that does this look good?

arjoly · 2014-09-22T07:38:27Z

Thanks for tackling this issue!

Can you also update the documentation and narrative documentation?

arjoly · 2014-09-22T07:39:20Z

sklearn/metrics/tests/test_classification.py

+    assert_equal(zero_one_loss(y1, y2), 0.5)
+    assert_equal(zero_one_loss(y1, y1), 0)
+    assert_equal(zero_one_loss(y2, y2), 0)
+    assert_equal(zero_one_loss(y2, [(), ()]), 1)


This is not multi-class multi-output, but multi-label sequence. This should result in an error.

akshayah3 · 2014-09-22T09:23:00Z

@arjoly I have made some changes you suggested!

arjoly · 2014-09-22T09:26:26Z

sklearn/metrics/tests/test_common.py

+    y_pred = random_state.randint(0, 4, size=(20, 5))
+    n_samples = y_true.shape[0]
+
+    for name in ["accuracy_score","zero_one_loss"]:


Here I would add a constant METRICS_WITH_MULTICLASS_MULITOUTPUT at the top and loop over it.

coveralls · 2014-09-22T09:34:28Z

Coverage increased (+0.01%) when pulling 04c02b7 on akshayah3:metrics into 9580431 on scikit-learn:master.

coveralls · 2014-09-22T10:28:16Z

Coverage increased (+0.01%) when pulling 43861b6 on akshayah3:metrics into 9580431 on scikit-learn:master.

coveralls · 2014-09-22T11:12:58Z

Coverage increased (+0.01%) when pulling edfe486 on akshayah3:metrics into 9580431 on scikit-learn:master.

akshayah3 · 2014-09-22T11:18:36Z

@arjoly Does this look good now?

arjoly · 2014-09-22T12:27:15Z

doc/modules/multiclass.rst

@@ -74,8 +74,9 @@ tasks :ref:`Decision Trees <tree>`, :ref:`Random Forests <forest>`,

 .. warning::

-    At present, no metric in :mod:`sklearn.metrics`
-    supports the multioutput-multiclass classification task.
+    At present, metrics such as accuracy_score and zero_one_loss in


I would say :

At present, only the :fun:`accuracy_score`and :fun:`zero_one_loss` support multioutput-multiclass classification task

Hm this paragraph could be removed. Since thanks to you, we will have metrics now.

arjoly · 2014-09-22T12:31:48Z

For the narrative doc, I was thinking in updating this page / file

akshayah3 · 2014-10-07T09:33:49Z

@jnothman @arjoly Any changes to be done?

akshayah3 · 2014-10-10T06:02:14Z

@arjoly I have adressed the comments, Does this look good?

arjoly · 2014-10-10T08:56:56Z

sklearn/metrics/tests/test_common.py

+    random_state = check_random_state(0)
+    y_true = random_state.randint(0, 4, size=(20, 5))
+    y_pred = random_state.randint(0, 4, size=(20, 5))
+    for name in ALL_METRICS.keys():


You don't need the keys to iterate over all keys

akshayah3 · 2014-11-06T07:53:02Z

@arjoly Sorry for the late reply. I was busy with my university exams.
Could you review the latest commit?

arjoly · 2014-11-07T10:46:44Z

sklearn/metrics/tests/test_common.py

+
+    for name in ALL_METRICS:
+        if (name not in METRICS_WITH_MULTICLASS_MULITOUTPUT and
+                    name not in MULTIOUTPUT_METRICS):


Here, I think that here it should be an or instead of an and

@arjoly I dont think so, the test is to raise an Exception for all the metrics which do not support multiclass-multioutput inputs and note that Multioutput_Metrics do support them hence the and

Hm sorry, I misread the not.

arjoly · 2014-11-07T12:19:37Z

Can you ensure that we still get meaninfull error message?

Now, we have

In [1]: from sklearn.metrics import precision_score

In [2]: import numpy as np

In [3]: precision_score(np.array([[1, 2], [0, 3], [4, 3]]), np.array([[1, 2], [0, 3], [4, 3]]))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-c99aa70a6c5e> in <module>()
----> 1 precision_score(np.array([[1, 2], [0, 3], [4, 3]]), np.array([[1, 2], [0, 3], [4, 3]]))

/Users/ajoly/git/scikit-learn/sklearn/metrics/classification.py in precision_score(y_true, y_pred, labels, pos_label, average, sample_weight)
   1043                                                  average=average,
   1044                                                  warn_for=('precision',),
-> 1045                                                  sample_weight=sample_weight)
   1046     return p
   1047 

/Users/ajoly/git/scikit-learn/sklearn/metrics/classification.py in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight)
    843     label_order = labels  # save this for later
    844     if labels is None:
--> 845         labels = unique_labels(y_true, y_pred)
    846     else:
    847         labels = np.asarray(labels)

/Users/ajoly/git/scikit-learn/sklearn/utils/multiclass.py in unique_labels(*ys)
     85     # Check that we don't mix label format
     86 
---> 87     ys_types = set(type_of_target(x) for x in ys)
     88     if ys_types == set(["binary", "multiclass"]):
     89         ys_types = set(["multiclass"])

/Users/ajoly/git/scikit-learn/sklearn/utils/multiclass.py in <genexpr>((x,))
     85     # Check that we don't mix label format
     86 
---> 87     ys_types = set(type_of_target(x) for x in ys)
     88     if ys_types == set(["binary", "multiclass"]):
     89         ys_types = set(["multiclass"])

/Users/ajoly/git/scikit-learn/sklearn/utils/multiclass.py in type_of_target(y)
    297         # known to fail in numpy 1.3 for array of arrays
    298         return 'unknown'
--> 299     if y.ndim > 2 or (y.dtype == object and len(y) and
    300                       not isinstance(y.flat[0], string_types)):
    301         return 'unknown'

TypeError: len() of unsized object

While previously it was returning

In [6]: precision_score(np.array([[1, 2], [0, 3], [4, 3]]), np.array([[1, 2], [0, 3], [4, 3]]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-c99aa70a6c5e> in <module>()
----> 1 precision_score(np.array([[1, 2], [0, 3], [4, 3]]), np.array([[1, 2], [0, 3], [4, 3]]))

/Users/ajoly/git/scikit-learn/sklearn/metrics/classification.py in precision_score(y_true, y_pred, labels, pos_label, average, sample_weight)
   1033                                                  average=average,
   1034                                                  warn_for=('precision',),
-> 1035                                                  sample_weight=sample_weight)
   1036     return p
   1037 

/Users/ajoly/git/scikit-learn/sklearn/metrics/classification.py in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight)
    829         raise ValueError("beta should be >0 in the F-beta score")
    830 
--> 831     y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    832 
    833     label_order = labels  # save this for later

/Users/ajoly/git/scikit-learn/sklearn/metrics/classification.py in _check_targets(y_true, y_pred)
     89     if (y_type not in ["binary", "multiclass", "multilabel-indicator",
     90                        "multilabel-sequences"]):
---> 91         raise ValueError("{0} is not supported".format(y_type))
     92 
     93     if y_type in ["binary", "multiclass"]:

ValueError: multiclass-multioutput is not supported

?

akshayah3 · 2014-11-08T06:59:21Z

@arjoly Any more changes to be made?

ogrisel · 2014-12-16T10:08:00Z

doc/modules/model_evaluation.rst

@@ -293,6 +300,10 @@ In the multilabel case with binary label indicators: ::
  >>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
  0.5

+In the case of multiclass-multioutput: ::


Please add a blank line for readability of the source. Also you can write multiclass-multioutput:: directly instead of multiclass-multioutput: ::.

jnothman · 2017-05-29T23:38:26Z

@Akshay0724, there was a request at #3453 that this be finished up. Do you intend to complete it, or should we find another contributor?

arf1372 · 2019-02-04T13:07:25Z

Don't any developer wants to resolve conflicts within this?
I need a multiclass-multioutput metric for grid search in my task and I see no support of such metric in sk-learn unfortunately.

@jnothman I'll be happy contribute to this one as I need it personally.

jnothman · 2019-02-04T21:52:19Z

Thanks for the ping. I don't think it would hurt to support multioutput for accuracy. We should probably review this.

akshayah3 added 2 commits September 20, 2014 14:37

Added metrics support for multiclass-multioutput classification

f4648c7

Made small changes in tests

6e245b7

Small change in _check_targets method

392f18a

jnothman reviewed Sep 22, 2014
View reviewed changes

arjoly reviewed Sep 22, 2014
View reviewed changes

Added test in test_common.py

04c02b7

arjoly reviewed Sep 22, 2014
View reviewed changes

Small change in test_common.py

43861b6

Changed some tests in test_classification.py

edfe486

arjoly reviewed Sep 22, 2014
View reviewed changes

arjoly reviewed Oct 10, 2014
View reviewed changes

MechCoder force-pushed the master branch from 6deaea0 to 3f49cee Compare November 3, 2014 12:36

Merged tests

e7ea114

arjoly reviewed Nov 7, 2014
View reviewed changes

Better error message in the case of multiclass-multioutput inputs

f0e47e1

ogrisel reviewed Dec 16, 2014
View reviewed changes

amueller added the Waiting for Reviewer label Dec 10, 2015

This was referenced May 29, 2017

Missing multiclass-multioutput support automl/auto-sklearn#292

Closed

Add multioutput-multiclass support to metrics #3453

Open

github-actions bot added module:metrics module:utils labels Mar 2, 2020

cmarmo added help wanted Stalled and removed Waiting for Reviewer labels Sep 29, 2020

adrinjalali closed this Jan 22, 2021

adrinjalali deleted the branch scikit-learn:master January 22, 2021 10:54

Uh oh!

[MRG] Added metrics support for multiclass-multioutput classification #3681

[MRG] Added metrics support for multiclass-multioutput classification #3681

Uh oh!

Conversation

akshayah3 commented Sep 20, 2014

Uh oh!

akshayah3 commented Sep 21, 2014

Uh oh!

jnothman commented Sep 21, 2014

Uh oh!

jnothman commented Sep 21, 2014

Uh oh!

coveralls commented Sep 21, 2014

Uh oh!

akshayah3 commented Sep 21, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arjoly commented Sep 22, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akshayah3 commented Sep 22, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Sep 22, 2014

Uh oh!

coveralls commented Sep 22, 2014

Uh oh!

coveralls commented Sep 22, 2014

Uh oh!

akshayah3 commented Sep 22, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arjoly commented Sep 22, 2014

Uh oh!

akshayah3 commented Oct 7, 2014

Uh oh!

akshayah3 commented Oct 10, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akshayah3 commented Nov 6, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arjoly commented Nov 7, 2014

Uh oh!

akshayah3 commented Nov 8, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnothman commented May 29, 2017

Uh oh!

arf1372 commented Feb 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnothman commented Feb 4, 2019 via email

Uh oh!

Uh oh!

arf1372 commented Feb 4, 2019 •

edited

Loading