Skip to content

[WIP] ENH Multilabel confusion matrix #4126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[WIP] ENH Multilabel confusion matrix #4126

wants to merge 1 commit into from

Conversation

raghavrv
Copy link
Member

@raghavrv raghavrv commented Jan 19, 2015 •
edited
Loading

Fixes #3452

Based on #3614 (This is rewritten to conform to the referenced implementations). Thanks @Magellanea!

  • binarized_multilabel_confusion_matrix -> multilabel_confusion_matrix
  • Return MCM of shape (n_labels, 4) instead of numpy struct array.
  • Add multilabel_confusion_matrix entry to test_common.py
  • Add multilabel_confusion_matrix tests to test_classification.py
  • Add what's new entry
  • Add to metrics doc.

Rewrite TODO

  • Rewrite multilabel_confusion_matrix to conform to more standard approaches.

Ref - http://www.clips.ua.ac.be/~vincent/scripts/confusionmatrix.py
Ref2 - http://www.clips.ua.ac.be/~vincent/pdf/microaverage.pdf
Ref3 - http://metaoptimize.com/qa/questions/8964/confusion-matrix-for-multiclass-multilabel-classification

@raghavrv raghavrv changed the title Multilabel confusion matrix [WIP] Multilabel confusion matrix Jan 19, 2015
@raghavrv
Copy link
Member Author

@jnothman @arjoly Please take a look... Have to add tests and doc entry...

@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch from f05e92a to bdff9e2 Compare January 19, 2015 22:29
jnothman
jnothman reviewed Jan 19, 2015
View reviewed changes
sklearn/metrics/classification.py Outdated
mcc = np.empty(shape=(n_labels, 4), dtype=int)

for label_idx in range(n_labels):
y_pred_col = y_pred.getcol(label_idx)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't remember the state of things, but I think this needs to support dense or sparse y and you should use the count_nonzero helper.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick comment! will fix this by tomorrow... soon ;)

@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch 3 times, most recently from 72fc7b0 to b811206 Compare January 20, 2015 18:36
@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch from b811206 to 7b2e166 Compare January 27, 2015 14:10
@raghavrv
Copy link
Member Author

I am unable to validate the below inputs for y_true, y_pred for the multilabel_confusion_matrix without setting additional constraint that if y_type = set('binary', multilabel-sequences') => y_type = set('multilabel-sequences')

>>> y_true = [['A'], ['A']]       =>   type = 'binary'
>>> y_pred = [['B','C'], ['A']]   =>   type = 'multilabel-sequences'

If this is done the test which checks for mix of binary and multilabel-sequences should be scraped... Is that okay? Should multilabel-sequences and binary types not mix for any other context?

EDIT:

Or perhaps I should locally deal with it and binarize it before the _check_targets helper is used?

Also looks like unique_labels needs a similar modification...

@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch from 7b2e166 to 3752f9e Compare February 1, 2015 18:54
@jnothman
Copy link
Member

jnothman commented Feb 4, 2015

Don't worry about multilabel-sequences. They're deprecated and do not need to be supported in new code.

@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch 3 times, most recently from 8f16312 to 63b16a9 Compare February 4, 2015 16:39
@raghavrv raghavrv changed the title [WIP] Multilabel confusion matrix [MRG] Multilabel confusion matrix Feb 4, 2015
@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch 3 times, most recently from c55cd16 to 7d5741d Compare February 4, 2015 18:17
@raghavrv
Copy link
Member Author

raghavrv commented Feb 4, 2015

@jnothman This is done... please take a look when you find time!

@coveralls
Copy link

Coverage Status

Coverage increased (+0.01%) to 95.0% when pulling 7d5741d on ragv:multilabel_confusion_matrix into 31c5497 on scikit-learn:master.

@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch from 7d5741d to 530bf04 Compare February 4, 2015 18:32
ogrisel
ogrisel reviewed Feb 5, 2015
View reviewed changes
sklearn/metrics/classification.py Outdated
@@ -186,6 +187,74 @@ def accuracy_score(y_true, y_pred, normalize=True, sample_weight=None):
return _weighted_sum(score, sample_weight, normalize)


def multilabel_confusion_matrix(y_true, y_pred):
"""Compute True positive, False positive, True negative, False negative
for a multilabel classification problem.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please follow PEP 257: one liner short description, blank line, possibly multi-line detailed description.

Having a short first line is important for many dev tools (e.g. IDEs) and API documentation tools such as the API summary page built with sphinx.

def multilabel_confusion_matrix(y_true, y_pred):
    """Compute the confusion matrix for a multilabel classification problem.

    Compute True positive, False positive, True negative, False negative rates for each
    binary classification sub-problem.

    ...
    """
    ...

@raghavrv
Copy link
Member Author

raghavrv commented Feb 6, 2015

@ogrisel I think I did this wrong... According to table 7 of the reference paper and your answer in metaoptimize ( ;) ) the implementation should use tp tn fp fn but must be presented it in a different way. Could you advise if my understanding is true and whether I should modify it that way?

@raghavrv
Copy link
Member Author

raghavrv commented Feb 6, 2015

The website seems to be down... Kindly refer the cached page instead.

jnothman
jnothman reviewed Feb 10, 2015
View reviewed changes
doc/modules/model_evaluation.rst Outdated
Each row contains the values for the 4 expressions of the F-Score namely
:math:`TP, FP, TN, FN`.

For each label (row) the values are computed as follows :
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is described more succinctly under Binary Classification. Reference that instead.

@raghavrv
Copy link
Member Author

@jnothman Could you kindly confirm if this is the correct way to represent the multilabel confusion matrix, rather than the current approach? That ( as given in the link ) seems to be the form suggested in the paper too... sorry I didn't refer that properly!

@jnothman
Copy link
Member

I think as @ogrisel says on metaoptimize, the notion of confusion matrix when it comes to multilabel data is not especially standardised. This tp, fp, fn, tn breakdown is useful, but admittedly it doesn't really show confusion (and the same information is more-or-less conveyed by precision_recall_fscore_support(..., average=None)).

The sort of confusion matrix shown in table 7 of the reference paper may be useful, although I'm not actually sure that the partial scores (i.e. dividing the counts among the number of predicted) help much with interpretation, but I admit that at least the marginals make some sense. Are there other references that show this confusion matrix (a textbook would be a good start)?

Another interesting analysis akin to a confusion matrix would essentially treat each distinct labelling as a class and while with this combinatoric explosion might make a complete multiclass confusion matrix hard to use, just getting a list of the most frequent confusions might be insightful.

Rather than doing this blind, why don't you make an example called "Analysing confusion in multilabel classification" and use a real dataset (e.g. Reuters Corpus classification) to actually work out which tabulation (or visualisation) helps you explain best what's going on?

@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch from 530bf04 to c31c96a Compare May 22, 2015 22:41
@raghavrv raghavrv changed the title [MRG] Multilabel confusion matrix [WIP] Multilabel confusion matrix Dec 11, 2015
@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch from c31c96a to ecf4441 Compare January 24, 2016 15:53
@raghavrv raghavrv changed the title [WIP] Multilabel confusion matrix [WIP] ENH Multilabel confusion matrix Apr 20, 2016
@pramitchoudhary
Copy link

@raghavrv Were you able to get your changes in, or is it still under review. Do you need any help ?

@raghavrv
Copy link
Member Author

@pramitchoudhary If you would like to take over, please go ahead! There is some comparison (between the possible forms) that needs to be done to fix the API.

@Magellanea @raghavrv
ENH Implementing multilabel confusion matrix
96189d9 
FIX Use structured array for access label of multilabel binarized confusion matrix.
DOC Minor DOC test fix.
FIX Clean up and minor fixes.
FIX map function from python 2 to python 3
@raghavrv raghavrv force-pushed the multilabel_confusion_matrix branch from ecf4441 to 96189d9 Compare October 24, 2016 21:44
@rth
Copy link
Member

rth commented Aug 3, 2020

Closing, resolved in #11179

@rth rth closed this Aug 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
help wanted module:metrics Stalled
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add multi-label support to the confusion matrix metric
10 participants
@raghavrv @jnothman @coveralls @pramitchoudhary @rth @ogrisel @amueller @lesteve @TomDLT @Magellanea