I think I'm +1 for moving _is_label_indicator_matrix and _is_multilabel to the metrics module (and of course, to make them public).

Since those function don't assess the performance of an estimator, I am not sure that the metrics module is the best place. I was thinking about a sklearn.multilabel module, a sklearn.utils.multilabels module or to put those functions in sklearn.multiclass.

Let's put them in sklearn.multiclass.

+1 for multiclass

+1 for multiclass too.

When I put is_label_indicator_matrix and is_multilabel into the multiclass, I got circular import.
Do you advise to do lazy import?

Another possible place would be in the utils.

what is the circle?

what is the circle?

In preprocessing, LabelBinarizer needs is_multilabel and is_label_indicator_matrix that are (will be) in multiclass.
In multiclass, unique_classes needs LabelBinarizer in preprocessing.

Maybe the best place is in preprocess. The narrative doc says:

The sklearn.preprocessing package provides several common utility functions and transformer classes to change raw feature vectors into a representation that is more suitable for the downstream estimators.

So it would be logical to find function check or analyze raw data.

In preprocessing, LabelBinarizer needs is_multilabel and
is_label_indicator_matrix that are in multiclass.
In multiclass, unique_classes needs LabelBinarizer in preprocessing.

OK, I thank that tells me that we need to move things in utils.

All right! I will create a new utils module.

There is now a sklearn.utils.multiclass (to rename to sklearn.utils.multilabels?).

I have pull the unique_labels functionality from LabelBinarizer and concentrate everything in unique_labels to get rid of the circular import problem.

Could you add multilabel support to precision / recall / f1 score? Once this is done, the multilabel tests in the multiclass module can be updated to use the metrics directly:
https://github.com/arjoly/scikit-learn/blob/ed98486d0c6b0072afe3b8b96a764037c90d2ad5/sklearn/tests/test_multiclass.py#L35

I intended to do that in my next pull request.
But ok, I will have a look at that tomorrow.

+1 for separate pr

+1 for separate pr

The voice of the reason: small and reviewable pr.

Don't worry @mblondel I intend to add an another pr with precision, recall and f-score.
It is pretty high on my todo list.

Perhaps one thing that could change is the name of the new utils module:

• sklearn.utils.classification
• sklearn.utils.multilabels
• sklearn.utils.multiclass

I rebase on top of master.

No worries.

Could you discuss the relationship between hamming loss and zero-one loss in the docstring? Thanks.

@mblondel I think that I have taken your remarks into account.

By the way, I add some more invariance tests.

In the multiclass (not multilabel) case, they are the same, right?

No, they differ. In the hamming loss, you divide each error by the number of labels.

One small example

In [22]: y2 = np.random.randint(0, 4, size=(5, ))

In [23]: y1 = np.random.randint(0, 4, size=(5, ))

In [24]: y1
Out[24]: array([2, 0, 3, 2, 2])

In [25]: y2
Out[25]: array([3, 1, 2, 1, 2])

In [26]: hamming_loss(y1, y2)
Out[26]: 0.40000000000000002

In [27]: zero_one_loss(y1, y2)
Out[27]: 0.80000000000000004

But thinking about it, the hamming loss is always smaller than the zero one loss. I will correct this.

Did you decide to add this normalization or is it always implemented like this in multilabel papers? http://en.wikipedia.org/wiki/Hamming_distance uses the unormalized count.

I'm asking because this is important that our implementation of the metrics is as standard as possible. We could add a normalize option (the question is, what should be the default value?).

The following papers agree on the normalization with the number labels:

1. Grigorios Tsoumakas, Ioannis Katakis. Multi-Label Classification: An Overview. International Journal of Data Warehousing & Mining, 3(3), 1-13, July–September 2007.
2. Jesse Read, Bernhard Pfahringer, Geoff Holmes, Eibe Frank. Classifier Chains for Multi-label Classification. Machine Learning Journal. Springer. Vol. 85(3), (2011).
3. Zhang, M.L. and Zhou, Z.H. ML-KNN: A lazy learning approach to multi-label learning
4. Gjorgji Madjarov, Dragi Kocev, Dejan Gjorgjevikj, Sašo Džeroski. An extensive experimental comparison of methods for multi-label learning. Pattern Recognition. Vol. 45(9), (2012).
5. Wei Gao and. Zhi-Hua Zhou. On the Consistency of. Multi-Label Learning. JMLR.

Great. Maybe you can cite the first one then.

Great. Maybe you can cite the first one then.

Done

I will have time this week to work on the precision, recall and F-measure metrics to support the multi-labels format. Furthemore, I would like to add the jaccard similiarty measure (an example based accuracy measure).

What do you advise to me? I will need some of the function in utils.multiclass.

Maybe do a PR on top of this one? We really should try to get this one in :-/

If I do a pr on top of this one, will I have problem if I rebase this one on top of master?

This one merges cleanly, no reason to rebase (though I'd like to rebase -i it prior to the actual merge to squash some commits).

You can branch off current master, merge this branch into your new branch, then add the functionality you want.

This one merges cleanly, no reason to rebase (though I'd like to rebase -i it prior to the actual merge to squash some commits).

You can branch off current master, merge this branch into your new branch, then add the functionality you want.

I will do as you suggest! Thanks!

Btw, it's easier if you first squash the commits using a rebase -i. It doesn't have to be all in one commit, but lots of microcommits make it harder to cherry-pick when an intermediate release is done.

Or someone can give this one a second +1 and we merge it ;)

I'll try to review the PR this afternoon. I'll merge it if I think it's ready.

awesome, thanks :)

We've got an inconsistency in the documentation. The dev docs say utils is off-limits to end users, while the multiclass docs now advises their use. I'm going to move the latter remark to the dev docs.

Ok, pushed to master after squashing. Thanks @arjoly for tackling this important problem: evaluation can be dull and it can make your head hurt, but it's crucial for a machine learning toolkit.

Thanks to all reviewers !!!

We have a failure on the Numpy 1.3/Scipy 0.7 build bot: ValueError: 0-d arrays can't be concatenated. It seems limited to the model_selection.rst doctests.

Can any one have a look at the failing doctest?

https://jenkins.shiningpanda-ci.com/scikit-learn/job/python-2.6-numpy-1.3.0-scipy-0.7.2/1656/console

https://jenkins.shiningpanda-ci.com/scikit-learn/job/python-2.6-numpy-1.3.0-scipy-0.7.2/

I will have a look this afternoon.

I am not able to install numpy 1.3 with python 2.6. :-$
So it is a bit hard to investigate :-(

I think you can reproduce it with numpy 1.3 on python 2.7. I don't see why it would be specific to 2.6.

No, I suspect it's Numpy-specific.

I am working on it.
I have the proper numpy version now, but the installation of scipy 0.7 failed...

I bet SciPy has very little to do with this, so you can try a later version first to see if you get the failures.

(Otherwise, try finding an old version of a Linux distro that has these versions and install it in a VM.)

I haven't been able to install to have python 2.7, numpy 1.3 with scipy 0.7.
I got the following error constantly:

error: Command "g++ -pthread -fno-strict-aliasing -I/home/ajoly/opt/local/include -DNDEBUG
-g -fwrapv -O3 -Wall  -fPIC -I/home/ajoly/git/numpy-1.3/numpy/core/include 
-I/home/ajoly/opt/python/include/python2.7 -c scipy/sparse/sparsetools/csr_wrap.cxx
 -o build/temp.linux-i686-2.7/scipy/sparse/sparsetools/csr_wrap.o" failed with exit status 1

Same with scipy 0.6 or any scipy 0.7.x version...

With python 2.6, I am not able to install numpy 1.3 due to a problem with unicode character (ucs2 and ucs4 problem).

Lastly, scipy 0.8 need at least numpy 1.4...
This makes me crazy...

I suppose that a simple np.asarray could solve the problem, but I am not able to investigate.
Or to handle the multiclass directly ...

Any suggestion for a linux distro with the required package (and if possible easy installation of) python 2.6, numpy 1.3 and scipy 0.7?

Ubuntu lucid should do.