precision[size_true == 0]

Could you please clarify to me the datatype of y_true and y_pred? Your use of in seems to suggest they are sequences of iterables (tuples?) over labels. But I got the impression from the use of LabelBinarizer and the indicator matrix case that we had a binary array. I don't know whether I've misread some code.

If it's represented as a binary array, can't you do something like:

true_pos = (y_true & y_pred).sum(axis=0)`
false_pos = (~y_true & y_pred).sum(axis=0)
false_neg = (y_true & ~y_pred).sum(axis=0)

?
If there's some reason not to use a binary array, would set intersection and difference be better (in the case of many labels) than iterating through every label for each sample?

Argh. I got it, I think. You're handling two different formats for the same multilabel data. Is that necessary?

Yes, I support a dense binary array indicator matrix and a list of list of label. Both have their pros and cons.
I support both format to avoid memory copy and to keep the pros of each format.

So is set intersection not a better idea than doing a label-by-label comparison?

You are right. Done in ce395ac,

The second condition here is redundant: you should have already returned from that case above.

I just realised the implications of this condition. Here you're checking if one is a sequence of sequences and the other is an indicator matrix. What if the one was an array of sequences? I know at the moment that's not allowed by is_multilabel, but I'm not sure we want to write code that would fail if is_multilabel is changed to accept it.

Accepting an array of sequences in the future will mean we can perform fast vectorized operations. Note that's the format used in scipy.sparse.lil_matrix.rows, which is essentially the same data structure that we're working with.

So I think we need a helper function to do this multilabel_compatible check.

I am not sure what the goal of this check is (the docstring does tell me
all the shapes possible in the list of labels). However, it is a very
weak test: what if I pass in an ndarray and a memmap? Also, a list is a
valid array-like, and we tend to expect code to behave right for lists.

I am not sure if it would solve your usecase, but could you simply
convert both inputs to ndarrays, using np.asarray, and test for their
shape/dimension? Judging by the discussion, I have the impression that
you want something more flexible, but I don't understand how it would
work/what would be the purpose (examples in the docstring are great for
these purposes).

Basically, there are two forms of multilabel targets supported, corresponding to dense and sparse representations of the same thing.

So,

y = [
[0, 3],
[1],
[0, 2]
]

is a sequence-of-sequences representation. If the first element of y is a sequence, but not a string or ndarray, we assume y is multilabel and sequence-of-sequences. It is akin to lil_matrix, and looks like the non-multilabel targets format. (But I should correct my post above: lil_matrix uses an array of lists. But this comparison fails with an array of lists; we reject an array of arrays.) It is useful when there are very many sparsely-assigned classes.

The same data could be represented as an indicator matrix (dense-like):

y = array([
[T F F T],
[F T F F],
[T F T F]
])

The type comparison above is to make sure y_pred and y_true are of the same form, and if not, to convert to the latter. In using type, it would make an error if the sequence-of-sequences was itself an array, being compared to a label indicator matrix (the condition should be True, but would be False).

but could you simply convert both inputs to ndarrays, using np.asarray, and test for their shape/dimension?

If there is redundant labels in the list of list of label format, it won't work in all case.

However, it is a very weak test: what if I pass in an ndarray and a memmap?

I have no experience with memmap, so I haven't consider this possibility.
Is there a test somewhere that check that scikit-learn works correctly with memmap? I had a quicklook in test_common.py and found nothing.

I added several format invariance tests (see test_format_invariance_with_1d_vectors and test_multilabel_representation_invariance in test_metrics) to check that different formats give the same answer. Any suggestions is welcome.

In using type, it would make an error if the sequence-of-sequences was itself an array, being compared to a label indicator matrix (the condition should be True, but would be False).

Can you give an example?

take my two ys from above and wrap them in np.asarray. Both would have the same type (but not the same dtype).