You define the precision and recall as in average="sample". However, this definition failed in the multiclass case.

I prefer the more verbose definition of avg_precision, avg_recall and avg_F. I think that the redefinition of y_j and w_j for every average parameter is missleading, especially with the previous definition of y as label set .

I am a bit confused by that comment:

• I didn't actually define y as a label set. I leave intentionally unspecified what it's a set of. P, R and F can be calculated over any two sets. So I don't see how my description is suited to average="samples".
• Does "I prefer the more verbose definition of avg_precision..." mean you would rather each formula be expanded out? I personally like the formulas being a function of vectors y and w, but certainly think all should use the same set notation.

I didn't actually define y as a label set. I leave intentionally unspecified what it's a set of. P, R and F can be calculated over any two sets. So I don't see how my description is suited to average="samples"

In the narrative doc, you say

For these purposes, it is clearer to redefine our metrics in terms of sets.

For a true set :math:`\hat{y}` and predicted set :math:`y`, we may redefine:

.. math::

    \text{precision} = \frac{\left| y \cap \hat{y} \right|}{\left|y\right|},

.. math::

    \text{recall} = \frac{\left| y \cap \hat{y} \right|}{\left|\hat{y}\right|}.

you also say

Defining :math:`y_l` and :math:`\hat{y}_l` to be sets of samples with label
+:math:`l`

But later in the same section, :math:y_j consists of either all (sample, label) pairs, either samples assigned label :math:jor either labels assigned to sample :math:j.

So I don't see how my description is suited to average="samples".

The two definitions that you gave of precision and recall

    \text{precision} = \frac{\left| y \cap \hat{y} \right|}{\left|y\right|},
    \text{recall} = \frac{\left| y \cap \hat{y} \right|}{\left|\hat{y}\right|}.

are for one sample in the multilabel case in the case average="samples" and those will reduce to classification accuracy in the multiclass case.

Does "I prefer the more verbose definition of avg_precision..." mean you would rather each formula be expanded out? I personally like the formulas being a function of vectors y and w, but certainly think all should use the same set notation.

Yes, I prefer that each formula be expanded. I think it is more eplicit

Okay. That's not how I intended it to be read, which means it is unclear. I meant y and \hat{y} to be sets of arbitrary objects. For micro-averaging, they are sets of (sample, label) pairs; for samples averaging they are sets of (label) per sample; for the others, they are sets of (sample) per label. I thought that generalisation would help clarify it.

Would it be okay if I said explicitly that y and \hat{y} were made up of (sample, label) pairs, and then define y_l to be the subset of y with label l (perhaps y_l = {(s, l') \in y | l' = l}), and elsewhere y_s to be the subset of y with sample s or something similar?

It is precisely this sort of subset selection that defines all sorts of variations of precision and recall (e.g. variant metrics for named entity recognition that have lenient matches on boundary), and I think this set-based definition should be prominent.

How about this version? I think this makes clear the similarities and differences between the different metrics.

added notes on accounting for label imbalance in "weighted"; merged as 7ebfd57.

Does this sort of documentation change belong in What's New?

Does this sort of documentation change belong in What's New?

I don't think so, sorry.

I don't mind! Just checking because I hadn't at first realised it was common to update What's New within a patch, so now I'm curious to know what qualifies.

Just checking because I hadn't at first realised it was common to
update What's New within a patch, so now I'm curious to know what
qualifies.

Think of it in the shoes of a user: it's the document that you want to
read to see what has changed in scikit-learn with a new release.

Got it. Thanks.