I'd really like to get in #1381 first, hopefully shortly after the release. Not sure how this fits into the interface implemented there.

@jnothman @amueller I will bring this PR back to life now. I need to rebase and resolve a few conflicts.

@jnothman @amueller I've rebased and cleaned up a few things. A few of the metrics now support sample_weights but I think someone more knowledgeable of the other metrics should implement it there. This PR should otherwise be ready for review, particularly for the grid searching with sample weights.

It probably comes down to personal preference, but I generally dislike the metrics being calculated multiple times depending on the parameters. I.e. I would much rather something like this at the top, then treat having sample_weight as the normal case:

if sample_weight is None:
    sample_weight = 1
    total_weight = len(y)
else:
    total_weight = np.sum(sample_weight)

(assuming sample_weight is being used as a multiplier; where it's passed to np.average, np.bincount, etc, it's fine to leave sample_weight=None.)

Couple of questions:

1. Does wanting to weight samples when fitting necessarily imply weighting should be used in scoring, and vice-versa?
2. Are there references or reference implementations of the metrics with weights? Or are we following our intuitions?

In particular, how does weighting interact with multilabel outputs? I assume the weight is applied to the instance as a whole, spread uniformly over the proposed labels for that instance (where that's meaningful).

You also need a more general invariance test: for any metric, integer array sample_weight, and real k, the following should be equal: metric(y1, y2, sample_weight * k) == metric(np.repeat(y1, sample_weight), np.repeat(y2, sample_weight)).

I wrote:

Does wanting to weight samples when fitting necessarily imply weighting should be used in scoring, and vice-versa?
Thinking further: assuming weighting at fit and score time is the most common use case, the user that really wants this flexibility can wrap their scorer or estimator to discard the sample_weight parameter.

This may be close to all the testing you need:

def test_sample_weight_invariance():
    y1, y2, _ = make_prediction(binary=True)
    int_weights = np.randint(10, size=y1.shape)
    for name, metric in METRICS_WITH_SAMPLE_WEIGHT:
        unweighted_score = metric(y1, y2, sample_weight=None)
        assert_equal(unweighted_score,
                     metric(y1, y2, sample_weight=np.ones(shape=y1.shape)),
                     msg='For %s sample_weight=None is not equivalent to '
                         'sample_weight=ones' % name)

        weighted_score = metric(np.repeat(y1, int_weights),
                                np.repeat(y2, int_weights))
        assert_not_equal(unweighted_score, weighted_score,
                         msg="Unweighted and weighted scores are unexpectedly "
                             "equal for %s" % name)
        assert_equal(weighted_score,
                     metric(y1, y2, sample_weight=int_weights),
                     msg='Weighting %s is not equal to repeating samples'
                         % name)
        for scaling in [2, 0.3]:
            assert_equal(weighted_score,
                         metric(y1, y2, sample_weight=int_weights * scaling),
                         msg='%s sample_weight is not invariant under scaling'
                             % name)
        assert_equal(weighted_score,
                     metric(y1, y2, sample_weight=list(scaling)),
                     msg='%s sample_weight is not invariant to list vs array'
                         % name)

@jnothman thanks for your help! I finally had some time to get back to this. This PR should now be ready.

Regarding your questions:

1. Does wanting to weight samples when fitting necessarily imply weighting should be used in scoring, and vice-versa?

Yes. Performance on samples with larger weights matters more than performance on samples with smaller weights. So each sample should contribute by its weight to the overall score.

1. Are there references or reference implementations of the metrics with weights? Or are we following our intuitions?

Probably, but as you can see in the PR the changes are quite trivial.

In other words, I am unaware of existing implementations of all of these metrics that account for sample weights. That would be helpful though. But, again, the modifications required here are not too serious.

In my field of research (high energy particle physics), samples are almost always weighted and the weights can vary a lot. So any classifier I use must account for the sample weights and any evaluation of a classifier's performance must also account for the sample weights.

One thing that would be worth thinking is the integration with the new scorer interface.

Hm we could have reviewed two pr. One with the metrics and one on top of the other with grid searching and cross validation score.

@arjoly yes, true. But, I need the weighted score metrics from this PR (for the grid_search unit tests), and I don't want to have a bunch of cherry-picked commits in the other PR.

I see... would it be fine to keep rebasing a grid_search PR on this one? I can split them again... Although all I have left is to add sample_weight to the scorers and then add unit tests.

Are there references or reference implementations of the metrics with weights? Or are we following our intuitions?

Probably, but as you can see in the PR the changes are quite trivial.

For some metrics, I agree that taking into the weight is straightforward, such as the mean squared error. But for other metrics such as the roc_auc_score or r2_score, I would be glad to see some references.

@arjoly, I think definitions are pretty straight forward in terms of invariance with integer weights.

Coverage remained the same when pulling 1ea2c7e on ndawe:weighted_scores into 613cf8e on scikit-learn:master.

@ndawe, is this still a WIP, or is it ready to be reviewed for merging?

Changes Unknown when pulling 0ce2180 on ndawe:weighted_scores into * on scikit-learn:master*.

Cleaned up the history. Will now start a PR for sample_weight in the scorer interface.

Coverage remained the same when pulling bf4cd4c on ndawe:weighted_scores into d62971d on scikit-learn:master.

The grid_search, rfe, learning_curve and cross_validation part of this PR should wait for #3340, which will conflict in many ways.

Thanks @vene. Sounds good. I've updated the description with a link to your #3401

Thanks Noel! I was thinking that the refactoring in #3340 might be a bit too big (and therefore slow to get in) while the remaining part of this PR are much more manageable, so I'd rather try to get #3340 broken into two and merge this faster. IMO these are important usability issues. We'll take this up tomorrow at the sprint. Ping @amueller, @arjoly and @GaelVaroquaux

In the grid search code, would it be conceivable to use a simple convention: that all fit_params that start with the prefix sample_ need to be passed masked to the estimator to be tuned. This would allow to handle sample_weight and sample_group (for learning to rank).

The current convention in https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cross_validation.py#L1184 is more general in assuming anything with length n_samples should be masked. The trickier part is differentially routing these to fit, decision_function and the metric.

@jnothman, @ndawe
Actually there's no question of routing: the convention only applies to the fit_params, which naturally should only be sent to fit. We would have a separate scorer_params or something: one for every function of an estimator that _fit_and_score can call.

I really need this right now because of sample groups (ie query_id).

I would also prefer the convention of giving them names that start with sample: sample_weights, sample_groups, etc, it's more explicit and less bug-prone (what if i make an off-by-one error when building the sample_weights vector?)

closing as superseeded by #13432. This code is too outdated to be useful, I think.