Coverage decreased (-0.03%) when pulling f1f6a3c on vene:weighted_score_params into 07560e4 on scikit-learn:master.

I don't dispute the utility of scorer_params in other cases, but I think we are best off supporting a sample_weight parameter directly in GridSearchCV et al. This is consistent with the standard fit interface across scikit-learn, and makes things like nested CV straightforward.

@jnothman I am not convinced either. But there are two attributes that need to have this behaviour: sample_weight and sample_group, and it made me think that it'd be cleaner to implement them together.

This is clearly a generalizable approach, which is tempting---but I can't come up with other examples of arguments that should be passed like this, though. Also I can't think of other scorer_params that are not data-dependent and can't be set when instantiating a Scorer object.

So I'm torn.
But I need sample_group though.

As far as I'm concerned, scikit-learn is almost entirely concerned with data where samples may be assumed iid. sample_group falls outside (or at least on the periphery) of that space, while sample_weight is something that most estimators, metrics, etc. could/should support. However, it is useful that the more generic components in scikit-learn also be compatible with learning tasks where additional information like sample_group is needed. I think this is sufficient for arguing that sample_weight and sample_group be considered different classes of citizen.

(Sorry, slipped.)

On a related note, split_params may also be needed in the context of #3340 and nested cross-validation, in order to handle LeaveOneLabelOut. Or is that getting too crazy?

On a completely different note, I doubt that scorer_params should be an attribute of GridSearchCV et al., mostly because it will add confusion to the metrics-scorer interfaces. Basically, scorers should be designed such that all parameters other than estimator are data-aligned, and hence scorer_params should only be provided as a kwarg to fit.

Another way of looking at it is that arbitrary kwargs could be provided to fit, and the GridSearchCV constructor could have a parameter that defines their routing: fit_param_routing={'sample_weight': ['fit', 'score'], 'sample_group': ['score'], 'labels': ['split']} (or the inverse mapping). (NB: In nested CV, all parameters should be routed to 'fit'.)

I see your point and I agree. I was thinking the same: something like sample_group should definitely not be defined in scikit-learn itself, but the cross-val and search API should allow for something like this to be implemented easily on top, without having to copy-paste and hack the code.

Something that kind of collides with your suggestion is that fit_params is already a constructor argument, while by design stuff that goes into fit_params will be data-dependent, so this is a bit weird. I think it would be cleaner to route arbitrary **kwargs passed to the grid search's fit method.

fit_params I think has long legacy, but I don't think it would be harmful to deprecate it. I only worry that the routing approach is a bit too abstracted. Although perhaps the argument that in general the base estimator's fit should not receive sample_group but that a nested cross-validator's fit should, suffices to say that a mechanism of this power is necessary.

Also closing this one since SLEP006 provides the API and it's already implemented for scorers.