With this update, is there any reason to keep get_scorer as public api?

This is true that it could only be private then.

If this is only used to modify make_scorer's kwargs (and not its more fundamental params), then wouldn't get_scorer(name, **kwargs) be more natural?

If this is only used to modify make_scorer's kwargs (and not its more fundamental params), then wouldn't get_scorer(name, **kwargs) be more natural?

Indeed, it makes the parameters bypassing less odd I think.

My only concern is about the documentation and how we can do to make this part understandable.
My proposal would be to:

• if you wish to get a predefined scorer, use get_scorer;
• if you get an error because you need to pass extra parameter within a CV object, use get_scorer and pass the parameters;
• if you want to go free solo designing your own scorer from a callable, use make_scorer.

For the second point, I think that we should raise a better error message. Usually, a user will use a CV object and get something that he needs to explicitly pass some parameters. I don't know if we can easily detect who is calling the scoring function (a CV object or the user directly) but we could at least mention that if you use a CV object, use get_scorer (or currently make_scorer).

I think make_scorer(..., **kwargs) is a little more natural when compared to get_scorer(..., **kwargs). When providing **kwargs, we are making a "new scorer" and not "getting a preexisting one".

So if we really need another verb like customize_scorer, so be it (I'd rather not)

I agree to not have another function.

I think allowing make_scorer to take a string, and nominally to modify
whether greater is better when you do so is super awkward.

Thinking it over, passing make_scorer another scorer is a bit strange. (In this case the scorer is passed as a string.)

@glemaitre Should we close this and try to move forward with #17962?

@glemaitre Should we close this and try to move forward with #17962?

I am fine closing. But this is still not clear where to we go forward. Are we ok with:

1. Add get_params and set_params to the scorer API. If so, these methods should only update the scoring function parameters (i.e. the one in scorer._kwargs) or the scorer parameters (i.e. scorer._scoring_func, scorer._sign, and scorer._kwargs). Here, I would be in favour of the former.
2. Making the scorer mutable, we need to make get_scorer returning a copy.
3. (optional maybe) accept strings in get_scorer or we are fine with getting scorer followed by scorer.set_params()?

Adding get_params and set_params elevates the scorer object to be a little more than a "callable'. This would help resolve other types of issues as noted by @jnothman #17962 (comment) I am a concerned with how users will provide their own custom scorers. Would their scorers need to have the *_params interface?

As for this specific issue with "adjusting a str represented by a scorer", adding **kwargs to get_scorer would be okay as a user facing feature. It is still a little weird that nothing happens with the **kwargs if a callable is passed to get_scorer.

Now that we have #22866 and get_scorer returns a copy, I think we can update this PR to get_scorer(name, **kwargs).

Would we also get the list of params from the signature of the underlying
function? We'll have to test if that works nicely with partials etc.

If it is a callable, I see two options:

1. get_scorer(callable, **kwargs) map directly to partial(callable, **kwargs) and do no checking.
2. Do nothing with **kwargs. (I prefer this option since I prefer to deprecate callable support in get_scorer)

From an API point of view, I prefer to only support strings in get_scorer and error for keys that is not in _SCORERS.

So 2 years later, looking at all the different changes I have the following opinion:

• _Scorer could be made public through a Scorer object and we could have 2 factory methods from_function and from_string to create 2 scorers. This would the proper OOP way.
• If we feel that the above is too much OOP for scikit-learn, it only means that we need to have two functions to call the private factories and do not expose _Scorer.

So I assume that this should be part of the discussion of the potential future scoring API change: #28995