1000x agreed

On Thu, Jul 23, 2015 at 12:17 PM, Arnaud Joly notifications@github.com
wrote:

Using grid search with the "mean_square_error", the "mean_absolute_error",
the "median_absolute_error" or the "log_loss"` as scoring parameter lead
to the negation of those metrics.
This is confusing especially for new users.

I suggest that we prefix those strings by "neg-" or "negative_". This
would make clear from the start that the score is obtained from the
negation of the loss / error.

—
Reply to this email directly or view it on GitHub
#5023.

See also #2439.

There is also the solution of flipping the sign back but more people seem to favor the neg- prefix solution.

I'd prefer to flip the sign back (which needs an additional line in GridSearchCV and friends).
As we can't seem to be able to get a consensus for that, I guess we have to go with some prefix like that.

Prefix could just be - of course...

Recently, I met one more people that was bitten by this. This person was very frustrated after loosing hours to understand what was happening.

We could quickly implement both strategies (prefix and flipping sign back) in a branch and try them on an actual use case. This will probably help make our mind.

well we had flipping the sign back in master. The main criticism was that it requires one line of code in GridSearchCV iirc.
I'm not sure implementing the prefix will help us decide, because it is more about user perception. The implementation is pretty straight-forward, right?
The question is just if users will be confused that we don't have mse, but instead negative_mse / -mse

We should A/B test our API :)

well we had flipping the sign back in master

I didn't know, when?

When we introduced the scoring. I thought you removed it in the refactoring.

Strange, I'm my self +1 for flipping the sign back.

My bad, it was @larsmans 393b996#diff-b1caa89fb43c7835879f68a3ec494560

Yeah, that was me... +1 on the neg_ prefix, I think I suggested that somewhere, too.

https://issues.apache.org/jira/browse/SPARK-10097 ^^

https://issues.apache.org/jira/browse/SPARK-10097 ^^

They managed to close the issue in under 24 hours. Lol.

@larsmans why don't you like the other approach?

@jnothman They seem to be slightly more efficient then us lol. But I think they also have a couple of people full time on it.

Because it requires adding attributes on functions, which at the time I deemed to hacky and complicated for custom metrics. However, I'm willing to drop that concern. We need to solve the issue one way or another.

Maybe we can discuss this at the sprint. This is really bad: https://github.com/scikit-learn/scikit-learn/pull/5498/files#diff-ff4438be9ee77932848f5abfcec060efR690

Initially I was in favor of having an explicit flag to state whether higher is better. I think I still prefer this solution over the neg- prefix solution.

We just had a discussion and agreed on the neg- fix

Working on this now.

Working on it with @lvrzhn

@lvrzhn do you have a pull request?

Is anybody on this? If not, I'd like to take a shot at it. Also, is this supposed to be fixed at 0.17x or 0.18? This part of the code changed a lot between those milestones.

David, I'm on it. Will do pull request today.

On Fri, Dec 11, 2015, 01:45 David Thaler notifications@github.com wrote:

Is anybody on this? If not, I'd like to take a shot at it. Also, is this
supposed to be fixed at 0.17x or 0.18? This part of the code changed a lot
between those milestones.

—
Reply to this email directly or view it on GitHub
#5023 (comment)
.

@davidthaler @lvrzhn Could I take up this issue?

it looks like the PR at #6028 had been accidentally closed. I've reopened it on the basis that there was no indication it should be closed. @lvrzhn are you still intending to fix it up?

Ah! Sorry I missed that PR!

+1 from a confused new user

fixed in #7261.