I'm happy to see this separate scorer, but I don't think this needs to be a separate function to MSE, merely a parameter. Not sure if others will agree though...

I guess I was thinking of the normalize option in other metrics as comparable.

As discussed at the last sprint (I can't find notes, bad on us): we want to deprecate MSE, and only use 'negated_mse' to always have consistently bigger is better:

https://sourceforge.net/p/scikit-learn/mailman/message/31632671/

So we should have a negated_rmse

@jnothman Thank you for the suggestion. Some people might think these as over-engineering. However, I am concerned that the parameter approach (such as the normalize option) might cause confusion to other people who want to use this method. I'm a fairly open-minded person and I will follow a better suggestion if there is one. Honestly speaking, I think this is the best implementation so far. Anyway, I will always welcome good opinions.

@GaelVaroquaux I remember reading about that discussion. I would suggest the scikit-learn teams to obtain a separate pull request about negated_mse and negated_rmse. If I implement negated_rmse in this pull request, we fall into a dangerous situation where we implement only negated_rmse but not negated_mse. I'll send another pull request about both negated_mse and negated_rmse if time allows.

PS) Sorry for the test failure. :( I'll fix the problem as soon as possible.

@GaelVaroquaux I understand. You have a point.

I’ve decided to implement negated_rmse as you mentioned. But I am still not sure about the amendment that came out during this discussion. Is the following right?

1. Rename the scoring option of cross_val_score, from root_mean_squared_error to negated_rmse.
2. In addition to No.1, set the greater_is_better option to True when calling the make_scorer method. (ex: root_mean_squared_error_scorer = make_scorer(root_mean_squared_error, greater_is_better=True) in scorer.py)

Any updates on this?

I read the following discussion about the negated_mse issue at #2439. Consequently, I decided it was enough to just change the scoring option of cross_val_score, similar to the suggestion by @GaelVaroquaux. I welcome any opinions. :D

hmm, this seems reasonable (from an api design viewpoint). I'll take a look at the code in a few days when I get a chance.

@nelson-liu Sure you can. :D

the implementation lgtm, @GaelVaroquaux do you have any thoughts on the api design?

can you please rebase? and it should be just neg_, not negated_, but yes, that's the way to do it.
Sorry for the slow reply :(

@amueller Sure. :D Wait a moment.

I just finished to rebase this pull request. Are there anythings I can help you with?

I still would rather not see root_mean_squared_error as a separate function, and documented as a separate metric, particularly if that documentation doesn't emphasise why that sqrt transformation is helpful. Note that elsewhere we have options like euclidean_distances(..., squared=False) and accuracy with a normalize parameter. I don't really see how we gain by an effective alias, except in scoring.

Thanks for your opinion. :D

In my understanding, I'm not sure whether we should treat this in the same way as euclidean_distances or not. Usually we can treat both euclidean_distances (squared=True) and euclidean_distances (squared=False) in the same way because it's conceptually equal. (Usually we use squared=False option just for performance optimization) Therefore, although these two methods don't return the same results, we don't need to describe precisely how these two approaches are different.

However, I think the root_mean_squared_error is different. Usually we choose the RMSE when large errors are particularly undesirable. These are conceptually different. Therefore, we'll need to treat these two methods as separate methods and separate documentation. (And you're right. the reason why I have a mind to implement this is for the scoring option of the cross_val_score)

I'll update the documentation as soon as possible, and I'll especially emphasise when this method would be helpful, and why this approach is sometimes better than MSE.

@shaynekang this is a global monotonic transformation, so it doesn't change anything. If there was a sqrt on the inside that would be different. But this has exactly the same properties as mse.
So I tend to agree with @jnothman

• error = mean_squared_error(y_true, y_pred, sample_weight, multioutput)
• return error ** 0.5

btw, is this as good as np.sqrt? It might take a different code path.

I just did timings and, to my surprise, both take the same amount of
time.

Indeed, in terms of model selection, RMSE and MSE are exactly the same thing.

The question is how far down we want to lower the bar for people who don't understand these things, or don't know how to compute a sqrt in Python. My hunch is usually that lowering the bar too much ends up onboarding people that we shouldn't onboard, because they will be only cost and no benefit.

I just did timings and, to my surprise, both take the same amount of time.

https://github.com/numpy/numpy/blob/c90d7c94fd2077d0beca48fa89a423da2b0bb663/numpy/core/src/multiarray/number.c#L522

though explicit is better than implicit ;)

So I get the impression that core dev consensus is something like -1 for a
separate metric, +0.5 for a separate scorer??

On 15 October 2016 at 06:17, Andreas Mueller notifications@github.com
wrote:

I just did timings and, to my surprise, both take the same amount of time.

https://github.com/numpy/numpy/blob/c90d7c94fd2077d0beca48fa89a423
da2b0bb663/numpy/core/src/multiarray/number.c#L522

though explicit is better than implicit ;)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6457 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz6wJVEWnMbXYNNRT1tKFtESAo4L8_ks5qz9VlgaJpZM4HkaRG
.

@jnothman yeah I think so.

@rth, you merged #13467. Am I wrong saying that this PR is no longer needed, then? Could it be closed? Thanks!

Yes, thanks @cmarmo!

Thanks for to all contributors in this PR. Closing as resolved.