Skip to content

Fixing MAE and MSE errors. should not be negative when used in cross_val #2823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

arahuja
Copy link

@arahuja arahuja commented Feb 4, 2014

Setting greater_is_better=True causes the sign in _PredictScorer to be negative. This leads to negative values when using LinearRegression in cross_val_score

Alternatively, we could adjust the return statement in _PredictScorer.call
return self._sign * self._score_func(y_true, y_pred, **self._kwargs)

Also, I found this mailthread on the issue: https://www.mail-archive.com/scikit-learn-general@lists.sourceforge.net/msg08945.html

That mailthread implies that this is not the correct fix, but I wanted to discuss how to correct this.

@jnothman
Copy link
Member

jnothman commented Feb 4, 2014

This is not a good solution. Those scorers provide an objective function
for grid search, and are standardised to report greater-is-better metrics.

On 4 February 2014 15:22, arahuja notifications@github.com wrote:

Setting greater_is_better=True causes the sign in _PredictScorer to be
negative. This leads to negative values when using LinearRegression in
cross_val_score

Alternatively, we could adjust the return statement in PredictScorer.
call_
return self._sign * self._score_func(y_true, y_pred, **self._kwargs)

Also, I found this mailthread on the issue:
https://www.mail-archive.com/scikit-learn-general@lists.sourceforge.net/msg08945.html

That mailthread implies that this is not the correct fix, but I wanted to

discuss how to correct this.

You can merge this Pull Request by running

git pull https://github.com/arahuja/scikit-learn regression-scorer-fix

Or view, comment on, or merge it at:

#2823
Commit Summary

  • Fixing MAE and MSE errors. should not be negative when used in
    cross_val_score

File Changes

Patch Links:

Reply to this email directly or view it on GitHubhttps://github.com//pull/2823
.

@mblondel
Copy link
Member

mblondel commented Feb 4, 2014

This is a known issue, see #2439.

But, indeed, your changes are not the right fix: they will result in grid search selecting the models with worst MSE.

So, closing this PR.

@mblondel mblondel closed this Feb 4, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants