Skip to content

ENH Add mean_pinball_loss metric for quantile regression #19415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 63 commits into from
Feb 18, 2021
Merged

ENH Add mean_pinball_loss metric for quantile regression #19415

merged 63 commits into from
Feb 18, 2021

Conversation

sdpython
Copy link
Contributor

@sdpython sdpython commented Feb 9, 2021

Reference Issues/PRs

Example: Fixes #18911.

What does this implement/fix?

Add function pinball_error as a new regression scoring function used to estimate quantile regression with quantile != 0.5.

@sdpython sdpython changed the title Add pinball_error metrics, fixes issue #18911 [WIP] Add pinball_error metrics, fixes issue #18911 Feb 9, 2021
@sdpython sdpython changed the title [WIP] Add pinball_error metrics, fixes issue #18911 Add pinball_error metrics, fixes issue #18911 Feb 9, 2021
ogrisel
ogrisel reviewed Feb 9, 2021
View reviewed changes
Copy link
Member

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sdpython! Here is a first pass of review comments:

ogrisel
ogrisel reviewed Feb 10, 2021
View reviewed changes
Copy link
Member

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I find the updated example very interesting.

@ogrisel ogrisel changed the title Add pinball_error metrics, fixes issue #18911 Add pinball_loss metric for quantile regression, fixes issue #18911 Feb 11, 2021
@ogrisel
Copy link
Member

ogrisel commented Feb 15, 2021

@sdpython @lorentzenchr I pushed the renaming in my last commits. I think I also addressed most of the pending review comments. Let me know what you think.

@ogrisel ogrisel changed the title Add pinball_loss metric for quantile regression, fixes issue #18911 Add mean_pinball_loss metric for quantile regression, fixes issue #18911 Feb 15, 2021
lorentzenchr
lorentzenchr reviewed Feb 15, 2021
View reviewed changes
Copy link
Member

@lorentzenchr lorentzenchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ogrisel Accompanying you improving examples is a pleasure and a lot of fun. Now I learned how to highlight cells in displayed pandas tables:smiley:

@ogrisel
Copy link
Member

ogrisel commented Feb 15, 2021

Now I learned how to highlight cells in displayed pandas tables😃

I googled after your comment and found this solution on stackoverflow :)

@ogrisel ogrisel requested a review from lorentzenchr February 16, 2021 17:48
lorentzenchr
lorentzenchr approved these changes Feb 16, 2021
View reviewed changes
Copy link
Member

@lorentzenchr lorentzenchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ogrisel @lorentzenchr
Update sklearn/metrics/tests/test_regression.py
7820618 
Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
@lorentzenchr
Copy link
Member

The CI failure of test_imports_strategies in macOS pylatest_conda_forge_mkl seems unrelated. Therefore, I dare to merge...

@lorentzenchr lorentzenchr changed the title Add mean_pinball_loss metric for quantile regression, fixes issue #18911 ENH Add mean_pinball_loss metric for quantile regression Feb 18, 2021
@lorentzenchr lorentzenchr merged commit 6a6217f into scikit-learn:main Feb 18, 2021
@lorentzenchr lorentzenchr mentioned this pull request Feb 18, 2021
Open
4 tasks
@ogrisel
Copy link
Member

ogrisel commented Feb 19, 2021 •
edited
Loading

Thanks for the PR @sdpython! and @lorentzenchr for the reviews :)

@glemaitre glemaitre mentioned this pull request Apr 22, 2021
Merged
12 tasks
@ghost
Copy link

ghost commented May 26, 2021

Should this be median pinball loss for alpha=0.5/tau=0.5 (as per https://www.tensorflow.org/addons/api_docs/python/tfa/losses/pinball_loss)?

@lorentzenchr
Copy link
Member

This PR implemented the mean of the pinball loss as a metric, which equals 1/2 * absolute loss for alpha=0.5 (which elicits the median). I think the tensorflow fomula has a typo (i.e. is incorrect).

@airmilesabdullah
Copy link

hi there, I noticed that when my pandas dataframe has more than 10,000 rows, I get an out-of-memory error. Why does this occur and is there a way around it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogrisel ogrisel ogrisel approved these changes

@glemaitre glemaitre glemaitre left review comments

@lorentzenchr lorentzenchr lorentzenchr approved these changes

Assignees
No one assigned
Labels
module:ensemble module:metrics Waiting for Reviewer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add quantile loss as metric
6 participants
@sdpython @ogrisel @lorentzenchr @GaelVaroquaux @airmilesabdullah @glemaitre