Skip to content

MAINT DOC HGBT leave updated if loss is not smooth #26254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 8, 2023
Merged

MAINT DOC HGBT leave updated if loss is not smooth #26254

merged 2 commits into from
Aug 8, 2023

Conversation

lorentzenchr
Copy link
Member

@lorentzenchr lorentzenchr commented Apr 21, 2023 •
edited
Loading

Reference Issues/PRs

Popped up while working on #25964.

What does this implement/fix? Explain your changes.

HGBT leave updates now rely on loss.differentiable and the reasons and differences to the standard gradient boosting algo are explained.

Any other comments?

It is hard to find a reference for gradient boosting with 2nd order loss approximation (using hessians) and non-smooth losses.
Edit: https://arxiv.org/abs/1808.03064 explicitly considers the different boosting schemes and mentions the problem of non-smooth loss functions with Newton boosting.

@lorentzenchr lorentzenchr added the Quick Review For PRs that are quick to review label Apr 21, 2023
@glevv
Copy link
Contributor

glevv commented Apr 21, 2023 •
edited
Loading

It is hard to find a reference for gradient boosting with 2nd order loss approximation (using hessians) and non-smooth losses.

Don't know if that's what you're after, but maybe xgboost and hinge
https://github.com/dmlc/xgboost/blob/master/src/objective/hinge.cu
or lightgbm and l1
https://github.com/microsoft/LightGBM/blob/ce0813efea9a12e543ce8feca4952407e1d05e86/src/objective/cuda/cuda_regression_objective.cu#L136
They obviously don't use "real" hessians.

I think catboost also has a lot of non smooth loss functions, but they don't use 2nd order optimization algorithms for them.

thomasjpfan
thomasjpfan approved these changes Apr 29, 2023
View reviewed changes
Copy link
Member

@thomasjpfan thomasjpfan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lorentzenchr
Copy link
Member Author

@glevv Do you want to give this PR a review?

@glevv
Copy link
Contributor

glevv commented May 4, 2023 •
edited
Loading

@lorentzenchr Sorry, I don't think I will be able to

@lorentzenchr
Copy link
Member Author

@glevv No problem. I thought I just ask as you seemed interested in the PR:smirk:

@lorentzenchr lorentzenchr added this to the 1.3 milestone Jun 1, 2023
@jeremiedbb jeremiedbb modified the milestones: 1.3, 1.4 Jul 6, 2023
OmarManzoor
OmarManzoor approved these changes Aug 8, 2023
View reviewed changes
Copy link
Contributor

@OmarManzoor OmarManzoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @lorentzenchr

@OmarManzoor OmarManzoor merged commit ed01199 into scikit-learn:main Aug 8, 2023
@lorentzenchr lorentzenchr deleted the hgbt_doc_smooth_loss branch August 8, 2023 17:00
TamaraAtanasoska pushed a commit to TamaraAtanasoska/scikit-learn that referenced this pull request Aug 21, 2023
@lorentzenchr @TamaraAtanasoska
MAINT DOC HGBT leave updated if loss is not smooth (scikit-learn#26254)
3ebc8e7
REDVM pushed a commit to REDVM/scikit-learn that referenced this pull request Nov 16, 2023
@lorentzenchr @REDVM
MAINT DOC HGBT leave updated if loss is not smooth (scikit-learn#26254)
5f63946
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thomasjpfan thomasjpfan thomasjpfan approved these changes

@OmarManzoor OmarManzoor OmarManzoor approved these changes

Assignees
No one assigned
Labels
Documentation module:ensemble Quick Review For PRs that are quick to review
Projects
None yet
Milestone
1.4
Development

Successfully merging this pull request may close these issues.
5 participants
@lorentzenchr @glevv @thomasjpfan @OmarManzoor @jeremiedbb