ENH Monotonic Contraints for Tree-based models #13649

samronsin · 2019-04-15T22:29:01Z

Continuation of PR #7266 addressing issue #6656 and #18982.

TODO:

~~implement logic for new trees in [MRG+2] Faster Gradient Boosting Decision Trees with binned features #12807~~ (already done by [MRG] Monotonic constraints for GBDT #15582)
discuss API (not sure about introducing 2 new kwargs increasing and decreasing) => let's go with the montonic_cst array
add tests
improve tests as the current ones don't fail despite erroneous implementation

DecisionTrees RandomForests GradientBoosting

NicolasHug · 2019-04-15T22:46:59Z

Very interested in this, please ping me when you have something!

sklearn/ensemble/forest.py

samronsin · 2019-06-17T16:47:16Z

I added a few tests to check that the implementation actually enforced the monotonicity constraints, but they are failing...
Unless my tests are wrong, this suggests that the current implementation is not correct.

samronsin · 2019-06-20T17:31:32Z

Two issues:

montonicity for classification is with respect to class 0 => switch to class 1 ?
a small fraction of rows (~5% for 1000 rows, which is hidden in the current tests...) fail to satisfy the monotonicity constraints -- I suspect this could be due to unfortunate float64 to float32 conversions at during the predict call

NicolasHug · 2020-01-29T12:58:10Z

@samronsin LMK when you need reviews.

I've implemented monotonic constraints for the hist-GBDTs in #15582

It's still under review so your comments are more than welcome there. I think you'll also find some tests worth using. They helped me a great deal in debugging my code

NicolasHug · 2020-01-29T13:05:59Z

montonicity for classification is with respect to class 0 => switch to class 1 ?

Yes. Maybe this be done by just reversing the constraint? Not sure

a small fraction of rows (~5% for 1000 rows, which is hidden in the current tests...) fail to satisfy the monotonicity constraints -- I suspect this could be due to unfortunate float64 to float32 conversions at during the predict call

You don't seem to be constraining the values of the children based on the average value of a pair of siblings. You'll need to introduce lower and upper bounds to the nodes values for monotonic constraints to be properly enforced (see https://github.com/scikit-learn/scikit-learn/pull/15582/files#diff-9bd2ee07fb6817a0aee54f959d32de8aR139).

ogrisel

LGTM! Thank you very much @samronsin!

sklearn/ensemble/_forest.py

sklearn/tree/_criterion.pyx

… supported

Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>

jjerphan

LGTM! Thank you for your notable contribution, your persistence, and your patience, @samronsin.

@lorentzenchr: are you interested in having a look at this PR before merging it?

samronsin · 2023-06-25T12:06:55Z

Thank you @jjerphan, @ogrisel and @NicolasHug for your invaluable help on this PR ! Very happy to see this work landing eventually !

jjerphan · 2023-06-27T06:30:01Z

At yesterday's monthly meeting, we mentioned that it would be appropriate to test data structure consistency between HGBT's and DecisionTree's when constraints of monotonicity are used.

I think we better do it in a subsequent PR.

I will merge this PR on Friday if no-one objects in the meantime.

lorentzenchr

Some minor comments.

doc/whats_new/v1.4.rst

sklearn/ensemble/_forest.py

sklearn/tree/_classes.py

…trees

jjerphan · 2023-06-30T17:31:26Z

I would wait for @lorentzenchr's approval before merging.

jjerphan · 2023-07-01T12:18:15Z

🎉

Co-authored-by: Pat O'Reilly <patrick.oreilly256@gmail.com> Co-authored-by: dsleo <leooleds@gmail.com> Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com> Co-authored-by: Julien Jerphanion <git@jjerphan.xyz>

pat-oreilly and others added 9 commits September 2, 2016 16:49

Monotonic features in tree-based models

1b0a960

DecisionTrees RandomForests GradientBoosting

refactoring monotonic check into a function

953651f

tidy up monotonic check function

63c3f9c

fix typo 'decreasin'

6407e99

Merge branch 'master' into monotonic-trees

92f7537

Clean up dirty merge

119d5a3

PEP8

c331e82

PEP8

ab6d185

PEP8

ce9377f

jnothman reviewed Apr 15, 2019

View reviewed changes

sklearn/ensemble/forest.py Outdated Show resolved Hide resolved

samronsin added 6 commits April 25, 2019 23:25

Update docstrings for increasing/decreasing parameters

fe21579

Merge branch 'master' into monotonic-trees

642dcb0

Cleanup

544cced

Cleanup

20c7929

Merge branch 'master' into monotonic-trees

8cd9954

Add basic monotonicity tests

49747c6

samronsin added 5 commits June 17, 2019 18:48

PEP8

37b536b

PEP8

ef2fb1c

Simplify split_passes_monotonic_check function

072ef99

Fix split_passes_monotonic_check function

605e1c8

Fix tests (still something broken though)

018d6c2

amueller added the Needs work label Aug 6, 2019

samronsin added 3 commits January 29, 2020 11:30

Merge branch 'master' into monotonic-trees

347e2c6

Fix PEP8 and missing import in tests

b217ec8

Fix faulty Splitter call in _classes.py

be8b38a

samronsin force-pushed the monotonic-trees branch from 6303f8c to 65c0ad0 Compare June 22, 2023 11:59

Mark missing values as unsupported

16a8863

samronsin force-pushed the monotonic-trees branch from 65c0ad0 to 16a8863 Compare June 22, 2023 12:05

ogrisel approved these changes Jun 22, 2023

View reviewed changes

sklearn/ensemble/_forest.py Outdated Show resolved Hide resolved

sklearn/tree/_criterion.pyx Show resolved Hide resolved

samronsin and others added 3 commits June 22, 2023 18:45

Update docstrings to specify training data with missing values is not…

c08c4bf

… supported

DOC Add changelog entries

e77c505

Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>

Merge branch 'main' into monotonic-trees

75242d9

jjerphan approved these changes Jun 24, 2023

View reviewed changes

lorentzenchr reviewed Jun 28, 2023

View reviewed changes

jjerphan reviewed Jun 28, 2023

View reviewed changes

sklearn/tree/_classes.py Outdated Show resolved Hide resolved

samronsin added 6 commits June 30, 2023 10:27

Add comment regarding monotonic_cst sign flipping for classification …

9b3bc22

…trees

Use np.isin to validate monotonic_cst

816778d

Update whats_new

88ae965

Fix lign too long

31d5edb

Merge branch 'main' into monotonic-trees

e763020

Add pointers to user guide

57c78a5

lorentzenchr approved these changes Jul 1, 2023

View reviewed changes

lorentzenchr merged commit b88b539 into scikit-learn:main Jul 1, 2023

lorentzenchr mentioned this pull request Sep 6, 2023

Monotonicity constraints for GradientBoostingClassifier and GradientBoostingRegressor #27305

Closed

Charlie-XIAO mentioned this pull request Oct 2, 2023

DOC GradientBoosting* will not implement monotonic constraints, use HistGradientBoosting* instead #27516

Merged

This was referenced Oct 20, 2023

Monotonic trees missing values #27630

Draft

FIX force node values outside of [0, 1] range for monotonically constraints classification trees #27639

Merged

samronsin deleted the monotonic-trees branch June 6, 2024 10:28

sebp mentioned this pull request Aug 18, 2024

Monotonicity constraint in Random Forest survival models sebp/scikit-survival#474

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH Monotonic Contraints for Tree-based models #13649

ENH Monotonic Contraints for Tree-based models #13649

samronsin commented Apr 15, 2019 •

edited by lorentzenchr

Loading

NicolasHug commented Apr 15, 2019

samronsin commented Jun 17, 2019

samronsin commented Jun 20, 2019

NicolasHug commented Jan 29, 2020

NicolasHug commented Jan 29, 2020 •

edited

Loading

ogrisel left a comment

jjerphan left a comment

samronsin commented Jun 25, 2023

jjerphan commented Jun 27, 2023

lorentzenchr left a comment

jjerphan commented Jun 30, 2023

jjerphan commented Jul 1, 2023

ENH Monotonic Contraints for Tree-based models #13649

ENH Monotonic Contraints for Tree-based models #13649

Conversation

samronsin commented Apr 15, 2019 • edited by lorentzenchr Loading

NicolasHug commented Apr 15, 2019

samronsin commented Jun 17, 2019

samronsin commented Jun 20, 2019

NicolasHug commented Jan 29, 2020

NicolasHug commented Jan 29, 2020 • edited Loading

ogrisel left a comment

Choose a reason for hiding this comment

jjerphan left a comment

Choose a reason for hiding this comment

samronsin commented Jun 25, 2023

jjerphan commented Jun 27, 2023

lorentzenchr left a comment

Choose a reason for hiding this comment

jjerphan commented Jun 30, 2023

jjerphan commented Jul 1, 2023

samronsin commented Apr 15, 2019 •

edited by lorentzenchr

Loading

NicolasHug commented Jan 29, 2020 •

edited

Loading