[MRG] FEA: Stacking estimator for classification and regression #11047

glemaitre · 2018-05-01T15:38:25Z

Reference Issues/PRs

closes #4816
closes #8960
closes #7427
closes #6674

requires #14305 to be merged.

What does this implement/fix? Explain your changes.

Implement 2 meta-estimators to perform stacking (classification and regression problems).

Any other comments?

glemaitre · 2018-05-01T16:29:50Z

@jnothman @amueller @GaelVaroquaux It is what I had in mind.

sklearn/ensemble/tests/test_stacking.py

glemaitre · 2018-05-01T19:25:16Z

‎Example and doc coming very soon :D

chkoar · 2018-05-06T12:31:27Z

sklearn/ensemble/_stacking.py

+        self.estimators = estimators
+        self.meta_estimator = meta_estimator
+        self.cv = cv
+        self.method_estimators = method_estimators


I believe that method_estimators is not obvious name. output_confidence, metafeatures or something like that could be more targeted. In case of classification, if output_confidence is True the the level 1 estimator should be trained on the (cross-validated or not) outputs of predict_proba or decision_function of level 0 estimators else the level 1 estimator should be trained on prediction outputs. Another option could be a parameter like metafeatures with several options like:

predictions

confidences

cv_predictions

cv_confidences

Of course we could use both metafeatures and the original space to train the level 1 estimator.

As for the confidence outputs an option could be to aggregate the outputs using a descriptive statistic like mean or median. So the dimensionality of the metafeatures will be the same as the number of classes.

What the use case in which we want to avoid the CV when fitting the final estimator.
method_estimators is the closest keywords that I found to be in accordance with the method keyword in the other estimator (in grid-search I think)

How about predict_method?

chkoar · 2018-05-06T12:35:32Z

It is very nice that you started this @glemaitre. If you need any help don't hesitate to ping me.

glemaitre · 2018-05-06T22:11:45Z

It is very nice that you started this @glemaitre. If you need any help don't hesitate to ping me.

Actually, it was some good PR which allow us to fix some issues. I would appreciate if you can make a review :)

glemaitre · 2018-05-06T22:14:30Z

@amueller I drafted the documentation as well as an example. I hope that it will motivate you to give a go for the review ;)

NB: we should improve the example once the ColumnTransformer is merged. There is a huge case in which we applied different classifier on different column and use that as estimators. This is more the use case that I am used too actually.

glemaitre · 2018-05-07T10:05:37Z

Artifacts for the documentation:

I see that there some glitches to be corrected.

TomDLT

I directly pushed minor improvements in the example.

Your implementation does not allow to pass X unchanged to the final estimator, does it?

sklearn/ensemble/_stacking.py

doc/whats_new/v0.20.rst

sklearn/ensemble/_stacking.py

glemaitre · 2018-05-23T13:47:25Z

Your implementation does not allow to pass X unchanged to the final estimator, does it?

It does not allow it. What do you propose. I see 2 solutions:

Create a keyword "passthrough=True/False" and stack X accordingly
Create a strategy "input" for the DummyRegressor which will return the input and can be added to the estimators list.

The first one is less complicated to discover as a user I think. WDYT?

TomDLT · 2018-05-23T13:52:59Z

The first one is less complicated to discover as a user I think.

I agree.

TomDLT · 2018-05-23T13:56:10Z

Don't forget the .. versionadded:: 0.20 label in both new classes docstrings.

rth · 2018-05-23T17:16:19Z

Your implementation does not allow to pass X unchanged to the final estimator, does it?

I'm not using stacking extensively, but IMHO this PR is complex enough; since passing X to the final estimator may not be the primary reason people would use stacking, maybe this feature can be proposed in a separate PR?

thomasjpfan · 2019-09-18T03:33:36Z

Currently this PR is dropping when the stack_method is predict_proba, which applies to binary and multiclass.

qinhanmin2014 · 2019-09-18T07:45:23Z

Regarding dropping the column when stack_method='predict_proba' should we control which class to drop?

I think we can have such an option, but we should not drop any columns by default.

glemaitre · 2019-09-18T08:29:13Z

I am not convinced that not dropping is a good default.

glemaitre · 2019-09-18T10:12:05Z

I added the drop parameter.

glemaitre · 2019-09-18T10:16:53Z

I am still having doubt regading the dropping. We are discussing which estimator should benefit or not from it. I am still convinced that dropping would be best because apart of the tree which might be using this info (and I am not convinced about it as mentoned #11047 (comment)), ~~it is really likely that user have linear and non-linear models in estimators and therefore dropping will be better than not dropping~~(this is really wrong)

qinhanmin2014 · 2019-09-18T10:49:57Z

I still prefer not to drop the first column by default, but since we've introduced drop parameter and the default final_estimator is linear model, I think that's acceptable.

qinhanmin2014 · 2019-09-18T10:58:32Z

Actually, apart from tree-based models, I'm not sure whether co-linear features are useful for regularized linear models (they're certainly not useful for unregularized linear models).
But I tend to believe that the influence is small, this is why I approve this PR.

jnothman

I'm really not sure about the importance of this dropping business. Without evidence to the contrary, I'd say YAGNI: dropping one of two columns in binary classification is obviously justified, for any final estimator, but otherwise it's unclear to me that we are providing the user with a useful parameter.

@glemaitre writes that "it is really likely that user have linear and non-linear models in estimators and therefore dropping will be better than not dropping". I don't get why having a particular type of estimator in estimators matters. Isn't the dropping all about what the final estimator does?

doc/whats_new/v0.22.rst

sklearn/ensemble/_stacking.py

glemaitre · 2019-09-18T11:20:56Z

Isn't the dropping all about what the final estimator does?

Completely true. Wrong reasoning on my side.

jnothman · 2019-09-18T11:40:27Z

Should we resolve the drop decision at the monthly meeting on Monday?

ogrisel · 2019-09-18T11:46:15Z

+1 for always dropping for binary classifiers and never dropping for multiclass to keep things simple. The other cases are YAGNI in my opinion. No need for an option.

jnothman · 2019-09-18T11:56:00Z

That's how I'm feeling too. Happy to see an option added later with empirical support

qinhanmin2014 · 2019-09-18T12:25:01Z

+1 for always dropping for binary classifiers and never dropping for multiclass to keep things simple. The other cases are YAGNI in my opinion. No need for an option.

I think this is a better solution. So there's +3, maybe enough?

qinhanmin2014 · 2019-09-18T12:42:29Z

examples/ensemble/plot_stack_predictors.py

+###############################################################################
+# Stack of predictors on a single data set
+###############################################################################
+# It is sometimes tedious to find the model which will best perform on a given


And somehow, I think the logic here is strange. I'm still not sure whether it's good to throw "bad" estimators to a stacker. It's true that a stacker can somehow filter "bad" estimators (e.g., by giving lower weight to them), but I think these "bad" estimators will still influence the performance of a stacker.
I think one should use cross validation to check the performance of all the estimators and throw some "good" estimators to a stacker to produce a "better" estimator.
But I'm not sure and I think this example is acceptable.

I agree. Stacking is mostly useful to slightly improve the accuracy by combining the strength of good models (hopefully diverse enough to get not too correlated errors).

Throwing in bad models is likely to negatively impact the performance of the ensemble (and it's computationally wasteful).

OK, so I replaced AdaBoostRegressor and KNNRegressor by HistGradientBoostingRegressor and RandomForestRegressor to have only strong learner and changed the conclusion slightly. I will add a bit of timing as well to show that training a stack learner is computationally expensive

glemaitre · 2019-09-18T12:42:43Z

Works for me

jnothman · 2019-09-18T20:06:20Z

Let's do this... Thanks to @caioaao and @glemaitre both for great conception of these competing APIs and for persistence in finding a solution we all can accept!!

GaelVaroquaux · 2019-10-03T03:43:30Z

Woooot! I hadn't seen that the merge had happened! This is great! Congratulations and thank you to all involved.

wderose · 2019-11-23T02:16:50Z

@caioaao @glemaitre : Thank you for this awesome feature. Should the stack_method="auto" priority match that of CalibratedClassifierCV?

decision_function -> proba -> predict

Instead of

proba -> decision_function -> predict

jnothman · 2019-11-23T10:46:14Z

That's a good question. Open a new issue so that we can consider it before release?

wderose · 2019-11-24T06:15:16Z

Raised the predict-function alignment issue in #15711.

glemaitre added 4 commits May 1, 2018 12:56

iter

fe4302c

iter

6bb88ef

iter

91d2415

EHN make base class for regressor classifier

6771d78

amueller reviewed May 1, 2018

View reviewed changes

sklearn/ensemble/tests/test_stacking.py Outdated Show resolved Hide resolved

glemaitre added 4 commits May 2, 2018 01:35

DOC add class documentation

002fee2

TST improve tests

7c2c509

TST improve tests

6f87761

TST finish drafting test

e8ddb07

chkoar reviewed May 6, 2018

View reviewed changes

glemaitre added 4 commits May 6, 2018 21:35

EXA add example

96f512c

DOC add whats new and api documentation

ac3f265

DOC add user guide doumentation

ca37e8d

TST tag as meta estimator for common test

91c28b6

glemaitre changed the title ~~[WIP] Stacking estimator for classification and regression~~ [MRG] Stacking estimator for classification and regression May 6, 2018

TST sort outuput of dict

673337c

glemaitre and others added 2 commits May 7, 2018 12:11

DOC fix current module

e0f49be

Improve figure rendering

c14c18b

TomDLT reviewed May 23, 2018

View reviewed changes

iter

fb2c5b2

qinhanmin2014 approved these changes Sep 18, 2019

View reviewed changes

jnothman reviewed Sep 18, 2019

View reviewed changes

doc/whats_new/v0.22.rst Outdated Show resolved Hide resolved

sklearn/ensemble/_stacking.py Outdated Show resolved Hide resolved

qinhanmin2014 reviewed Sep 18, 2019

View reviewed changes

revert back to dropping only binary

8599450

glemaitre force-pushed the is/4816 branch from de5316c to 8599450 Compare September 18, 2019 13:25

glemaitre added 5 commits September 18, 2019 15:37

update example

3af0deb

PEP8 + timing

a192cf8

docstring fix

a656b9e

PEP8

41851dd

docstring

9134f7f

jnothman merged commit bab5926 into scikit-learn:master Sep 18, 2019

thomasjpfan mentioned this pull request Apr 24, 2022

[MRG] Refactor pipeline namespace to make it more reusable #11446

Closed

[MRG] FEA: Stacking estimator for classification and regression #11047

[MRG] FEA: Stacking estimator for classification and regression #11047

Conversation

glemaitre commented May 1, 2018 • edited Loading

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

glemaitre commented May 1, 2018

glemaitre commented May 1, 2018 via email

chkoar May 6, 2018

Choose a reason for hiding this comment

glemaitre May 6, 2018

Choose a reason for hiding this comment

jnothman Mar 11, 2019

Choose a reason for hiding this comment

chkoar commented May 6, 2018

glemaitre commented May 6, 2018

glemaitre commented May 6, 2018

glemaitre commented May 7, 2018

TomDLT left a comment

Choose a reason for hiding this comment

glemaitre commented May 23, 2018

TomDLT commented May 23, 2018

TomDLT commented May 23, 2018

rth commented May 23, 2018

thomasjpfan commented Sep 18, 2019

qinhanmin2014 commented Sep 18, 2019

glemaitre commented Sep 18, 2019

glemaitre commented Sep 18, 2019

glemaitre commented Sep 18, 2019 • edited Loading

qinhanmin2014 commented Sep 18, 2019

qinhanmin2014 commented Sep 18, 2019

jnothman left a comment

Choose a reason for hiding this comment

glemaitre commented Sep 18, 2019

jnothman commented Sep 18, 2019 via email

ogrisel commented Sep 18, 2019

jnothman commented Sep 18, 2019 via email

qinhanmin2014 commented Sep 18, 2019

qinhanmin2014 Sep 18, 2019

Choose a reason for hiding this comment

ogrisel Sep 18, 2019

Choose a reason for hiding this comment

glemaitre Sep 18, 2019

Choose a reason for hiding this comment

glemaitre commented Sep 18, 2019

jnothman commented Sep 18, 2019

GaelVaroquaux commented Oct 3, 2019 via email

wderose commented Nov 23, 2019

jnothman commented Nov 23, 2019 via email

wderose commented Nov 24, 2019

glemaitre commented May 1, 2018 •

edited

Loading

glemaitre commented Sep 18, 2019 •

edited

Loading