Could you also change https://github.com/scikit-learn/scikit-learn/blob/master/examples/ensemble/plot_feature_transformation.py#L76 to make use of this new method?

I thought we didn't have this method because the base estimator doesn't need to be a tree?

This doesn't consider the base estimator, only the successive trees grown off the gradient of the base estimator.

Ah, I'm sorry, you meant that you can gradient boost other models other than trees. Currently it is hard coded as trees, and I haven't seen any movement towards including arbitrary gradient boosting thus far. Maybe @ogrisel can comment more?

I accidentally muddled this PR with my other PR. I am closing this one and will open a new one.

I thought we didn't have this method because the base estimator doesn't need to be a tree?

I don't know if it's useful to gradient boost other models in practice.

I know XGBoost can use linear models as base learner. However I have a hard time understanding why this is not equivalent to fitting a linear model directly to the loss function. Would love a simple explanation for that. Maybe @pprett knows?

I think it is similar to stacked linear models - by training over different
subsets (in this case residuals, etc.) you can get different results than
just 1 linear model. The subset operation (I think) acts as a weird kind of
non-linearity

On Tue, Sep 8, 2015 at 10:28 AM, Olivier Grisel notifications@github.com
wrote:

I thought we didn't have this method because the base estimator doesn't
need to be a tree?

I don't know if it's useful to gradient boost other models in practice.

I know XGBoost can use linear models as base learner. However I have a
hard time understanding why this is not equivalent to fitting a linear
model directly to the loss function. Would love a simple explanation for
that. Maybe @pprett https://github.com/pprett knows?

—
Reply to this email directly or view it on GitHub
#5222 (comment)
.

The linear model in xgboost is exactly same fitting a linear model to the loss function with parallel coordinate descent. It is implemented in the same interface as gradient boosting, because they are connected in nature.

i.e. Addictive linear model together in gradient boosting way is equivalent to coordinated descent for linear model.

Thanks @tqchen. That sounds different from what we would get by using linear models as base estimators in scikit-learn gb classes though.

Yes, I guess this was because the difference in terms of interface. xgboost's gbm class is a update style interface where choice can be made whether to add estimator, or improve the current estimator based on the statistics. For linear model, this allows inplace modification of the loss function.