Content related: do you think as the example is now people can understand what it is meant to show (if they didn't already know)? I was thinking of putting a comment by the gradient boosting part to say "this is the interesting bit, using a tree's apply method".

Build related:
Any ideas why a lot of the gaussian process tests are failing in the py3.4 build?

In a previous build a test in test_ridge.py failed (https://ci.appveyor.com/project/sklearn-ci/scikit-learn/build/1.0.1287/job/wdqjboo3dvnw6lk4) which looks like a Heisenbug?

Should I file an issue for those or are they known to be strange?

I think what I meant in #4549 was show a PCA on these vs on the original data? I'm not entirely following my wording, though.

Thanks for working on this!

Can you post the plot here?
Also, it would be good to reference this in the user guide.

Undecided if zooming in on the top left (interesting) part is more helpful or confusing for others.

edit: saving the image while the figure is still open helps ...

the image looks white to me...

Updated the plot, and added a reference in the ensemble section where RandomTreesEmbedding is discussed. I looked in "Dataset transformations" but couldn't find an existing topic where this would fit there. Somehow expected to see tree based transformations there as well.

If you can't quite remember the PCA comment, and I can't work it out either should we skip it/make a second example?

looks good. Maybe it would be interesting to compare training the lr on the same training set vs a hold-out set? or at least mention that?

You mean like this (just showing the RF part here but I changed it for all of the models):

X, y = make_classification(n_samples=80000)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
X_train, X_train_lr, y_train, y_train_lr = train_test_split(X, y, test_size=0.5)

...
# Supervised transformation based on random forests
rf = RandomForestClassifier(max_depth=3, n_estimators=n_estimator)
rf_enc = OneHotEncoder()
rf_lm = LogisticRegression()
rf.fit(X_train, y_train)
rf_enc.fit(rf.apply(X_train))
rf_lm.fit(rf_enc.transform(rf.apply(X_train_lr)), y_train_lr)

...

Hard to tell if it makes a difference. What is the idea behind using a different dataset for fitting the lr instead of using the same as for fitting the trees? We can change it or add it, but you'll have to provide the sentence to motivate it as I don't know enough ;)

well the idea would be to be closer to "stacking". Intuitively training both on the same set should lead to crazy overfitting. but maybe not. let's keep it simple.

Ok. I looked at 8.8 of ESL but I'll have to read it a few more times before it sinks in. Changed the example to use different subsets for fitting of the trees and the LR model.

Well, I'm not entirely sure what the "industry standard" is.

Updated ROC curves. After discussing with @glouppe this morning I think it makes sense to split the training samples again. Updated the comment a bit to explain why/when this is important.

Showing inverse false positive rate on a log scale vs true positive rate will "enhance" the difference:

Also changed n_samples to 10k, n_estimators to 20, and random seed to 10.

plt.plot(tpr_rt_lm, 1 / fpr_rt_lm, label='RT + LR')
plt.plot(tpr_rf, 1 / fpr_rf, label='RF')
plt.plot(tpr_rf_lm, 1 / fpr_rf_lm, label='RF + LR')
plt.plot(tpr_grd, 1 / fpr_grd, label='GBT')
plt.plot(tpr_grd_lm, 1 / fpr_grd_lm, label='GBT + LR')
plt.yscale('log')
plt.ylabel('Inverse false positive rate')
plt.xlabel('True positive rate')
plt.title('ROC curve')
plt.legend(loc='best')
plt.show()

With "enhanced" differences I would be a bit afraid of sending a message like "this is always better than that" when this is a very zoomed-in view of a single run on a particular synthetic dataset.

(and currently it compares against a random forest using only part of the data, right?)

I have a slight preference for the plain ROC curve. The original intent of the example was to show how to use the individual tree's apply method. (Though @glouppe was surprised that GradientBoostedClassifier does not have a public apply)

Right now it compares RandomTreesEmbedding, RandomForestClassifier and GradientBoostedClassifier. Each of these is used as input for a LogisticRegression which is then used to classify. For the last two it also shows the performance without the LR step. The trees and LR are trained on different subsets of the data. For the comparison without the LR we could retrain the tree ensembles on the whole training set...though that makes it feel like a lesson on whether or not you should use this technique, which was not the original intent.

Shouldn't this at least link to the RandomTreesEmbedding docs?

Yeah, probably. I don't think we usually link from examples to docs. You can always click on the class to get to the api doc.

By link I meant at least mention it :) This seems to be doing the same thing that estimator does, except with more control over the type of trees.

It is so similar that we even use RandomTreesEmbedding in the example (RT+LR in the legend) :) What do you think of this modification?

I think the example is good enough to be merged. It is a nice demonstration of the apply method of tree-based methods. +1

Thanks!

Thanks! Now back to walking in the swiss alps: 🗻