This might benefit from a mention in the narrative docs too.

I updated the documentation of the with_std option as proposed and fixed the bug in BaggingRegressor. Regarding the narrative docs: any opinions on where this could be added?

Coverage decreased (-0.02%) when pulling 1780535 on jmetzen:variance_ensemble_regressors into 57f67d0 on scikit-learn:master.

@glouppe I extended the example (quantitative evaluation based on MSE and log probability under predictive distribution). The main point of the example is an illustration of the quite different predictive distributions of the ensemble-based regressors and the Gaussian Process.

@mblondel is this what you had in mind in issue #3271?

Coverage decreased (-0.03%) when pulling 6043345 on jmetzen:variance_ensemble_regressors into 57f67d0 on scikit-learn:master.

Could you show what the new example looks like for the lazy reviewers like me? :)

Sure, here is the generated figure and the terminal output

Comparison of predictive distributions of different regressors

A simple one-dimensional, noisy regression problem adressed by three different
regressors:

1. A Gaussian Process
2. A Random Forest
3. A Bagging-based Regressor

The regressors are fitted based on noisy observations where the magnitude of
the noise at the different training point is constant and known. Plotted are
both the mean and the pointwise 95% confidence interval of the predictions.
The mean predictions are evaluated on noise-less test data using the mean-
squared-error. The mean log probabilities of the noise-less test data are used
to evaluate the predictive distributions (a normal distribution with the
predicted mean and standard deviation) of the three regressors.

This example is based on the example gaussian_process/plot_gp_regression.py.

Mean-squared error of predictors on 1000 equidistant noise-less test datapoints:
* Random Forest: 0.78
* Bagging: 0.71
* Gaussian Process: 0.57
Mean log-probability of 1000 equidistant noise-less test datapoints under the (normal) predictive distribution of the predictors, i.e., log N(y_true| y_pred_mean, y_pred_std) [less is better]:
* Random Forest: -12.44
* Bagging: -14.20
* Gaussian Process: -26.62
In summary, the mean predictions of the Gaussian Process are slightly better than those of Random Forest and Bagging. The predictive distributions (taking into account also the predictive variance) of the Gaussian Process are considerably better.

I am still not convinced by this example. It is misleading in my opinion.
The mean squared error and the prediction variance of the individual trees
should not mistaken with the mean squared error and the prediction variance
of the forest.

On 15 September 2014 18:47, Jan Hendrik Metzen notifications@github.com
wrote:

Sure, here is the generated figure and the terminal output
[image: figure_1]
https://cloud.githubusercontent.com/assets/1116263/4274869/7a2d818a-3cf7-11e4-9ff5-18048825ef94.png
Comparison of predictive distributions of different regressors

A simple one-dimensional, noisy regression problem adressed by three
different
regressors:

1. A Gaussian Process
2. A Random Forest
3. A Bagging-based Regressor

The regressors are fitted based on noisy observations where the magnitude
of
the noise at the different training point is constant and known. Plotted
are
both the mean and the pointwise 95% confidence interval of the predictions.
The mean predictions are evaluated on noise-less test data using the mean-
squared-error. The mean log probabilities of the noise-less test data are
used
to evaluate the predictive distributions (a normal distribution with the
predicted mean and standard deviation) of the three regressors.

This example is based on the example
gaussian_process/plot_gp_regression.py.

Mean-squared error of predictors on 1000 equidistant noise-less test
datapoints:

• Random Forest: 0.78
• Bagging: 0.71
• Gaussian Process: 0.57
  Mean log-probability of 1000 equidistant noise-less test datapoints under
  the (normal) predictive distribution of the predictors, i.e., log N(y_true|
  y_pred_mean, y_pred_std) [less is better]:
• Random Forest: -12.44
• Bagging: -14.20
• Gaussian Process: -26.62
  In summary, the mean predictions of the Gaussian Process are slightly
  better than those of Random Forest and Bagging. The predictive
  distributions (taking into account also the predictive variance) of the
  Gaussian Process are considerably better.

—
Reply to this email directly or view it on GitHub
#3645 (comment)
.

Could you be a bit more specific what you find misleading? What do you mean with prediction variance of the individual trees? DecisionTreeRegressor does not define a prediction variance. Or do you mean the variance from the Bias-variance decomposition of the MSE?

The predictive distributions (taking into account also the predictive variance) of the Gaussian Process are considerably better.

This is where I dont agree. The variance of the predictions of the individual trees (as computed when with_std=True) does not correspond to the variance of the predictions of the ensemble. Hence it is not correct and misleading to use this as a way to characterize the predictive distribution of the forest (at least not like that). In fact, the beauty of making an ensemble is that it erases the part of the error that is specifically due to the variance of the individual models.

So do you think that adding with_std to random forests is a bad idea?

@mblondel : No, it may be used as a good proxy for characterizing the certainty of the predictions and identifying hard cases. Yet, caution should be taken regarding the exact interpretation of these values.

@glouppe Could you suggest a small paragraph which would explain your point for the purpose of this example?

As a side remark, somewhat relevant to this PR :), I should say that I have always been uncomfortable with the 'predict' method of Gaussian processes, that had a different signature than the standard predict. I am afraid that this is the door open to more and more API slippage.

I am afraid that this is the door open to more and more API slippage.

What do you propose? :)

What do you propose? :)

I honestly have no good solution, this is why I phrase this as a side
comment. I just want to mention that, to me, this is a code smell, and
whoever has a proposal here would be greatly appreciate.

I don't think that this should be an argument against merging this PR.

@glouppe I think I got your point now. We should probably distinguish between the variance of the inferential procedure and what a fitted estimator (can) return as standad deviation of the prediction. The decision tree inference procedure has high variance but a single fitted decision tree provides no way of estimating this variance (does it?). The inference procedure for an ensemble like RF has less variance than that of an individual tree. What is returned by predict of a fitted RF/BaggingRegressor (via with_std) is related to the variance of the decision tree inference procedure. Does that make sense?

@GaelVaroquaux One alternative to the with_std/eval_MSE option of predict could be to add a predict_proba method in regressors (just as in classifiers) which returns (for every query point) an object representing the predictive distribution. For example an instance of scipy.stats.norm.
The main disadvantage of this would be the complex return type of the method. Not sure if I like that; just mention it as an option.

@jmetzen This is an interesting idea but for RF I am not sure we can assume any parametric form for the predictive distribution...

@glouppe I think I got your point now. We should probably distinguish between the variance of the inferential procedure and what a fitted estimator (can) return as standad deviation of the prediction. The decision tree inference procedure has high variance but a single fitted decision tree provides no way of estimating this variance (does it?). The inference procedure for an ensemble like RF has less variance than that of an individual tree. What is returned by predict of a fitted RF/BaggingRegressor (via with_std) is related to the variance of the decision tree inference procedure. Does that make sense?

Yes, this is exactly that. Accordingly, I would either not extend the GP example with RF/Bagging, or try to explain this as best as possible in the example.

Looking forward to seeing this merged in someday! I am too noob to help sadly TT

Taking the discussion a bit late, but wouldn't a bootstrap approach be usable here ?

Such a method would be able to estimate the variance of the prediction for any estimator at the cost of computing model and predictions on a number of resampled datasets.

I can't seem to find a proper reference for this method but you can find a sketch here

@jmetzen What shall we do with this PR? The changes on the old GPs are no longer relevant and the comparison with forests was tricky to explain.

Personally, I would still be interested in having this functionality in bagging estimators.

True, some parts of this PR are now obsolete. We could still add the option "return_std" (that's how it is called in GPR now) to BaggingRegressor and RandomForestRegressor. The extra LOCs and computation are fairly modest; the explanation how the interpretation of the value returned differs from the value returned by GPR's predict is slightly more involved.

@amueller's comment regarding how well-calibrated the estimate would be is also valid; I have no experience on this.

Does anyone have an idea for an example which illustrates a valid use-case of return_std for ensembles?

Shouldn't this be done via quantile regression forests (http://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf)?
An implementation seems relatively straightforward, basically to record all of values in the tree leaves, not just the mean. For prediction interval or estimating a quantile, the requested quantile can be calculated on the stored values.

Closing in favour of #5532.