In the original PR, we had a short discussion whether to have all includes at the very beginning or each import where it is first needed. I'm fine with both approaches.

I really prefer to split large import blocks for long examples. Otherwise the reader as to scroll-up a lot to know where a function / class comes from.

With min_weight_fraction_leaf=0.01, we could avoid non-positive predictions of the random forest. Could we try the same for the HGBR, in this case with min_samples_leaf=...?

Because max_leaf_nodes is limited and the number of trees very large, I don't think this is going to change anything. Also with the Poisson loss and its log link that are now enabled we don't have any negative predictions for HGBRT anymore.

Just as an explanation: For large sample sizes, the different losses do not matter much (as long as they are strictly consistent for the expected value). Without interactions and other cheerful feature engineering for linear models, the tree based models will win.

I agree. I even doubt that it's possible to cheerfully feature engineer enough to compete with the tree based models when n_samples > 1e5.

Maybe not on this and most other datasets, but I have seen otherwise:smirk:

Would love to be proven wrong ;)

I feel like this is an example of where I rather not see our library be used lol (it also has a lot of ethical concerns, bias etc, which we don't tackle).

Not that we should remove it really, just saying I personally probably would :D

Let us please agree on not having that discussion here. I understand and admire your point of view and would be very glad to talk with you about this controversial (and never ending) subject. At last, note that neither gender nor race are in the data. :no_fitting_emoji:

The issue is that the data is biased and we can't even measure that bias because we don't even have the features we need (like race and gender). Not using those features doesn't mean we're not going to have biased model discriminating against a certain group, and I'm very very worried about putting an example out there which people would then use as a reference to [unintentionally] discriminate against people. @romanlutz, do you happen to have a good example for this one?

I'm all for better examples for controlling for bias, but I also don't think this PR is the right place for this discussion. It merely refactors an existing example, we should have this discussion in a separate issue.

As a side note, I imagine there could indeed be some sample selection bias in the the data (i.e. company chooses customers), however the target variable (frequency or cost of accidents) shouldn't be too biased, I think? At least significantly less biased than in other examples such as https://scikit-learn.org/dev/auto_examples/inspection/plot_linear_model_coefficient_interpretation.html for predicting wage. Also pricing policy of an insurance company doesn't directly impact how often one has accidents, so at least there is no immediate feedback loop with the training data. I'm not an expert on this, and there are likely other effects, but I'm just saying that we should discuss it in a separate issue.

I don't disagree with the suggestions to discuss this elsewhere. For more information on what @adrinjalali is referring to I recommend "Big Data's Disparate Impact" by Barocas and Selbst. file:///C:/Users/rolutz/AppData/Local/Temp/SSRN-id2477899.pdf section I.D. The entire piece is actually relevant for such a scenario, but that's the section that discusses excluding sensitive features such as race & gender. Other than that there's still potential bias in how the data got collected (I.A, I.B, I.C).

I think there's value in acknowledging such potential shortcomings so that people don't assume that it's the best (or only) way to approach the task. We wouldn't want users to end up on my list of questionable or unethical use cases. With the acknowledgment it should be clear that it's just a demonstration of scikit-learn. At least that's my point of view :-) Thanks for asking!

@adrinjalali and @romanlutz I opened #16715 if you want to address this topic further.

that's some large change in alpha, interesting!

...?

Also, it's not the squared error, but rather the linearity assumptions that cause negative predictions.

The tree-based model does not make the linear assumption between E[y] and X and can still predict negative values. But I will mention the implicit identity link function of Ridge.

this is raising

/home/circleci/project/sklearn/datasets/_openml.py:655: UserWarning: Version 1 of dataset freMTPL2freq is inactive, meaning that issues have been found in the dataset. Try using a newer version from this URL: https://www.openml.org/data/v1/download/20649148/freMTPL2freq.arff

Yes but unfortunately there is nothing we can do about it apart from contacting the original uploader. Let me do that.

Do the curves that are plotted below have a specific name?

Surely they are different from calibration curves as we have them in the UG ?

The calibration curves currently in the user guide are only defined for classification problems. They are similar to those int this example (plot the mean observed values per prediction bin) when using the quantile binning strategy.

It would make sense to extend the calibration_curve tool to support regression but this should be done in a dedicated PR.

should "is not" be "does not"?

Though I'm not sure I understand the plot for the dummy: if it always predicts the mean, why do we have multiple dots on the x-axis? I would have assumed the plot should just be a single dot?

We bin by quantile so the content of each bin is independent of y hat (since y hat is constant) and therefore uniformly random.

I agree it's an edge case. But it really high light that this notion of probabilistic calibration and ranking power are quite independent of one another.

It's curious that the HGBDT overestimates the number of claim, much more than the PoissonRegressor, while both curves look pretty similar

I am not sure that this difference is significant. One would need to bootstrap to get confidence intervals but this is too expensive and complicated for this example.