You have 2 samples and request early_stopping. From the verbose, it seems that the validation score is always nan probably because you are using 0 or 1 sample (10% of 2 samples). Since the coefficients are stored only if the score is decreasing, it actually never happens and thus _best_coefs is never set.

This is really a corner case and we choose to detect it much earlier even before to start trying to optimize the problem because we don't have enough samples in the validation set.

The error also seems to completely disappear when there are more than around 10-25 training examples.

Be aware that you should not do any machine learning with so few samples.

You have 2 samples and request early_stopping. From the verbose, it seems that the validation score is always nan probably because you are using 0 or 1 sample (10% of 2 samples). Since the coefficients are stored only if the score is decreasing, it actually never happens and thus _best_coefs is never set.

The example provided is 2 samples but it's happening with up to 10 samples.

Yes you need at least 2 samples to compute the R2 scores. By default, you take 10% validation. so above 10 samples, you get the required 2 samples (with the rounding).

Yes you need at least 2 samples to compute the R2 scores. By default, you take 10% validation. so above 10 samples, you get the required 2 samples (with the rounding).

Aha got it

This is really a corner case and we choose to detect it much earlier even before to start trying to optimize the problem because we don't have enough samples in the validation set.

Does the earlier check actually get made? If yes, why did it not catch this case? Or was your comment a "we should detect it much earlier" -> a PR would be helpful/welcome to implement this?

Sorry for the confusion.

I meant that we should create a PR to detect this case (i.e. not enough samples to compute the validation metric) and raise an error.

Hi @glemaitre ,
As nobody seemed to have taken it up, I tried to attempt to design a fix based on your recommendation. Would you like to have a look at it and suggest improvements? (See PR above.)
Thank you.

Hi @glemaitre,

I hope you're doing well. I came across this issue and wanted to check if someone is already working on it. If it's still open and no one is currently assigned, I’d happily take it up and work on resolving it.

Looking forward to your response.

It looks like #24788 has been open. Before to open a new PR, I'll like to have a look at this open PR because it seems that we did not provide feedback.