Can we maybe have a call, 30 minutes or so. We could discuss the subject while having the notebook in a shared screen.
I believe interactivity can greatly help in discussing this topic, because it is quite subtle and prone to misunderstandings.

Skype, Zoom, anything else, you name it :)

https://stats.stackexchange.com/a/492581/343156
This is the only written exlanation I have found so far, and I believe I can explain it somewhat more clearly, with that practical example.

Anyways, I have elaborated the notebook in written form, hope the idea has become more clear.

Finally found an article where R squared out of sample is expressed in explicit and clear manner

https://dachxiu.chicagobooth.edu/download/ML.pdf

see 1.8 Performance evaluation

Note: in the denominator there is no substration, that is y_train.mean() is assumed to be 0, but this is a hack specific to stock return prediction.

@glemaitre you are right. In the notebook I provided, the main factor in producing negative R squared was the lack of shuffling, as demonstrated by you in that notebook

However, the initial issue I was referring to, is still there and also contributes to negative R squared.

To show this

• dataset was shuffled
• the model was tuned so that is's bad enough to consistently result in negative R squared in cross-validation, but still good enough to consistently result in positive R squared out of sample in cross-validation

All the changes were made in the same notebook

Let me know if it convinces you about usefulness of R squared out of sample, or you still have doubts.

I see a potential blocker here regarding the API. I am unsure of the way forward here.

got it, thank you

Any updates on this?

My recommendation is to write your own code to do R^2 out of sample, however, unfortunately, it will not be possible to integrate it with some existing scikit-learn mechanisms, such as cross-validation search models and so on.

The code for R^2 oos is very simple

y_baseline = np.mean(y_train)
r2 = 1 - np.mean((y_pred - y_test) ** 2) / np.mean((y_baseline - y_test) ** 2)

Noted, thank you.