Describe the workflow you want to enable

Add more fine-grained diagnostic, similar to ROC or Precision-Recall curves, to regression problems. It appears that this library has a lot of excellent tools for classification, and I believe it would benefit from some additional tools for regression.

Describe your proposed solution

Compute Regression Error Characteristic (REC) [1] curve - for each error threshold the percentage of samples whose error is below that threshold. This is essentially the CDF of the regression errors. Its function is similar to that of ROC curves - allows comparing performance profiles of regressors beyond just one summary statistic, such as RMSE or MAE.

I already implement a pull-request:
#31380

Screenshot from the merge request:

If you believe this feature is useful, please help me with reviewing and merging it.

Describe alternatives you've considered, if relevant

Regression Receiver Operating Characteristic (RROC) curves, proposed [2], which plot over-prediction vs under-prediction, are a different form of diagnostic curves for regression. They may also be useful, but I think we should begin from somewhere, and I belive it's better to begin from REC, both because the paper has more citations, and because it turned out to be very useful for me at work, and I believe it can be similarly useful to other scientists.

Additional context

References

[1]: Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
[2]: Hernández-Orallo, J., 2013. ROC curves for regression. Pattern Recognition, 46(12), pp.3395-3411.