DOC Clarify usage of d2_pinball_score with model selection tools #31239
Conversation
sklearn/metrics/_regression.py
Outdated
| This metric is not a built-in scoring string for use in model selection | ||
| tools such as `GridSearchCV` or `RandomizedSearchCV`. | ||
|
|
||
| To use it as a custom scoring function, wrap it using | ||
| :func:`~sklearn.metrics.make_scorer`: | ||
|
|
||
| >>> from sklearn.metrics import make_scorer, d2_pinball_score | ||
| >>> scorer = make_scorer(d2_pinball_score, alpha=0.95) | ||
| >>> # Then use it as `scoring=scorer` in RandomizedSearchCV or GridSearchCV |
There was a problem hiding this comment.
| This metric is not a built-in scoring string for use in model selection | |
| tools such as `GridSearchCV` or `RandomizedSearchCV`. | |
| To use it as a custom scoring function, wrap it using | |
| :func:`~sklearn.metrics.make_scorer`: | |
| >>> from sklearn.metrics import make_scorer, d2_pinball_score | |
| >>> scorer = make_scorer(d2_pinball_score, alpha=0.95) | |
| >>> # Then use it as `scoring=scorer` in RandomizedSearchCV or GridSearchCV | |
| This metric is not a built-in scoring string for use in model selection | |
| tools such as `GridSearchCV` or `RandomizedSearchCV`. | |
| To use it as a custom scoring function, wrap it using | |
| :func:`~sklearn.metrics.make_scorer`. See Examples for details. |
We can keep the code snippet in the examples section.
There was a problem hiding this comment.
Moved the scorer usage example to the Examples section and cleaned up the Notes.
Thanks for the feedback — ready for your next review whenever you have time!
sklearn/metrics/_regression.py
Outdated
| >>> # Using with make_scorer | ||
| >>> from sklearn.metrics import make_scorer | ||
| >>> scorer = make_scorer(d2_pinball_score, alpha=0.95) |
There was a problem hiding this comment.
| >>> # Using with make_scorer | |
| >>> from sklearn.metrics import make_scorer | |
| >>> scorer = make_scorer(d2_pinball_score, alpha=0.95) | |
| Using with :func:`~sklearn.metrics.make_scorer`: | |
| >>> from sklearn.metrics import make_scorer, d2_pinball_score | |
| >>> pinball_95_scorer = make_scorer(d2_pinball_score, alpha=0.95) | |
| >>> from sklearn.model_selection import GridSearchCV | |
| >>> from sklearn.svm import LinearSVC | |
| >>> grid = GridSearchCV( | |
| ... LinearSVC(), | |
| ... param_grid={"C": [1, 10]}, | |
| ... scoring=pinball_95_scorer, | |
| ... cv=5, | |
| ... ) |
Expand the example a bit.
There was a problem hiding this comment.
Expand examples section to show how to use d2_pinball_score with
make_scorer and GridSearchCV. Also clarify in the Notes that
d2_pinball_score is not a built-in scorer string.
There was a problem hiding this comment.
@yuanx749 note that LinearSVC is a classifier (suited to model discrete class observations in the target variable), while d2_pinball_score is a metric for quantile regression problems. Better use a (quantile) regression model in the example.
…_scorer and GridSearchCV
e4f733b to
fe85a06
Compare
| >>> y_true = [3, -0.5, 2, 7] | ||
| >>> y_pred = [2.5, 0.0, 2, 8] | ||
| >>> d2_pinball_score(y_true, y_pred, alpha=0.95) | ||
| 0.968... |
There was a problem hiding this comment.
This should help fix the broken tests.
| 0.968... | |
| 0.578... |
There was a problem hiding this comment.
Next time, please run pytest --doctest-modules path/to/the/code/you/change.py when editing doctests.
Alternatively, read the logs of the failing continuous integration reports linked from the PR to find out what caused the failures.
| ... scoring=pinball_95_scorer, | ||
| ... cv=2, | ||
| ... ) | ||
| >>> _ = grid.fit(X, y) |
There was a problem hiding this comment.
Maybe you could display the value of grid.best_params_ to make the example more complete. E.g. something like the following:
>>> grid.fit(X, y).best_params_
{"fit_intercept": True}
Run the doctest locally to check that this is actually the best param:
$ pytest -v --doctest-modules sklearn/metrics/_regression.py
sklearn/metrics/_regression.py
Outdated
| >>> # Using with make_scorer | ||
| >>> from sklearn.metrics import make_scorer | ||
| >>> scorer = make_scorer(d2_pinball_score, alpha=0.95) |
There was a problem hiding this comment.
@yuanx749 note that LinearSVC is a classifier (suited to model discrete class observations in the target variable), while d2_pinball_score is a metric for quantile regression problems. Better use a (quantile) regression model in the example.
| >>> X = np.array([[1], [2], [3], [4]]) | ||
| >>> y = np.array([2.5, 0.0, 2, 8]) | ||
| >>> grid = GridSearchCV( | ||
| ... LinearRegression(), |
There was a problem hiding this comment.
It would make more sense to tune the fit_intercept parameter of QuantileRegressor(quantile=0.95) instead of LinearRegression. LinearRegression predicts an estimate of E[y|X] instead of an estimate of Q_{0.95}(y|X).
Reference Issues/PRs
Towards #28671
This PR addresses confusion discussed by users trying to use d2_pinball_score directly as a string value for the scoring parameter in model selection APIs such as GridSearchCV and RandomizedSearchCV.
Although d2_pinball_score is a valid scoring function, it is not registered as a string scorer and must be wrapped using make_scorer. This was not clearly explained in the docstring.
What does this implement/fix? Explain your changes.
This improves the documentation of sklearn.metrics.d2_pinball_score by:
Adding a note to clarify that this metric is not a valid string identifier for the scoring parameter in model selection tools.
Providing a code example showing how to use make_scorer to wrap the function correctly for use with GridSearchCV or RandomizedSearchCV.
Adding a usage snippet under the Examples section for easy discoverability.
This change is intended to make usage of d2_pinball_score more transparent and reduce common user errors and confusion.
Any other comments?
This PR does not affect the behavior of the function or its API — it is documentation-only.
The motivation arose from real-world usage in probabilistic forecasting, where d2_pinball_score is useful but hard to integrate into model selection workflows due to missing documentation around make_scorer.