Skip to content

DOC Clarify usage of d2_pinball_score with model selection tools #31239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+28 −11
Open

DOC Clarify usage of d2_pinball_score with model selection tools #31239

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
76cca9c
DOC Clarify usage of d2_pinball_score with model selection tools
MaddyRizvi Apr 22, 2025
afd3071
DOC Update d2_pinball_score docstring: move scorer example to Example…
MaddyRizvi Apr 26, 2025
fe85a06
Expand examples section to show how to use d2_pinball_score with make…
MaddyRizvi Apr 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 28 additions & 11 deletions sklearn/metrics/_regression.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1750,7 +1750,12 @@ def d2_pinball_score(
This metric is not well-defined for a single point and will return a NaN
value if n_samples is less than two.

References
This metric is not a built-in scoring string for use in model selection
tools such as `GridSearchCV` or `RandomizedSearchCV`.
To use it as a custom scoring function, wrap it using
:func:`~sklearn.metrics.make_scorer`. See Examples for details.

References
----------
.. [1] Eq. (7) of `Koenker, Roger; Machado, José A. F. (1999).
"Goodness of Fit and Related Inference Processes for Quantile Regression"
Expand All @@ -1761,18 +1766,30 @@ def d2_pinball_score(

Examples
--------
>>> import numpy as np
>>> from sklearn.metrics import d2_pinball_score
>>> y_true = [1, 2, 3]
>>> y_pred = [1, 3, 3]
>>> d2_pinball_score(y_true, y_pred)
0.5
>>> d2_pinball_score(y_true, y_pred, alpha=0.9)
0.772...
>>> d2_pinball_score(y_true, y_pred, alpha=0.1)
-1.045...
>>> d2_pinball_score(y_true, y_true, alpha=0.1)
1.0
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> d2_pinball_score(y_true, y_pred, alpha=0.95)
0.968...
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should help fix the broken tests.

Suggested change
0.968...
0.578...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Next time, please run pytest --doctest-modules path/to/the/code/you/change.py when editing doctests.

Alternatively, read the logs of the failing continuous integration reports linked from the PR to find out what caused the failures.


Using with :func:`~sklearn.metrics.make_scorer`:

>>> from sklearn.metrics import make_scorer, d2_pinball_score
>>> pinball_95_scorer = make_scorer(d2_pinball_score, alpha=0.95)
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.linear_model import LinearRegression
>>> X = np.array([[1], [2], [3], [4]])
>>> y = np.array([2.5, 0.0, 2, 8])
>>> grid = GridSearchCV(
... LinearRegression(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would make more sense to tune the fit_intercept parameter of QuantileRegressor(quantile=0.95) instead of LinearRegression. LinearRegression predicts an estimate of E[y|X] instead of an estimate of Q_{0.95}(y|X).

... param_grid={"fit_intercept": [True, False]},
... scoring=pinball_95_scorer,
... cv=2,
... )
>>> _ = grid.fit(X, y)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you could display the value of grid.best_params_ to make the example more complete. E.g. something like the following:

    >>> grid.fit(X, y).best_params_
    {"fit_intercept": True}

Run the doctest locally to check that this is actually the best param:

$ pytest -v --doctest-modules sklearn/metrics/_regression.py

"""

y_type, y_true, y_pred, multioutput = _check_reg_targets(
y_true, y_pred, multioutput
)
Expand Down
Loading