Skip to content

FEAT allow configuring automatically requested metadata #31401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

adrinjalali
Copy link
Member

@adrinjalali adrinjalali commented May 20, 2025

Alternative to #30946

The approach here is to avoid adding a new function to the base class, and instead handle it via the same get_metadata_routing.

However, this solution changes the signature of the method, which is not idea. We could at the same time deprecate the method and replace it with a __sklearn_get_metadata_routing__ method with a new signature, and the __sklearn__ pattern as the rest of the developer API we have now.

cc @antoinebaker @ogrisel @StefanieSenger

This is not complete, putting here for us to be able to have a more concrete view of the idea.

The idea is that developer / user code looks like this:

import numpy as np
import sklearn
from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification

sklearn.set_config(enable_metadata_routing=True, metadata_request_policy="auto")

class DefaultRoutingClassifier(ClassifierMixin, BaseEstimator):
    # Class-level default request for fit method
    __metadata_request__fit = {"sample_weight": True}

    def get_metadata_routing(self, **auto_requests):
        # Each instance can configure metadata which should be requested by default if
        # `set_config(metadata_request_policy="auto")` is set. These request values are
        # passed to the parent's `get_metadata_routing` method.
        requests = super().get_metadata_routing(predict="groups")
        return requests

    def fit(self, X, y, sample_weight=None):
        self.classes_ = np.array([0, 1])
        print(sample_weight)
        return self

    def predict(self, X, groups=None):
        print(groups)
        return np.ones(len(X))

X, y = make_classification()

pipeline = Pipeline([
    ("scaler", StandardScaler().set_fit_request(sample_weight=False)),
    ("classifier", DefaultRoutingClassifier().set_predict_request(groups="my_groups")),
])
pipeline.fit(X, y, sample_weight=np.ones(len(X)))
pipeline.predict(X, my_groups=np.ones(len(X)) + 1)


pipeline = Pipeline([
    ("scaler", StandardScaler().set_fit_request(sample_weight=False)),
    ("classifier", DefaultRoutingClassifier()),
])
pipeline.fit(X, y, sample_weight=np.ones(len(X)))
pipeline.predict(X, groups=np.ones(len(X)) + 1)

pipeline = Pipeline([
    ("scaler", StandardScaler().set_fit_request(sample_weight=False)),
    ("classifier", DefaultRoutingClassifier().set_predict_request(groups=False)),
])
pipeline.fit(X, y, sample_weight=np.ones(len(X)))
pipeline.predict(X, groups=np.ones(len(X)) + 1)

Copy link

❌ Linting issues

This PR is introducing linting issues. Here's a summary of the issues. Note that you can avoid having linting issues by enabling pre-commit hooks. Instructions to enable them can be found here.

You can see the details of the linting issues under the lint job here


mypy

mypy detected issues. Please fix them locally and push the changes. Here you can see the detected issues. Note that the installed mypy version is mypy=1.15.0.


sklearn/tests/test_metadata_routing.py:1078: error: Definition of "get_metadata_routing" in base class "_RoutingNotSupportedMixin" is incompatible with definition in base class "_MetadataRequester"  [misc]
sklearn/ensemble/_weight_boosting.py:342: error: Definition of "get_metadata_routing" in base class "_RoutingNotSupportedMixin" is incompatible with definition in base class "_MetadataRequester"  [misc]
sklearn/ensemble/_weight_boosting.py:867: error: Definition of "get_metadata_routing" in base class "_RoutingNotSupportedMixin" is incompatible with definition in base class "_MetadataRequester"  [misc]
Found 3 errors in 2 files (checked 558 source files)

Generated for commit: a35bf40. Link to the linter CI: here

@StefanieSenger StefanieSenger added the Metadata Routing all issues related to metadata routing, slep006, sample props label May 20, 2025
Copy link
Contributor

@StefanieSenger StefanieSenger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the naming of "auto-requests", that I think is very intuitive.

Still, auto-requests here are contrasted with "empty" metadata requests and "default" metadata requests. If I understood correctly, then the "default" is "empty" requests, so these should both be referred to via the same terms.

Comment on lines +1590 to +1591
Auto-requested metadata. These metadata request values override default
request values if `set_config(metadata_request_policy="auto")` is set.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe explain this a bit better:

Suggested change
Auto-requested metadata. These metadata request values override default
request values if `set_config(metadata_request_policy="auto")` is set.
Auto-requested metadata. Keyword arguments where the key is a method
name (e.g., 'fit', 'predict') and the value is a metadata request
(either a str or a list of str). These override default request values
if `set_config(metadata_request_policy="auto")` is set.

# Here's an example demonstrating both approaches:


class DefaultRoutingClassifier(ClassifierMixin, BaseEstimator):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we rename this, since it's not a router?

Suggested change
class DefaultRoutingClassifier(ClassifierMixin, BaseEstimator):
class DefaultClassifier(ClassifierMixin, BaseEstimator):

Comment on lines +175 to +176
- `"auto"`: Metadata is requested if the consumer has flagged it as an
auto-request.
Copy link
Contributor

@StefanieSenger StefanieSenger May 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to find a formulation that communicates that this entails that a sub-part of the metadata can be requested. I think it's not clear otherwise.

I would also avoid talking about "the" consumer, since there can be several.

Suggested change
- `"auto"`: Metadata is requested if the consumer has flagged it as an
auto-request.
- `"auto"`: Only the subset of metadata flagged by consumers for
auto-request will be requested.

Parameters
----------
**auto_requests : str or list of str
Auto-requested metadata. These metadata request values override default
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default are the empty requests, right?

Suggested change
Auto-requested metadata. These metadata request values override default
Auto-requested metadata. These metadata request values override empty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Metadata Routing all issues related to metadata routing, slep006, sample props module:utils
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants