Skip to content

FEAT multioutput routes metadata #22986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 44 commits into from
Jul 17, 2022

Conversation

adrinjalali
Copy link
Member

Towards #22893

This is the first PR which handles deprecation of old code where users pass metadata w/o setting request values. This PR adds the machinery to do so, and adds routing to MultiOutput estimators.

@agramfort may be interested in this as well.

ping @jnothman @thomasjpfan @lorentzenchr

Copy link
Member

@jnothman jnothman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one of the cases where the current system just works, and the proposal seems highly bureaucratised :(

I do wonder if we should just be using fit params without distinguishing from partial_fit. Did we have use cases for distinguishing fit and partial_fit params?

@adrinjalali
Copy link
Member Author

This is one of the cases where the current system just works, and the proposal seems highly bureaucratised :(

I started writing this:

Another alternative that I have explored for meta-estimators such as this one, which "only" pass metadata to the single sub-estimator they have, is to behave like a consumer rather than a router, and expose their sub-estimator's requests.

And realized there could potentially an issue with this implementation since MultiOutputClassifier implements its own score, which would make it a consumer and a router. In this case it's not an issue since the score method doesn't accept any metadata.

I do wonder if we should just be using fit params without distinguishing from partial_fit. Did we have use cases for distinguishing fit and partial_fit params?

If we do that, do we want to also consider predict_proba, predict_log_proba, decision_function, and predict all having the same request and we could have common tests making sure they all accept the same metadata in their signature. Should we do that?

@adrinjalali adrinjalali added this to the 1.2 milestone Mar 29, 2022
@adrinjalali
Copy link
Member Author

This is one of the cases where the current system just works, and the proposal seems highly bureaucratised :(

I'm not sure what to do with this concern @jnothman 😆

But I also don't think it's too bad. If this estimator would accept sample_weight for its score (which it does override, but somehow doesn't take sample_weight into account), then it would be a router AND a consumer, and the SLEP works perfectly fine.

I do wonder if we should just be using fit params without distinguishing from partial_fit. Did we have use cases for distinguishing fit and partial_fit params?

Seems like we're keeping them separate then? (#22988)

Copy link
Member

@jnothman jnothman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the very slow review!!

@adrinjalali
Copy link
Member Author

The summary of the latest change:

Previously the code would try to route parameters, and if there was an exception, it would try to assume all those parameters as requested, and then route again.

Now, the exception includes enough information that when raised, we can check if those relevant parameters can be routed and warned or if there should be an error.

It also made me change the signature of warn_on method to include which metadata for each method should raise a warning. This means for instance, if previously fit was routing sample_weight but not other metadata, we would warn on sample_weight having not request value, but raise an error on others, since the routing of other metadata is newly implemented.

Copy link
Member

@glemaitre glemaitre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At a first read, it seems pretty straightforward.
I will have to make an additional pass and look more into the details of the _is_default_request just to be sure that there is nothing that can go sideways.

Copy link
Member

@thomasjpfan thomasjpfan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I hope the complexity of the routing mechanism does not get any higher than this.

Copy link
Member

@jnothman jnothman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is certainly much neater than the more magic solution before. I'm not sure we're going to quite achieve @thomasjpfan's request that "the complexity of the routing mechanism does not get any higher than this." but at least this lays most of the groundwork, so that the rest of the implementations are complicated, not complex. Congrats @adrinjalali!

In terms of the impact on users, however, this remains one of the cases where the implicit behaviour was actually okay: forwarding on any kwargs in fit to the child estimator. Now we will force the user to explicitly request these. I'd like to know from others: Is this a reasonable burden on the user? Is there a way to avoid it without major inconsistencies between simple routing as presented here and the more complex routing of a GridSearchCV or Pipeline (where there are many candidate destinations)?

@adrinjalali
Copy link
Member Author

Opened #23928 to discuss the verbosity issue.

@glemaitre
Copy link
Member

The lines reported not covered are the dummy estimators. I think this is fine if they are not tested.

@glemaitre glemaitre merged commit b93a9e6 into scikit-learn:sample-props Jul 17, 2022
@glemaitre
Copy link
Member

Thanks @adrinjalali

@adrinjalali adrinjalali deleted the slep6/multioutput branch July 17, 2022 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants