Skip to content

FEA return final cross-validation score in SequentialFeatureSelector #31483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

cboseak
Copy link

@cboseak cboseak commented Jun 4, 2025

Reference Issues/PRs

What does this implement/fix? Explain your changes.

  • Added an attribute (e.g., final_cv_score_) that stores the mean cross-validation score of the final model with the selected features. This would avoid having to run another cross-validation externally to get the final performance score.
    • Currently, when using SequentialFeatureSelector, it internally performs cross-validation to decide which features to select, based on the scoring function. However, the final cross-validation score (e.g., recall) is not returned by the SFS object.

Copy link

github-actions bot commented Jun 4, 2025

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: 83c7534. Link to the linter CI: here

@betatim betatim changed the title per issue 31473, return final cross-validation score in SequentialFea… FEA return final cross-validation score in SequentialFea… Jun 6, 2025
Copy link
Contributor

@OmarManzoor OmarManzoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @cboseak

Copy link
Contributor

@OmarManzoor OmarManzoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @cboseak

@OmarManzoor OmarManzoor added the Waiting for Second Reviewer First reviewer is done, need a second one! label Jun 11, 2025
@adrinjalali adrinjalali removed the Waiting for Second Reviewer First reviewer is done, need a second one! label Jun 12, 2025
@adrinjalali adrinjalali changed the title FEA return final cross-validation score in SequentialFea… FEA return final cross-validation score in SequentialFeatureSelector Jun 12, 2025
@cboseak
Copy link
Author

cboseak commented Jun 12, 2025

See latest changes to address your comments

Copy link
Member

@adrinjalali adrinjalali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd need more opinions on this to see if we'd like to include it.

cc @scikit-learn/core-devs

Comment on lines +364 to +374
cloned_estimator = clone(self.estimator)

return cross_val_score(
cloned_estimator,
X_new,
y,
cv=cv,
scoring=self.scoring,
n_jobs=self.n_jobs,
params=params if _routing_enabled() else None,
).mean()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cloned_estimator = clone(self.estimator)
return cross_val_score(
cloned_estimator,
X_new,
y,
cv=cv,
scoring=self.scoring,
n_jobs=self.n_jobs,
params=params if _routing_enabled() else None,
).mean()
return cross_val_score(
clone(self.estimator),
X_new,
y,
cv=cv,
scoring=self.scoring,
n_jobs=self.n_jobs,
params=params if _routing_enabled() else None,
)

cv=cv,
scoring=self.scoring,
n_jobs=self.n_jobs,
params=params if _routing_enabled() else None,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be a _raise_for_params for this.

check_is_fitted(self)

X_new = X[:, self.support_]
cv = check_cv(self.cv, y, classifier=is_classifier(self.estimator))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could have a self._get_cv() to not repeat this line.

Comment on lines +357 to +358
score : float
The mean cross-validation score of the selected feature set.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might want to return raw values instead.

….feature.rst


update based on PR suggestions

Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add option to return final cross-validation score in SequentialFeatureSelector
3 participants