Skip to content

TST check n_features_in_ in pipeline module #20192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 7, 2021
Merged

TST check n_features_in_ in pipeline module #20192

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a6f05c7
TST check n_features_in_ in pipeline module
glemaitre Jun 2, 2021
db4e5dd
TST add FeatureUnion in metaestimator check
glemaitre Jun 2, 2021
abdd88d
Add documentation
glemaitre Jun 3, 2021
7b4aa7b
typo
glemaitre Jun 3, 2021
3d2f3c0
typo
glemaitre Jun 3, 2021
db8c2dc
Apply suggestions from code review
ogrisel Jun 3, 2021
e0c7d68
Merge branch 'main' into pipeline_n_features_in_
ogrisel Jun 3, 2021
08a2c86
Update sklearn/pipeline.py
ogrisel Jun 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions sklearn/pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,13 @@ class Pipeline(_BaseComposition):
Read-only attribute to access any step parameter by user given name.
Keys are step names and values are steps parameters.

n_features_in_ : int
Number of features seen during :term:`fit`. Only defined if the
underlying first estimator in `steps` exposes such an attribute
when fit.

.. versionadded:: 0.24

See Also
--------
make_pipeline : Convenience function for simplified pipeline construction.
Expand Down Expand Up @@ -826,6 +833,15 @@ class FeatureUnion(TransformerMixin, _BaseComposition):
If True, the time elapsed while fitting each transformer will be
printed as it is completed.

Attributes
----------
n_features_in_ : int
Number of features seen during :term:`fit`. Only defined if the
underlying first transformer in `transformer_list` exposes such an
attribute when fit.

.. versionadded:: 0.24

See Also
--------
make_union : Convenience function for simplified feature union
Expand Down
1 change: 0 additions & 1 deletion sklearn/tests/test_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,6 @@ def test_search_cv(estimator, check, request):
N_FEATURES_IN_AFTER_FIT_MODULES_TO_IGNORE = {
'model_selection',
'multioutput',
'pipeline',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remark: even if removed from the list, the estimators of this module are not tested anyway (skipped in the instance constructions)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is true that we only have a negative test for n_features_in_ via sklearn.tests.test_metaestimator.test_meta_estimators_delegate_data_validation and _generate_meta_estimator_instances_with_pipeline that is updated in this pipeline.

It would be great to have a new positive common test for the presence of n_features_in_ on tabular dataset in sklearn.tests.test_metaestimator.

}

N_FEATURES_IN_AFTER_FIT_ESTIMATORS = [
Expand Down
5 changes: 2 additions & 3 deletions sklearn/tests/test_docstring_parameters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,6 @@ def _construct_searchcv_instance(SearchCV):
'naive_bayes',
'neighbors',
'neural_network',
'pipeline',
'preprocessing',
'random_projection',
'semi_supervised',
Expand All @@ -215,10 +214,10 @@ def test_fit_docstring_attributes(name, Estimator):
attributes = doc['Attributes']

IGNORED = {'ClassifierChain', 'ColumnTransformer',
'CountVectorizer', 'DictVectorizer', 'FeatureUnion',
'CountVectorizer', 'DictVectorizer',
'GaussianRandomProjection',
'MultiOutputClassifier', 'MultiOutputRegressor',
'NoSampleWeightWrapper', 'Pipeline', 'RFE', 'RFECV',
'NoSampleWeightWrapper', 'RFE', 'RFECV',
'RegressorChain', 'SelectFromModel',
'SparseCoder', 'SparseRandomProjection',
'SpectralBiclustering', 'StackingClassifier',
Expand Down
44 changes: 29 additions & 15 deletions sklearn/tests/test_metaestimators.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from sklearn.exceptions import NotFittedError
from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.linear_model import Ridge, LogisticRegression
from sklearn.preprocessing import StandardScaler, MaxAbsScaler


class DelegatorData:
Expand Down Expand Up @@ -185,6 +186,19 @@ def _generate_meta_estimator_instances_with_pipeline():
else:
yield Estimator(estimator)

elif "transformer_list" in sig:
# FeatureUnion
transformer_list = [
("trans1", make_pipeline(TfidfVectorizer(), MaxAbsScaler())),
(
"trans2",
make_pipeline(
TfidfVectorizer(), StandardScaler(with_mean=False)
),
),
]
yield Estimator(transformer_list)

elif "estimators" in sig:
# stacking, voting
if is_regressor(Estimator):
Expand All @@ -211,21 +225,21 @@ def _generate_meta_estimator_instances_with_pipeline():
# They should be able to work on any data and delegate data validation to
# their inner estimator(s).
DATA_VALIDATION_META_ESTIMATORS_TO_IGNORE = [
"AdaBoostClassifier",
"AdaBoostRegressor",
"BaggingClassifier",
"BaggingRegressor",
"ClassifierChain",
"IterativeImputer",
"MultiOutputClassifier",
"MultiOutputRegressor",
"OneVsOneClassifier", # input validation can't be avoided
"RANSACRegressor",
"RFE",
"RFECV",
"RegressorChain",
"SelfTrainingClassifier",
"SequentialFeatureSelector" # not applicable (2D data mandatory)
"AdaBoostClassifier",
"AdaBoostRegressor",
"BaggingClassifier",
"BaggingRegressor",
"ClassifierChain",
"IterativeImputer",
"MultiOutputClassifier",
"MultiOutputRegressor",
"OneVsOneClassifier", # input validation can't be avoided
"RANSACRegressor",
"RFE",
"RFECV",
"RegressorChain",
"SelfTrainingClassifier",
"SequentialFeatureSelector", # not applicable (2D data mandatory)
]

DATA_VALIDATION_META_ESTIMATORS = [
Expand Down