Reference Issues/PRs

This is a draft implementation of #6424.
It doesn't really introduce anything new in the API, but I'm happy to move this to a SLEP.
Below is an initial description. Happy to include feedback in the SLEP.

Fixes #6425

What does this implement/fix? Explain your changes.

The main idea of this is to make compound scikit-learn estimators less opaque by providing "feature names" as strings.

Motivation

We've been making it easier to build complex workflows with the ColumnTransformer and I expect it will find wide adoption. However, using it results in very opaque models, even more so than before.
We have a great usage example in the gallery that applies a classifier to the titanic data set. To me this is a very simple standard usecase.

Markdown doesn't let me paste this as details, so just look here:
https://scikit-learn.org/dev/auto_examples/compose/plot_column_transformer_mixed_types.html

However, it's impossible to interpret or even sanity-check the LogisticRegression instance that's produced here, because the correspondence of the coefficients to the input features is basically impossible to figure out.
This PR enables using get_feature_names to obtain the semantics for the coefficients:

preprocessor.get_feature_names(X.columns)

['num__age',
 'num__fare',
 'cat__embarked_C',
 'cat__embarked_Q',
 'cat__embarked_S',
 'cat__embarked_missing',
 'cat__sex_female',
 'cat__sex_male',
 'cat__pclass_1',
 'cat__pclass_2',
 'cat__pclass_3']

I think this is essential information in any machine learning workflow and it's imperative that we allow the user to get to this information in some way.

The proposed API will add a method get_feature_names to all supported (see below) transformers, with a (possibly optional, see below) parameter "input features" which is an array-like of strings.

Alternative Interfaces

To me there are four main options for interfaces to enable this:

1. Implement transformative get_feature_names as in this PR
2. Implement a more comprehensive feature description language (as done in ELI-5, I think)
3. Tie in more strongly with pandas and use dataframes / column names
   a) to output feature semantics.
   b) to determine feature semantics
4. Leave it to the user.

While I think 2) and 3) a) is are valid option for the future, I think trying to implement this now will probably result in a gridlock and/or take too much time. I think we should iterate and provide something that solves the 80% use-case quickly. We can create a more elaborate solution later, in particular since this proposal/PR doesn't introduce any concepts that are not in sklearn already.
3 b) is discussed below.

I don't think 4) is a realistic option. I assume we can agree that the titanic example above is a valid use-case, and that getting the semantics of features is important. Below is the code that the user would have to write to do this themselves. This will become even harder in the future if the pipeline will do cloning.

numeric_fitted = preprocessor.named_transformers_.num

num_features_transformed = np.array(numeric_features)[np.logical_not(np.isnan(numeric_fitted.named_steps.imputer.statistics_))]
categorical_fitted = preprocessor.named_transformers_.cat
cat_features_transformed = categorical_fitted.named_steps.onehot.get_feature_names(categorical_features)

feature_names = np.hstack([num_features_transformed, cat_features_transformed])

Scope

I suggest we limit get_feature_names to transformers that either:

• leave columns unchanged
• Select a subset of columns
• create new columns where each column depends on at most one input column.
• PolynomialFeatures (or possibly algorithms that create combinations of O(1) features)

Also, I want the string to only convey presence or absence of features, or constant functions of the features. So scaling would not change a feature_name, while a log-transformation (or polynomial) might. This limits the complexity of the string (but also it's usefulness somewhat).

Together, these mean that there will be no support for multivariate transformations like PCA or NMF or KMeans.

Implementation

Given the above scope and API, and the current implementation of get_feature_names in ColumnTransformer there are two main mechanism that need to be implemented.

1. allow pipeline to pass around names
2. provide a mechanism for meta-estimators (ColumnTransformer and Pipeline and Feature Union) to "know what to do".

There are basically three cases the meta-estimators need to take care of:
a) The transformer does a non-trivial column transformation, like OneHotEncoder or feature selection
b) The transformer does "nothing" to the columns, like StandardScaler.
c) The transformer does a "too complex" operation to the columns, like PCA.

For a), only the estimator can handle this case, so the estimator needs to provide a function to do that - already implemented in several cases as a transformative get_feature_names. For b) the meta-estimator can simply do a pass-through, so we need to "flag" these in some way. There is no way for the meta-estimator to really handle c) so if the estimator is not "tagged" as being trival and doesn't implement get_feature_names the meta-estimator needs to bail in some way.

I added a "OneToOneMixin" to tag the trivial transformations. It would be possible to just use this as a tag, and let the meta-estimators handle the pass-through. Given that we already have the mechanism to handle the pass-through, I thought it would be simpler to just implement a pass-through get_feature_names (another alternative would be to add an estimator tag, but that also seems less elegant).

Right now the bail in case c) is a TypeError.

Limitations

• The general API requires "input features". In PolynomialFeatures this was optional. Unfortunately we have no way to know the input dimensionality of a fitted transformer in general, so automatically generating x1, x2, etc is not possible. This could be fixed by adding a required n_features_ to the API, which would probably be helpful but also would be a relatively heavy addition.

• Because we don't know the number of input features, there's no way to ensure the user passed the right length of input_features

• The implementation of get_feature_names in Pipeline is a hack, because it includes or excludes the last step based on whether the last step has transform. The reason for this is that given a trained pipeline with a classifier in the end, I want to be able to get the feature names, which would not include the last step. In preprocessing pipelines we always want to include all the steps, though.
  The real solution to this in my opinion is always to include the last step, and allow slicing the pipeline ([MRG+1] Pipeline can now be sliced or indexed #2568) to get the feature names for a pipeline with a final supervised step.

• Bailing to a TypeError if any "complex" transformation happens is a bit of a bummer. We could try to generate names like pca_1, pca_2, ... but to do this automatically we would need to know the output dimensionality, which we don't (unless we add n_outputs_ as required attribute to the API similar to n_features_ above)

Open Questions

• Do we want to require get_feature_names to accept input_features? Right now the vectorizers don't and it makes the code slightly more complex.

• How do we want to handle the hack in Pipeline.get_feature_names for the last step?

• Do we want to encode fixed univariate transformations ("scale", "log", "rank"?)

Possible Extensions

• don't require input_features and generate names
• generate names for "complex transformations"
• Use pandas column names as input_features if available (3b above)

I already discussed the requirements for the first two extensions (adding n_features_ and n_outputs_).
The last one would require storing the input column names if the input is a pandas dataframe. It shouldn't be hard to do, and would also enable solving #7242 and I'd like to do that, but it's not required for this proposal to be useful.

Todo

• add to narrative docs
• add to remaining estimators (feature selection is the only left?)
• allow input_feature for all get_feature_names methods?