feel free to squash my commits

So the current way to specify a transformer is like this:

ColumnTransformer({"name": (Transformer(), column), ..})

(where 'name' is the transformer name, and column is the column on which to apply the transformer).

There was some discussion and back-and-forth about this in the original PR, and other options mentioned are (as far I as read it correctly):

ColumnTransformer([("name", Transformer(), column), ..]) # more similar to Pipeline interface

or

ColumnTransformer([('column', Transformer()), ..])   # in this case transformer name / column name has to be identical

BTW, when using dicts, I would actually find this interface more logical:

ColumnTransformer({"column": ("name", Transformer()), ..})

which switches the place of column and transformer name, which gives you a tuple of (name, trans) similar to the Pipeline interface, and uses the dict key to select (which mimics getitem how also the values are selected from the input data).
But this has the disadvantage that we cannot extend this to multiple columns with lists (since lists cannot be dict keys).

ColumnTransformer({"column": ("name", Transformer()), ..})

The column is a numpy array, right, so it's not hashable.

I think we could use either the list or dict thing here, and have a helper make_column_transformer or somthing that does
make_column_transformer({Transformer(): column})
Transformer is hashable, so that works, and we can generate the name from the class name as in make_pipeline.

Oh I didn't fully read your comment. I think multiple columns are essential, so we can't do that...

Maybe i'm a little bit confused but then, does this overlap in scope with FeatureUnion?

If there is more than one transformer, we need to know what order to use for column_stacking their output, right? So if we use a dict with transformers as keys can we guarantee a consistent order?

Let's try to discuss this with @amueller before proceeding. I think I'll help on this today.

Additional problem that we encountered:

In the meantime (since the original PR was made), Transformers need 2D X values. Therefore, I made sure in the ColumnTransformer that I always pass through the subset of columns to the transformer as a 2D object, also when you apply the transformer only on a single column.

But, by ensuring this, the example using a TfidfVectorizer fails, because that one expects a 1D object of text samples.

So possible options:

1. Add a way to specify in the ColumnTransformer that for certain transformers the passed X values should be kept as 1D.
   E.g. we could have

   ColumnTransformer([('tfidf', TfidfVectorizer, 'text_col'), ...],
                     flatten_column={'tfidf': True})`
   

   where flatten_column (or other name like keep_1d) is False by default (satisfying the normal Transformers), but you can specify per transformer if you want to override this default of False.
   The use of a dict here is similar to the transformer_weights keyword.

2. Adapt the example (which means: letting the user write more boilerplate) to fix it, eg by adding a step in the pipeline to select the single column from the 2D object.
   We could add one line to the current pipeline that holds the TFIDF (this tuple is one of the transformers in the ColumnTransformer)

           # Pipeline for standard bag-of-words model for body
           ('body_bow', Pipeline([
   added -->   ('flatten', FunctionTransformer(lambda x: x[:, 0], validate=False)),
               ('tfidf', TfidfVectorizer()),
               ('best', TruncatedSVD(n_components=50)),
           ]), 'body'),
   

3. Adapt TfidfVectorizer to eg have a keyword that allows to specify that 2D data is expected (which would be False by default for backwards compatibility).
   If we would like to do this one, this might ideally be a separate PR and so the second option can be used as a temporary hack to the example to have it working and which can be removed in the other PR.

Another option would be:

4. Make a distinction between specifying the columns as a scalar or as a list: when using a scalar, the data is passed as 1D to the Transformers, as a list as 2D data.
   The disadvantage of this is that for all Transformers except of the text vectorizers, you will have to specify the single column as a list:

   ColumnTransformer([('tfidf', TfidfVectorizer, 'text_col')
                      ('scaler', StandardScaler(), ['other_col'])])
   

Added last update: added additional error for get_feature_names, and moved the docs to compose.rst.

As far as I am concerned, somebody can push the green button ;-)

Indeed: let's see how this flies!

Thanks Joris for some great work on finally making this happen!

Woohoo, thanks for merging!

Great work congrats !

Really nice!!! Let's stack those columns then ;)

Looking forward for the next release

Next release will be amazing !

OMG this is great! Thank you so much for your work (and patience) on this one @jorisvandenbossche

Thank you @jorisvandenbossche!! Great stuff.
I have a question regarding this feature (I'm very new in GitHub, not sure if this is the right place... apologies in advance):

Is there going to be an effort (I would like to contribute) to implement get_feature_names in the majority of transformers?

I find that one of the big advantages that DataFrameMapper from sklearn-pandas brought to us also is the ability to trace names of derived features (using aliases and df_out=True), with what this means for interpretability (e.g. get some feature importances for a tree based model after a fairly complex non-sequential preprocessing pipeline). Having get_feature_names working consistently in ColumnTransformer would be the bomb.

What do you guys think?

Thank you in advance.

@partmor See #9606 and #6425. What exactly was working with DataFrameMapper that's not currently working? I feel like the main usecase will be with OneHotEncoder/CategoricalEncoder who will provide a get_feature_names. For most multivariate transformations it's hard to get feature names, so I don't know how DataFrameMapper did that.

@amueller thank you for the links. For instance, if we want to use StandardScaler on a set of numeric variables, ColumnTransformer raises an exception because SC does not have get_feature_names implemented. In "1 to 1" column transformations like standard scaling in DataFrameMapper you could just passthrough the feature names: ['x0', 'x1',...] to ['x0', 'x1',...]. ColumnTransformer.get_feature_names() just raises exception by using SC in it.

@partmor yeah for univariate things like that it would be easy to implement. We should revisit #6425 keeping in mind that ColumnTransformer relies heavily on it.

Just learned of this class. Very happy to find it!

Was wondering if there ever might be support for allowing elements of columns to be ColumnConstructor themselves or even Transformers. (I actually have been writing a Transformer to do this in the last little while unaware of ColumnConstructor).

The composition operation is natural: a Transformer, given X will ultimately produce some columns that can be column stacked. This implicitly would require a Pipeline whenever one of the columns was a Transformer. Indeed, this is essentially how I've coded things in my use case (to be clear my base object is not really a Transformer so it can't be used in a GridSearch, it is just an object that can transform columns). It would likely be hard to expose parameters of such embedded Transformers but GridSearchCV should be able to work on the top level objects as in https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py

For a (perhaps far-fetched to sklearn) use case, consider interactions in regression models and in particular interactions between transformed columns and / or categorical variables. So, one might want to have something like

pcaX = ColumnTransformer(['pca(X)', PCA(n_components=4), range(10)])
color = ColumnTransformer(['color', OneHotEncoder(), [10])
interaction = ColumnTransformer(['pca(X):color', Interaction, [pca, color])

Here Interaction would be a transform that forms the columns of the interactions, i.e. pairwise products (this probably exists in sklearn but I'm not 100% certain). You might ask who wants interactions between PCA components and a categorical variable? Not sure, but you could replace PCA with some other rich basis and get varying coefficient models easily. Setting n_components in a GridSearch would seem to be tricky but maybe the dunder notation could just be propagated deeper?

If any interest, I have some code I could share that builds these objects though ColumnTransformer is not a Transformer per se. The on Transformer I have built actually takes a list of (my analog of)ColumnTransformer as constructor, but that's a distinction without a difference. My intent is to manipulate this other object by e.g. slicing the transformers_.

Lastly, I haven't yet figured out how well ColumnTransformer​ will respect pd.DataFrame. If a ColumnTransformer​ fit on a dataframe with columns ['X', 'Y', 'Z', 'W'] only refers to ['X', 'Z'] can one call transform​ on a dataframe with columns ['X', 'Z'] (implicitly assuming remainder="drop"​). I'm guessing this will fail in sklearn​ but it is kind of convenient in some cases if you're going to work with dataframes a lot.​

The behaviour will be defined as follows:

So if you drop, it should be fine.