I agree that using make_column_selector when selecting a DataFrame with integer column names is not great.

As you noted ColumnTransformer interprets integer selection as positional indices of the DataFrame. This behavior is detailed in the ColumnTransformer docstring:

Integers are interpreted as positional columns, while strings can reference DataFrame columns by name.

The semantics for ColumnTransformer was decided a while ago and changing it would be backward incompatible. For now, the workaround would be to avoid using integer column names.

Assuming we do not change ColumnTransformer, here is a API proposal: We can add a returns= kwarg to make_column_selector such that:

1. make_column_selector(..., returns="names"): The default which is what we do now.
2. make_column_selector(..., return="positional"): Returns the columns as positions. For your use case, you can use this to select on any datafarme regardless of dtype of the columns.

What could possibly break if you apply the same rule:

Integers are interpreted as positional columns, while strings can reference DataFrame columns by name.

to _get_column_indices ?

I skimmed through an older version of code, and I think it used to be exactly this way, see e.g. this commit of #14035 . Probably some time after that, someone changed the logic: instead of checking the type of passed indices they decided to check the type of the dataframe's column Index.

P.S. I got a bit confused with the details, but I have a strong feeling that the default behaviour of make_column_selector can be changed without any harm done. I see a little problem with your proposition: if I want my pipeline to work well when the data is like df['1'] and also when it is df[1], I have to use make_column_selector(..., return="positional") and completely forgo the convenience offered by strings.

if I want my pipeline to work well when the data is like df['1'] and also when it is df[1], I have to use make_column_selector(..., return="positional") and completely forgo the convenience offered by strings.

Selecting columns based on two different types is not well supported and would require a design discussion. It also breaks backward compatibility.

If this is a major issue, I think we could add a use_index_directly parameter to ColumnTransformer that turns off these semantics and selects from the column directly. For example:

ct = ColumnTransformer([('t2', Normalizer(norm="l1"), [1, 2])], use_index_directly=True)
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})

ct.fit_transform(df2)
# Internally we will do `df2[[1, 2]]` for selection.

What do you think?

Your solution seems reasonable to me. I think that some users might want ColumnTransformer to have this parameter even if they don't use make_column_selector or other callables. I would also be fine even with a cautionary note in the documentation as an interim solution.

What bothers me is why ColumnsTransformer and make_column_selector use different semantics in the first place. Namely, the output of make_column_selector is always names (str or int), while the input of ColumnsTransformer is either names (str) or positional (int). Afaiac, they were introduced together to be used together.

Notwithstanding your proposed solution, would it make sense to change the behaviour of make_column_selector and break the backward compatibility in the future? (if the policy allows, through a deprecation cycle or somehow). I wonder, whether ColumnsTransformer is the only "user" of make_column_selector or if there are conceivable alternative use cases for make_column_selector, in which the lack backward compatibility can break actual code.

Okay, I see why changing the default behaviour of make_column_selector in the future is even worse than changing it now.

After your proposed solution is implemented, I will be using the parameter ColumnTransformer(..., use_index_directly=True) to make the pipeline work with both str and int column names. If make_column_selector is changed after that, then I will have to remove use_index_directly=True from all my code. So it is now or never.

After your proposed solution is implemented, I will be using the parameter ColumnTransformer(..., use_index_directly=True) to make the pipeline work with both str and int column names.

Yup! That is the goal. The other benefit of use_index_directly is that it allows for using panda's multiindex: #13781