Scikit-learn currently has no concept of column labels. I agree this would be a nice to have feature. Don't hold your breath for it. Storing the column names at all would be an important first step, which I hope to achieve soonish. This will not be available in 0.18, and the consistency check probably not in 0.19 either. Scikit-learn operates on numpy arrays, which don't (currently) have column names. This might change in the future, and we might add better integration with pandas.

Thanks for replying! Looking forward to the new version! 💯

Thanks for the edifying conversation guys!

I think a wrapper to ensure column order is maintained between fit and predict/transform would be a good idea.

I would guess the way to do it would be respect column names if they were present both for fit and predict, and respect column ordering if they were missing in either (or both). There is also feature selection that would need to abide. I'm sure its not as easy as it might first appear.

In my opinion this is an sklearn issue not a pandas issue. As I said, not complaining, not expecting it to be terribly easy.

I would like to solve this inside scikit-learn, but I don't think there is consensus to do that (yet).
I would argue that we should protect our users against silent wrong results, other voices might say that users shouldn't pass dataframes?

other voices might say that users shouldn't pass dataframes?

You could print a warning that says "Beware when using Dataframes. sklearn uses column ordering not column labels. Downcast to numpy arrays to avoid this warning".

That would have saved my bacon here.

Pandas prints out a warning with the view vs. copy "gotcha", and that was enough to motivate me to study and pre-empt any problem. This issue seems similar. http://bit.ly/2qR1fDH , http://bit.ly/2pRbR7Q

@pjcpjc warning is an option, but this would be very very loud. Many people are passing dataframes.
I guess it depends a bit on whether we want them to pass dataframes or not. One problem is that if we warn, that basically tells users to use .values everywhere to convert to numpy arrays, so they don't get a warning. But then this error becomes silent again, so we won nothing. We just made the users change their code so that we can never hope to catch the error.

Also, I think having the estimator know the feature names is helpful for model inspection.

I'd prefer for sklearn to respect column names if they are present both for fit and predict, and respect column ordering if they were missing in either (or both). But I also think a warning would be better than how it works right now, where its easy to get a false sense of security and end up with buggy code, which was my experience with this issue.

You can see my bug fix here. The original version didn't have cell 18. A warning would have flagged me that cell 18 was needed. The additional code changes to pass numpy arrays would be minor, if needed.

Just one opinion.

I'm not sure you got the problem with the warning. If there's always a warning, people will change their code so they don't get a warning, which is by passing in df.values, i.e. a numpy array without column names. Then later they change their code and columns mismatch again but they get no warning because they only passed a numpy array. So in the end, adding the warning didn't safe them at all, just created more work.

I totally agree that we should do better then we're doing now, I just think a warning whenever a dataframe is passed will not help.

I think I understand your point. My point is if the warning links to a "see here for details" documentation page, and that page takes great pains to point out that you need to manage the column positions outside of sklearn, then my guess it will save quite a few people.

I also suspect that the people who ignore the warning, and/or don't take it seriously and just do the simplest possible thing to make the warning go away, are probably un-saveable in any event. They will find some other way to muck up their data science practice.

That said - agree to disagree. It would have helped me - but it certainly is possible I'm too unusual to be of guidance.

Ok, I see where you're coming from. It might be a good way to make people aware that this issue potentially exists at all. I still thinks it's a bit too loud and unspecific, though it might indeed save some people. I hope for a near-ish term better solution ;)

I think warning for all pandas DataFrame input is overkill. Lots of people are loading DataFrames from static sources (as we recommend) and not reordering columns.

The options, as far as I'm concerned are:

• add some documentation on Scikit-learn with Pandas and pitfalls therein.
• add a common test which ensures column name equality, otherwise raising a warning (and perhaps in a later version an error) when mismatched.

Obviously the latter is a substantial effort, and adds O(n_features) space to the pickle representation of an estimator unless we use a hash, but ... dataframes are important to support well.

If we do this, I'd suggest we extend check_array, at least to do the checking, and perhaps even to help store the estimator attribute. In terms of syntax, this could be as magical as check_X_y(X, y, ..., estimator=self) in fit and check_array(X, ..., estimator=self) in predict/transform, but maybe we'd prefer check_X_y(X, y, ..., store_column_hash=self) and check_array(X, ..., column_hash=self._column_hash). (We could make it even more magical with decorators or a metaclass, but we don't do that 'round here.)

There will be cases where this is hard to do because check_array is called by a function called by the estimator, rather than directly within it. We'd also need to do the same for warm_start and partial_fit.

In short, this is possible, but it'll be a big project and will touch everything.

I would rather not store hashes, because this information is very valuable for model inspection.
I'm not sure I like passing self to check_array, I guess the only alternative would be to return column_names and then store it, and pass column_names=self._column_names in predict / transform.
(if we do this, we probably want to really look into get_feature_names and the pipeline processing of the same, because it'd be AWESOME!)

So, for now if I wish to use fit, and then use transform on another df, what is the proper way to be able to use the proper column order?
How can i guaranty pandas dataframe columns to be in a specific order?

Currently, estimators have ad-hoc checks that n_features match from fit to predict.

Maybe we should implement in BaseEstimator (for use after check_array):

def _fit_consistency(self, X):
    self.__n_features = X.shape[1]
    if isinstance(X, pd.DataFrame):
        self.__columns = X.columns

def _check_consistency(self, X):
    if self.__n_features != X.shape[1]: raise ...
    if isinstance(X, pd.DataFrame):
        if not hasattr(self, 'BaseEstimator__columns'): warn ...
        if self.__columns != X.columns: raise ??
    elif hasattr(self, 'BaseEstimator__columns'): warn ...

I just wanted to add that this feature is extremely important out in the real-world where the model building process and the prediction process (exposed to users) often are completely separate.

The prediction process should not need to know about feature selection, feature order and so forth. It should just take all the features for the specific problem and the estimator knows which ones it needs in what order. This is how it works in other tools. It's extremely convenient and the fact I prefer that tool over sklearn. Sklearn however offers more estimator types and better performance (as in speed and prediction quality even for same model types). So it would be great to have this here too.

I would imagine this happens either through pandas column names or if you pass in numpy arrays fit and predict could have an additional optional parameter column_names (not that great to create this list). If no column names are present, it works like it does now.

I don't think your suggestion is going to happen.

Well, here are the options (not all mutually exclusive):

1. Deprecate implicit conversion of DataFrames to NumPy arrays; in the future, the user would have to call .values explicitly in the future
2. Document it: Pandas Interoperability section in User Guide #10453 is up for grabs and your writing on the subject is fluent, @kienerj!
3. Call df.sort_index(axis=1) in the future (with a warning for a deprecation period as it is not backwards compatible), with the risk that this (a) increases runtime/memory; (b) does not identify changes in column names where the number of columns is preserved
4. Put some PandasProtect wrapper or mixin in sklearn-pandas and tell people to use it if they want to be safe
5. Invent a PandasCheck transformer (either here or in sklearn-pandas) and tell Pandas users to always use pipelines and put this at the beginning
6. Store column names in fit and raise a warning (or perhaps error) in transform etc if not matched.

Number 5. is by far the easiest, although it requires users to be aware, so it's coupled with 2. It looks like this (with documentation and an awareness campaign):

class PandasCheck(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        self.columns_ = getattr(X, 'columns', None)
        return self
    def transform(self, X):
        check_is_fitted(self, 'columns_')
        if not np.all(self.columns_ == X.columns):
            raise ValueError('Mismatched columns in transform. Expected %r, got %r' % (self.columns_, X.columns))
        return X

I've found these functions useful:

from sklearn.utils import check_array
def check_array_pandas(X: pd.DataFrame,
                       feature_names: List,
                       *args, **kwargs):
    ''' A wrapper for sklearn check_array

    Used to check and enforce column names if pandas dataframes are used.

    Parameters
    --------
    X: pd.DataFrame
        The input you want to check.
    feature_names: list of column names
        The column names you'd like to check and their ordering you want to enforce.
    *args
        *args for check_array
    **kwargs
        **kwargs for check_array

    Returns
    --------
    X: array like
        With columns ordered like feature_names and all other columns removed.
        Also then put through check_array.

    Raises
    --------
    KeyError:
        If any feature_names aren't in X.columns.
    TypeError:
        If X is not a pd.DataFrame.
    ValueError:
        If feature_names is None.

    '''
    if feature_names is not None:
        if isinstance(X, pd.DataFrame):
            if isinstance(feature_names, (collections.Iterable)):
                X = X[feature_names]
            else:
                raise TypeError('feature_names should be iterable.')
        else:
            raise TypeError('X should be a DataFrame for safety purposes.')

    return check_array(X, *args, **kwargs)

Do this in fit:

self.feature_names_ = list(X.columns) if isinstance(X, pd.DataFrame) else None

And this in init

self.feature_names_ = None

Then just replace all calls to check_array with check_array_pandas

wouldn't that work then ?

Please see the test cases in #12936

@jnothman what do you think ?

• I suspect we should raise an error when the order is different from in fit, rather than reordering, in part to alert users who thought their code was working in previous scikit-learn versions but actually were mixing their features. We can always change it to reordering later.
• feature_names_ is ambiguous. It needs a better name.
• I am a little hesitant about putting self.feature_names_ = list(X.columns) if isinstance(X, pd.DataFrame) else None in each estimator. I would rather this abstracted away into a function or method, e.g. set_X_signature(self, X) with check_array supporting X_signature to check integrity.

We could potentially make setting (not just checking) the signature a job for check_array, e.g. check_array(X, set_signature=self)

Getting someone to implement these changes (preferably someone who knows how to edit files en-masse) is more the challenge. In practice where an estimator delegates checking of X to some other function, it's not going to be trivial to implement everywhere.

Its difficult because things don't super baseestimator much in the codebase.

I disagree on raising an error. Data frames are different philosophically than arrays. People are likely passing data frames in the wrong order at the moment, but they aren't wrong to do so they just aren't aware the sklearn package hasnt handled their edge case. Pandas dataframe column ordering has always existed, but it's just not talked about, because anyone using pandas would never iloc a column when indexing... You index columns by name.

Maybe a warning as we transition, but I wouldnt recommend an error.

I absolutely agree that a warning is a must. I agree with your reply 100%.

A compromise may be to issue an update that creates a warning ASAP, by collecting column names where appropriate and creating the architecture of the solution. Then in the next breaking update, implement automatic reordering.

As for extra columns, in my own code I have implemented a lot of what we are talking about in the context of cross validated feature selectors, such as RFECV. By accepting dataframes of any size, you allow these learners to always be correct. User can give them just the columns they selected on predict, the data they fit on on predict, or any arbitrary dataframe on predict. The goal in my mind is to enable the agnostic use of estimators, because these estimators are going to be pickled and long forgotten in a repo somewhere. I dont want to use them mistakenly when I dig them out of the filing cabinet.

Is there any workaround for this problem. Other than reordering of the columns.

Hello,
I went through the same issue and tried this

df2 = df2[df1.columns]
df2 is the data frame where the columns need to be ordered like in the df1.

Hm we still don't have a slep for this, right? I think we should try doing one. @NicolasHug did you want to do that?
This is related to feature names, but actually the current feature name ideas might not solve this issue.
@adrinjalali do you have thoughts on whether we want to do this before doing the feature names?

I think the options are mostly listed in #7242 (comment)

@amueller I know you do not like this, but I tend to prefer PandasCheck
which also enables #12627 (comment) . This approach only does the check in the beginning. The major argument against storing is "pandas users need to use a pipeline to hav column name consistency".

The alternative is to store the names everywhere. I can't get over the fact that to have this consistency, users would need to pay a price in memory. Do we want users to buy more ram to get column name consistency?

I'm with @ryanpeach

By accepting dataframes of any size, you allow these learners to always be correct.

and my initial comment to this issue. This is from a naive end-user point of view not knowing the fundamental architecture of sklearn and just being concerned about making it easy for me to get correct predictions.

Do we want users to buy more ram to get column name consistency?

How much RAM does storing even thousands of column names really use up? I mean if you really go all-in and have thousand very long column names, it adds up to couple 100s of MB. As long as it is optional, I don't see the issue given the huge convenience gain. I mean getting 64 GB of RAM doesn't cost all that much, compared to developer time or even worse wrong results.

I want 6). I think 1, 2 and 3 don't solve the issue. 4 and 5 solve the issue if the user is aware of the issue and actively takes steps to prevent a problem from occurring. They sort-of solve the issue but not really the core issue of user expectations and silent errors. Also they are really verbose.

I guess in that I disagree with @ryanpeach in #7242 (comment) but his solution is not even on the list ;)

However, if the first step is a ColumnTransformer, then we actual do the "right" thing in that we rearrange the columns - makes me think, is inserting ColumnTransformer(remainder='passthrough') to the beginning of a pipeline actually implementing the feature @ryanpeach wants?

@adrinjalali do you have thoughts on whether we want to do this before doing the feature names?

I think this is independent of output_feature_names_, and I'd be in favor of the same kind of validation we have in ColumnTransformer everywhere. I think implementing the input_feature_names_ only if the given input has feature names makes kinda sense, and it can be implemented even if it's not propagated through the pipeline.

This should also be really easy to implement now that we have @NicolasHug 's work on _validate_data merged, and I don't think we need a SLEP for it, for the same reason that we didn't need a SLEP to implement the validation in ColumnTransformer.

@adrinjalali Cool, I totally agree (and encouraged @NicolasHug to draft a PR) ;)
Would love to hear @jnothman's thoughts as well

Also see #17407

Now that we validate column names, this is partially fixed if the user uses a ColumnTransformer, right?

Removing the milestone.

We now validate feature names by default and raise a warning if the column names are not in the same order. In 1.2, we will start raising an error.