@jnothman , here's a continuation of #7327 . My big question is whether or not a new transformer object would be acceptable. There's a lot of complexity in here which is wrapped up with the range-checking behavior of the existing OneHotEncoder.

How do you feel about moving this functionality to a new class, perhaps named CategoricalEncoder, and leaving the existing OneHotEncoder the way it is?

There already was a failed attempt at a new class at #6559

My time is very limited at the moment due to teaching. You've marked this WIP. What do you feel remains to be changed before this is in a state to be reviewed-for-merge?

I did a bit more work on the code. I think the changes to OneHotEncoder are in good shape now, but I still want to look over the tests and see if there's any cases to add. This also needs an entry in the What's New, and I haven't looked over the documentation yet. I put up a checklist for these items. I'll be able to work on this more this week.

Changes with this latest update:

• Removed 'error-strict' option for handle_unknown and added 'auto-strict' option for values. I think it's easier to write the code for the options set up this way, and arguably the original OHE behavior is a choice about how to fit the data. "Auto" is the original behavior, while "auto-strict" only uses categories from the input data.
• Restored active_features_, since it's the easiest way to maintain the original class behavior with values='auto'.
• Restored n_values_. It still seems potentially of use, but I don't feel strongly about it.
• Removed the max value check. I'm handling this with the LabelEncoders now.
• Removed the cast to np.object. This converted inputs of e.g. integers into object arrays, which I'd rather not do. The downside(?) is that lists of mixed string and numerical types get converted into string types, which should only matter with values='auto'.

Sorry, looks like I have some test failures to deal with as well.

@jnothman , this is ready to be reviewed for merge now.

API thoughts:

I think the cleanest interface would be to have values accept either "auto" or a sequence of sequences which specify the expected levels for each categorical. handle-unknown should be either "ignore" or "error". (Possibly there could be an extra option to encode an "unknown values" column, but adding that would be out of scope for this PR.)

That wouldn't let users completely reproduce the current behavior of ignoring unseen integers inside a 0-to-max-fit range but erroring on integers outside of that range. I'm curious to know how many people rely on that behavior, but I don't know how I'd find out.

If we're going to keep the current OHE "auto" behavior, I'd add a new option for handle_unknown, perhaps "error-out-of-range", and ask users to switch to that if they want the old-style behavior.

How to get there:

Writing this, I switched back and forth between thinking of the current OHE behavior on "auto" mode as being a different way of fitting and a different way of raising errors. Now I'm back to thinking that it's about raising errors. I'm inclined to get rid of "auto-strict" and use "error-strict" again. I'd do that without significant code changes, by checking the handle_unknown value at fit time. If we don't keep the current "auto" behavior, I'd also deprecate active_features_.

I'm inclined towards error-strict from a user orientation. The auto-strict definition makes the current auto behaviour confusing vis-a-vis active_features_ (even if active_features_ should be more of an implementation detail than it is because of the current error behaviour).

I think you should always assume that someone has taken advantage of the current behaviour, and aim to maintain/deprecate it.

(Supporting strings here makes this very close in behaviour to DictVectorizer (especially since DictVectorizer currently assumes numeric features or single-valued categorical features). I suspect it won't be faster than DictVectorizer. What do we gain by this alternative interface, other than a reduction in user upset.)

I switched back to "error-strict" and got rid of "auto-strict". I was also more aggressive with deprecations. I moved active_features_ back to deprecated, which will hopefully remove some confusion about whether fit attributes are masked. There's a lot of deprecations now, but the OneHotEncoder would need much less code in another two versions.

Could you explain more about how someone would use a DictVectorizer for one-hot-encoding categoricals? It seems like there'd need to be a lot of extra preprocessing to make that work. With the OneHotEncoder in this PR, you can read in a CSV and send it straight into a Pipeline that starts with a OneHotEncoder. That's valuable to me.

class PandasVectorizer(DictVectorizer):
    def fit(self, X, y=None):
        return super(PandasVectorizer, self).fit(X.to_dict('records'), y)
    def fit_transform(self, X, y=None):
        return super(PandasVectorizer, self).fit_transform(X.to_dict('records'), y)
    def transform(self, X):
        return super(PandasVectorizer, self).transform(X.to_dict('records'))

are we wasting our time?

What are your thoughts on whether we need this functionality beyond Pandas/DictVectorizer? Should we start by putting an example of the above somewhere, with the possibility that we'll leave OHE alone?

That's a nice implementation. I'd forgotten about to_dict; I don't often have reason to use it.

I think there's a number of simple ways to write a transformer which one-hot-encodes categorical features. Most of the terrible complexity of this PR is so that the OneHotEncoder can do both the old thing and the new thing at the same time. That PandasVectorizer may be the simplest implementation, assuming you're using pandas, but I'd need to poke at speed and memory use before I trusted it for anything big.

Personally, I'd like to see an official, off-the-shelf solution for categorical expansion. My own use case is that I'm writing code to automate model fitting and prediction. It involves sending Pipeline objects to AWS and retrieving them afterwards. Right now I'm handling categorical expansion outside of the Pipeline (I've been trying to avoid custom code inside it so that users don't need to install anything extra), but it would be nice to be able to include all of the steps together.

FWIW, I think people are often surprised that scikit-learn doesn't already handle one-hot-encoding of string categoricals. I was surprised, and spent a while digging through the documentation to confirm it for myself. Coworkers I've talked to have also been surprised.

Whether this functionality comes through a modified OneHotEncoder or a new class, I do think that handling string categorical features is a common enough use case that it should be supported through code in scikit-learn.

Part of the confusion here is that numpy really doesn't do a neat job of managing strings and mixed data types. I mean, it has struct arrays / recarrays, but we're not even handling them here.

Perhaps what we need these days is a DataFrameVectorizer. In particular, this would record input column structures (perhaps even when MultiIndex is involved) and dtypes to enable inverse transformation. It would one-hot encode strings. An initial implementation could build upon DictVectorizer although it may be better to operate on a DataFrame column-wise. WDYT? @amueller?

I agree that numpy arrays are really the wrong data structure for mixed string/numerical data. I think that a transformer which operates column-wise on a DataFrame would be a useful addition. If this is going to be in scikit-learn itself, then my intuition is that column-wise operation would be more efficient (although a few more lines of code) than your PandasVectorizer suggestion.

I would happily work on such a contribution, if that's something you'd accept. Would you be willing to have a DataFrame-specific transformer? I don't think it would require a pandas dependency (except for testing), but it would assume that its input was a DataFrame.

ColumnTransformer is quite close to this, but we would need to allow using something like LabelEncoder. The trouble I foresee is that if we want to allow encoding non-textual data, you couldn't do it through the same framework simply because those transfrormers (e.g. OneHotEncoder) require rectangular arrays, not vectors.

I hadn't seen these last comments. I would like to also support arrays, maybe only the simple cases. I think @GaelVaroquaux would be ok with having a transformer that is mostly intended to work on dataframes.
I'm really not sure if keeping the OneHotEncoder is worth it.
We could add a new class to the proposed experimental module.
the main question then is which kind of ndarray inputs would we want to support. I would REALLY REALLY like to move forward on this....

Discussion with @GaelVaroquaux and @jorisvandenbossche concluded with: we want a new estimator, and it will work on numpy arrays, and we will one-hot encode all columns, and there's no way to subset the columns, and people should use ColumnTransformer for that, and we'll make sure that this use-case is very simple.
Basically ColumnTransformer should be the one-stop-shop for mixed data. If we do a selection of columns in OneHotEncoder, then we haven't solved the problem of scaling and imputation, and people would need to use the ColumnTransformer anyhow.

From #9012 the code I'd like to write:

data = pd.readcsv("stuff")
categorical = data.dtypes == "object"
preprocess = ColumnTransformer(
    ('categorical': make_pipeline(Imputer(strategy='mode'), CategoricalEncoder()), categorical)
    ('continuous': make_pipeline(Imputer(strategy='mean'), StandardScaler()), ~categorical))
X = data.fit_transform(preprocess)

Closing, since this isn't going to merge and has been superseded by #9151 .

Yes, sorry, @stephen-hoover. Feedback on the new (or old ..) API is always welcome!