On 19 June 2017 at 10:25, Andreas Mueller ***@***.***> wrote: ColumnTransformer([(f, LabelEncoder(), f) for f in fields]) Followed by some version of the one-hot-encoder, right? Or I guess with LabelBinarizer() it would be fine? If that's a correct implementation that allows for an inverse_transform and get_feature_names. I'm all for it. DictVectorizer().fit(frame.to_json(orient='records') Tbh I'm not familiar enough with how DictVectorizer treats integers and floats for that. Maybe a good argument would be the possibility of an exact inverse_transform which we decided is out-of-scope? There is also somewhere a hack that uses CountVectorizer(analyzer=lambda x: x) or something like that, and that also works for a single Column. If we actually decide (which was the consensus at the sprint between @GaelVaroquaux <https://github.com/gaelvaroquaux>, @jorisvandenbossche <https://github.com/jorisvandenbossche> an me) that we always want to transform all columns, then maybe one of these implementations could actually work. I would like something discoverable and with good feature names and the possibility to have some feature provenance in the future. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9151 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz67avpEDmrx4xAQQcUXb5FChCjjpPks5sFcAIgaJpZM4N9lyH> .

On 19 June 2017 at 18:26, Joris Van den Bossche ***@***.***> wrote: ColumnTransformer([(f, LabelBinarizer(), f) for f in fields]) If that's a correct implementation that allows for an inverse_transform and get_feature_names. I'm all for it. Problem with the LabelBinarizer is that it currently only works on strings, not numerical values (which could maybe be fixed), and that it doesn't play nice with the ColumnTransformer due to both X and y being passed in the transformers (but there is a PR to fix this?) It also doesn't give us a get_feature_names out of the box. DictVectorizer().fit(frame.to_dict(orient='records') Tbh I'm not familiar enough with how DictVectorizer treats integers and floats for that. Maybe a good argument would be the possibility of an exact inverse_transform which we decided is out-of-scope? DictVectorizer seems to work and gives us get_feature_names, but, it treats string values and numerical values differently. Strings are dummy encoded, but integer are just passed through. So not fully the behaviour we want. I also think the conversion to a dict (instead of working on the column as arrays) can become quite costly for larger datasets. There is also somewhere a hack that uses CountVectorizer(analyzer=lambda x: x) or something like that, and that also works for a single column. It is indeed using CountVectorizer(analyzer=lambda x: [x]) that gives us more or less exactly what we want. It also gives us get_feature_names (we only have to fix it to be able to deal with mixed strings and numerical values). So this could be used under the hood instead of LabelEncoder/OneHotEncoder. But given the quite different original use case, I am not sure this is a good way to go. Full experimentation of the different possibilities: http://nbviewer.ipython.org/d6a79e96b490872905e74202d0818ab2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9151 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz63Xf_VlQx-OoIJmIkwKrNYV3zPtbks5sFjCpgaJpZM4N9lyH> .

On 13 July 2017 at 05:59, Andreas Mueller ***@***.***> wrote: ***@***.**** commented on this pull request. Mostly looks good. Needs attribute documentation and then we can talk about what a good way to expose the attributes is. ------------------------------ In sklearn/feature_extraction/dict_vectorizer.py <#9151 (comment)> : > @@ -88,8 +89,8 @@ class DictVectorizer(BaseEstimator, TransformerMixin): See also -------- FeatureHasher : performs vectorization using only a hash function. - sklearn.preprocessing.OneHotEncoder : handles nominal/categorical features - encoded as columns of integers. + sklearn.preprocessing.CategoricalEncoder : handles nominal/categorical + features encoded as columns of arbitraty data types. is it? or strings or integers? What happens with pandas categorical? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > @@ -1730,13 +1728,14 @@ def _transform_selected(X, transform, selected="all", copy=True): class OneHotEncoder(BaseEstimator, TransformerMixin): - """Encode categorical integer features using a one-hot aka one-of-K scheme. + """Encode ordinal integer features using a one-hot aka one-of-K scheme. I don't think ordinal is right as they are not ordered. ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + categories : 'auto' or a list of lists/arrays of values. + Values per feature. + + - 'auto' : Determine categories automatically from the training data. + - list : ``categories[i]`` holds the categories expected in the ith + column. + + dtype : number type, default=np.float + Desired dtype of output. + + sparse : boolean, default=True + Will return sparse matrix if set True else will return an array. + + handle_unknown : str, 'error' or 'ignore' + Whether to raise an error or ignore if a unknown categorical feature is + present during transform. we always raise an error when categories is not auto and an unknown category is encountered during training, right? Maybe we should make that explicit? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + column. + + dtype : number type, default=np.float + Desired dtype of output. + + sparse : boolean, default=True + Will return sparse matrix if set True else will return an array. + + handle_unknown : str, 'error' or 'ignore' + Whether to raise an error or ignore if a unknown categorical feature is + present during transform. + + Examples + -------- + Given a dataset with three features and two samples, we let the encoder + find the maximum value per feature and transform the data to a binary maximum value? Unique values, right? Maybe add in a 999? or just a 5? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + find the maximum value per feature and transform the data to a binary + one-hot encoding. + + >>> from sklearn.preprocessing import CategoricalEncoder + >>> enc = CategoricalEncoder() + >>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]) + ... # doctest: +ELLIPSIS + CategoricalEncoder(categories='auto', dtype=<... 'numpy.float64'>, + handle_unknown='error', sparse=True) + >>> enc.transform([[0, 1, 1]]).toarray() + array([[ 1., 0., 0., 1., 0., 0., 1., 0., 0.]]) + + See also + -------- + sklearn.preprocessing.OneHotEncoder : performs a one-hot encoding of + integer ordinal features. This transformer assumes that input features I don't think ordinal is right. "This" -> OneHotEncoder (otherwise I feel it's ambiguous) ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + one-hot encoding. + + >>> from sklearn.preprocessing import CategoricalEncoder + >>> enc = CategoricalEncoder() + >>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]) + ... # doctest: +ELLIPSIS + CategoricalEncoder(categories='auto', dtype=<... 'numpy.float64'>, + handle_unknown='error', sparse=True) + >>> enc.transform([[0, 1, 1]]).toarray() + array([[ 1., 0., 0., 1., 0., 0., 1., 0., 0.]]) + + See also + -------- + sklearn.preprocessing.OneHotEncoder : performs a one-hot encoding of + integer ordinal features. This transformer assumes that input features + take on values in the range [0, max(feature)]. backticks? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + le = self._label_encoders_[i] + Xi = X[:, i] + if self.categories == 'auto': + le.fit(Xi) + else: + if not np.all(np.in1d(Xi, self.categories[i])): + if self.handle_unknown == 'error': + diff = np.setdiff1d(Xi, self.categories[i]) + msg = 'Unknown feature(s) %s in column %d' % (diff, i) + raise ValueError(msg) + le.classes_ = np.array(np.sort(self.categories[i])) + + return self + + def transform(self, X, y=None): + """Encode the selected categorical features using the one-hot scheme. Parameters? returns? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + "'ignore', got %s") + raise ValueError(template % self.handle_unknown) + + X = check_array(X, dtype=np.object, accept_sparse='csc', copy=True) + n_samples, n_features = X.shape + + self._label_encoders_ = [LabelEncoder() for _ in range(n_features)] + + for i in range(n_features): + le = self._label_encoders_[i] + Xi = X[:, i] + if self.categories == 'auto': + le.fit(Xi) + else: + if not np.all(np.in1d(Xi, self.categories[i])): + if self.handle_unknown == 'error': This is different from what the docstring says, but also reasonable behavior. ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + + X = check_array(X, dtype=np.object, accept_sparse='csc', copy=True) + n_samples, n_features = X.shape + + self._label_encoders_ = [LabelEncoder() for _ in range(n_features)] + + for i in range(n_features): + le = self._label_encoders_[i] + Xi = X[:, i] + if self.categories == 'auto': + le.fit(Xi) + else: + if not np.all(np.in1d(Xi, self.categories[i])): + if self.handle_unknown == 'error': + diff = np.setdiff1d(Xi, self.categories[i]) + msg = 'Unknown feature(s) %s in column %d' % (diff, i) during fit? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + return self + + def transform(self, X, y=None): + """Encode the selected categorical features using the one-hot scheme. + """ + X = check_array(X, accept_sparse='csc', dtype=np.object, copy=True) + n_samples, n_features = X.shape + X_int = np.zeros_like(X, dtype=np.int) + X_mask = np.ones_like(X, dtype=np.bool) + + for i in range(n_features): + valid_mask = np.in1d(X[:, i], self._label_encoders_[i].classes_) + + if not np.all(valid_mask): + if self.handle_unknown == 'error': + diff = np.setdiff1d(X[:, i], I guess negating valid_mask here might not necessarily make it faster? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + if self.handle_unknown == 'error': + diff = np.setdiff1d(X[:, i], + self._label_encoders_[i].classes_) + msg = 'Unknown feature(s) %s in column %d' % (diff, i) + raise ValueError(msg) + else: + # Set the problematic rows to an acceptable value and + # continue `The rows are marked `X_mask` and will be + # removed later. + X_mask[:, i] = valid_mask + X[:, i][~valid_mask] = self._label_encoders_[i].classes_[0] + X_int[:, i] = self._label_encoders_[i].transform(X[:, i]) + + mask = X_mask.ravel() + n_values = [le.classes_.shape[0] for le in self._label_encoders_] + n_values = np.hstack([[0], n_values]) same as np.array([0] + n_values), right? ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + categories : 'auto' or a list of lists/arrays of values. + Values per feature. + + - 'auto' : Determine categories automatically from the training data. + - list : ``categories[i]`` holds the categories expected in the ith + column. + + dtype : number type, default=np.float + Desired dtype of output. + + sparse : boolean, default=True + Will return sparse matrix if set True else will return an array. + + handle_unknown : str, 'error' or 'ignore' + Whether to raise an error or ignore if a unknown categorical feature is + present during transform. We should probably say what happens with the unknown value: all columns will be zero. ------------------------------ In sklearn/preprocessing/data.py <#9151 (comment)> : > + + Examples + -------- + Given a dataset with three features and two samples, we let the encoder + find the maximum value per feature and transform the data to a binary + one-hot encoding. + + >>> from sklearn.preprocessing import CategoricalEncoder + >>> enc = CategoricalEncoder() + >>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]) + ... # doctest: +ELLIPSIS + CategoricalEncoder(categories='auto', dtype=<... 'numpy.float64'>, + handle_unknown='error', sparse=True) + >>> enc.transform([[0, 1, 1]]).toarray() + array([[ 1., 0., 0., 1., 0., 0., 1., 0., 0.]]) + Attributes? seems pretty important ;) ------------------------------ In sklearn/preprocessing/tests/test_data.py <#9151 (comment)> : > @@ -1952,6 +1955,98 @@ def test_one_hot_encoder_unknown_transform(): assert_raises(ValueError, oh.transform, y) +def check_categorical(X): + enc = CategoricalEncoder() + Xtr1 = enc.fit_transform(X) + + enc = CategoricalEncoder() + Xtr2 = enc.fit_transform(X) + + enc = CategoricalEncoder(sparse=False) + Xtr3 = enc.fit_transform(X) + + assert_allclose(Xtr1.toarray(), Xtr2.toarray()) I don't understand this test. Determinism? ------------------------------ In sklearn/preprocessing/tests/test_data.py <#9151 (comment)> : > + assert sparse.issparse(Xtr1) + assert sparse.issparse(Xtr2) + return Xtr1.toarray() + + +def test_categorical_encoder(): + X = [['abc', 1, 55], ['def', 2, 55]] + + Xtr = check_categorical(np.array(X)[:, [0]]) + assert_allclose(Xtr, [[1, 0], [0, 1]]) + + Xtr = check_categorical(np.array(X)[:, [0, 1]]) + assert_allclose(Xtr, [[1, 0, 1, 0], [0, 1, 0, 1]]) + + Xtr = CategoricalEncoder().fit_transform(X) + assert_allclose(Xtr.toarray(), [[1, 0, 1, 0, 1], [0, 1, 0, 1, 1]]) We should probably explain the handling of constant features. Does this make the most sense? Not sure what else.... ------------------------------ In sklearn/preprocessing/tests/test_data.py <#9151 (comment)> : > + + Xtr = check_categorical(np.array(X)[:, [0, 1]]) + assert_allclose(Xtr, [[1, 0, 1, 0], [0, 1, 0, 1]]) + + Xtr = CategoricalEncoder().fit_transform(X) + assert_allclose(Xtr.toarray(), [[1, 0, 1, 0, 1], [0, 1, 0, 1, 1]]) + + +def test_categorical_encoder_errors(): + + enc = CategoricalEncoder() + X = [[1, 2, 3], [4, 5, 6]] + enc.fit(X) + + X[0][0] = -1 + msg = re.escape('Unknown feature(s) [-1] in column 0') Shouldn't it be feature value? Unknown category? ------------------------------ In sklearn/preprocessing/tests/test_data.py <#9151 (comment)> : > + enc.fit(X) + X[0][0] = -1 + Xtr = enc.transform(X) + assert_allclose(Xtr.toarray(), [[0, 0, 1, 0, 1, 0], [0, 1, 0, 1, 0, 1]]) + + +def test_categorical_encoder_specified_categories(): + X = np.array([['a', 'b']], dtype=object).T + + enc = CategoricalEncoder(categories=[['a', 'b', 'c']]) + exp = np.array([[1., 0., 0.], + [0., 1., 0.]]) + assert_array_equal(enc.fit_transform(X).toarray(), exp) + + # don't follow order of passed categories, but sort them + enc = CategoricalEncoder(categories=[['c', 'b', 'a']]) That's not documented, is it? ------------------------------ In sklearn/preprocessing/tests/test_data.py <#9151 (comment)> : > + assert_array_equal(enc.fit_transform(X).toarray(), exp) + + # multiple columns + X = np.array([['a', 'b'], ['A', 'C']], dtype=object).T + enc = CategoricalEncoder(categories=[['a', 'b', 'c'], ['A', 'B', 'C']]) + exp = np.array([[1., 0., 0., 1., 0., 0.], + [0., 1., 0., 0., 0., 1.]]) + assert_array_equal(enc.fit_transform(X).toarray(), exp) + + # when specifying categories manually, unknown categories should already + # raise when fitting + X = np.array([['a', 'b', 'c']]).T + enc = CategoricalEncoder(categories=[['a', 'b']]) + assert_raises(ValueError, enc.fit, X) + enc = CategoricalEncoder(categories=[['a', 'b']], handle_unknown='ignore') + enc.fit(X) We should check the outcome for that. We only get two columns, right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9151 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz65TNiOEMksLTr_q1OtNRfrK2KYR2ks5sNSWUgaJpZM4N9lyH> .