not tested

not tested

not tested

does going in steps of 10s make this faster?

• It is unclear why the pipeline does not contain the classifier. Now the preprocessing steps and the classifier are separated.
  Major consequence: Code is hard to understand.
  Minor consequence: The time measurements only involve the classification time.

One of the premises CountFeaturizer was built on (based on this Microsoft ML article on count featurization https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/data-transformation-learning-with-counts) was that it may be possible to reduce the time training and classification takes compared to something like one hot encoding due to there being less features generated. Here, the reason why the pipeline was split was because it's necessary to separate out the time it took to preprocess the data from the time it takes to train and use the classifier.

Can this experiment (running the three pipelines on an increasing number of trees) be put into a for loop? Removes duplicate code

(* Personally I'm a big fan of declaring the plotting labels and other stuff (l135, 136, 137) high up in the code, so
the script is reusable. for this it would be even better to put everything in a function but maybe this is me taking it
too far .. )

• _get_nested_counter: I know it's not a requirement to document private functions, but it would make the review
  so much easier to know what the components are supposed to do. E.g., what is a 'nested dictionary', what do you
  mean with layers and a 2d array of what in the end?

There is something in the CI tests (something to do with the pickle module) that does not allow you to do something like
a = {}
a[1] = {}
a[1][1] = 4

So this is a workaround

Should I add "This is a workaround due to some pickle issues in CI testing regarding creating nested dicts dynamically" to this as a comment?

• The code seems small and elegant for something with many functionalities.

• test_count_featurizer_inclusion() why not run all examples on both arrays?

• test_count_featurizer_multi_y same question as (1) three additional values doesn't make sense to me.
  Please explain in comments

Does my comment on line 2527 clarify this? The first column in y contributes to 1 of the columns being added since it only takes 1 unique value. The second column in y contributes to 2 of the columns being added due to having two values 0 and 1.

• line 2521 cf = CountFeaturizer(inclusion=[[0]]) this one can be checked independently (by crosschecking against
  an instance with normally invoked featurizer)
  ** same for line 2514

By this, do you mean comparing the output against cf = CountFeaturizer() and checking if they are equal? If you meant it that way, I don't think it will work because CountFeaturizer() is equivalent to CountFeaturizer(inclusion=[0, 1]) in this specific case.

• test_count_featurizer_multi_inclusion() the y value is differently encoded (2d array), which influences how to invoke
  the inclusion param. what happens if I invoke the inclusion in the vanilla way?

If by the comment you mean y = np.array([[0], [0], [0], [1]]) vs y = np.array([0, 0, 0, 1]) then it should be the same because of y = np.reshape(y, (-1, 1)). I can make a test case for that.

TODO: Test case for this scenario

• test_count_featurizer_multi_inclusion line 2527: Please explain how this results in 12 new outputs

Each column of y gets its own columns outputted for each unique value it can take. For example, here, the first column of y gets 2 unique values (0 and 1), so 2 columns are outputted for each inclusion list (since 'each' is used, there are 2 inclusion lists [0] and [1]). The second column of y gets 4 unique values (0, 1, 2, 3) so 4 columns are outputted for each inclusion list. As a result (4 + 2, the total # of different y values) x (2, the size of the inclusion list) = 12 columns are outputted

I would replace the last sentence with:

CountFeaturizer can be used as an alternative to one-hot encoding for non-linear estimators that do not work efficiently on high cardinality categorical variables such as ensemble of trees (random forests and gradient boosted trees).