Btw I think this PR will work better with #803, as this enforces a more consistent API.

The following classifiers fail on iris with default params: SGDClassifier, Perceptron, BernoulliNB.
I tried to normalize the features with Scaler. That didn't help but then NearestCentroid also failed.
Oh and btw, I am talking about training set performance.

I am a bit surprised that SGDClassifier has problems. It seems a bit hard to tune.

What surprises me is that Perceptron and NearestCentroid fail. Both are more simpler than SGDClassifier with less hyperparameters to tweak.

Also true. NearestCentroid "only" fails when using Scaler... but also shouldn't. You're right.

I haven't looked into it this much, as there are also other tests failing.

I have trouble testing the NB classifiers. Maybe @larsmans has some opinion on that. BernoulliNB by default thresholds at zero, so it can not handle non-negative data. MultinomialNB on the other hand assumes the entries to be counts, in particular non-negative. Neither raised any error when given arbitrary data.

My understanding is that MultiomialNB should raise an error on negative data.
I would like it if all estimators can handle zero mean data or raise an error if it doesn't make sense (as in the case of MultinomialNB).

@amueller Shuffle the iris dataset first. SGDClassifier and Perceptron are sensitive to the shuffling and by default the iris dataset is not shuffled.

@mblondel Thanks. That one bit me already so often ;) I reverted the SGD default changes since it seems to work. It is quite unstable though, and I think adding some more iterations by default might be good. What do you think?

You could add a list of estimators that must be handled specifically: this way you could add SGDClassifier with extra parameters.

Well, for the moment it works. My question was more: how sensible are the current defaults. Usually I use much more than 5 iterations. I guess these are good with huge, very redundant datasets. But would it hurt to do 20 iterations instead?

Ideally we should have a large number of iterations (e.g. 100) + some early stopping criterion based on a exponentially moving average of the test error measured from a validation test subsamples from the training set if not passed explicitly.

2012/6/12 Andreas Mueller
reply@reply.github.com:

I have trouble testing the NB classifiers. Maybe @larsmans has some opinion on that. BernoulliNB by default thresholds at zero, so it can not handle non-negative data. MultinomialNB on the other hand assumes the entries to be counts, in particular non-negative. Neither raised any error when given arbitrary data.

Well, BernoulliNB can handle negative data, though it doesn't really
make much sense to feed it that. It wants booleans.

My understanding is that MultiomialNB should raise an error on negative data.

That's right. It didn't, so far, because our tf-idf implementation was
buggy and could produce negatives.

@larsmans Maybe I misread you or you misread me: I wanted it say BernoulliNB can not handle all positive data, since it thresholds at zero by default. It handles zero-mean data "reasonably".

About tf-idf: was that fixed? I remember some PR hanging around a while ago.

@ogrisel wouldn't it also be good to stop on the training error? In my view, the early stopping is more of a convergence check than a regularization. And I'm a bit afraid of doing to much behind the scenes.

@amueller: ah, I get you now. No, BernoulliNB by default doesn't handle all-positive feature values nicely. I'm not sure whether that should raise an exception, though; a user might want to grid search for an appropriate threshold. (Or does the grid searcher catch exceptions from estimators?)

I think @ogrisel fixed tf-idf (?).

@larsmans: My idea was that having a per-feature median might be more natural as default threshold instead of 0. That was just something that came to my mind, not sure if it makes sense. I'm really not familiar with the kind of data that is used with this classifier.

The common application domains of Bernoulli NB, AFAIK, are text classification and NLP tasks like WSD with word window features. In both cases, the median is likely zero for each feature except the stop words.

Yes it's "fixed": it no longer outputs negatives values. However it does so in a non-standard way and I might change it again in the future. However last time I tried to implement the canonical normalization from the text books, the kmeans text clustering example was behaving much worse hence I decided to keep the current scheme for now as I did not have time to investigate further.

@larsmans @ogrisel Did the MultinomialNB check in #908.

Any comments? Should I merge? There are some smaller fixes that would be good to have in master. I can also cherry pick them and wait for a proper review on the testing stuff....

If all tests pass on your box, +1 for merging. More tests is always better :)

Ok then I'll have a look again tonight and if everything works I'll merge.

Runs fine. Still needs work but should be a good place to start from.
Merging now :)

It seems that some of the new tests are broken on python 2.6: https://jenkins.shiningpanda.com/scikit-learn/job/python-2.6-numpy-1.3.0-scipy-0.7.2/703/consoleText

Yes, I saw that but didn't have time to fix it. I hope I can do it later today.

Should be fixed now.

Thanks!