Is there a case were you want one and not the other?
I think using them together is a common use case, and so I would try to make that easy.
Maybe make these two classes private and add a third that does the whole thing?
Or just add the third class?
Removing public interfaces is hard, adding them is easy.

Stacking is the one use case I had in mind. Train several classifiers, take each of their outputs and feed them into another model to "combine" them. I know this from HEP work (eg http://arxiv.org/abs/0903.0850 (p6 of the PDF), I'm sure there are more these guys were just the first I could find the paper for). Might be something people do in kaggle competitions??

Not sure it is "worth it" when taking into account your comment on public interfaces and removing them. Having two private classes with a third, public one, that does it all in one step sounds like a good move.

Is there a way to mark my classifier as "two class only" for the common checks? Alternatively does someone have a good idea what ThresholdClassifier should do in a three (or more) class situation?

Related to the stacking use case: #6674

@amueller any idea how to mark ThresholdClassifier as "two class only"/deal with several of the common tests that feed three class problems to the classifier?

A good idea how to support multi class for ThresholdClassifier would be welcome, so far I could not think anything up.

Err also is PredictionTransformer the same as VotingClassifier by any chance? ;)

we need this for two-class classifiers: #6599. For now, add it manually to one of the skip lists. Maybe just skip the multi-class tests?

Err also is PredictionTransformer the same as VotingClassifier by any chance? ;)

Aha! Yes, because I didn't read the docs for VotingClassifier to realise it has a transform(). D'oh.

well, I knew it but still needed to reread the previous discussion to see that that was the case lol.