GridSearchCV and RandomizedSearchCV compare the scores to select the best estimator.

@amueller Good point. I would argue that this change is still okay, because:

1. This change only affects user-defined scorer arguments.

2. The user might want their user-defined scorer to return non-numbers.Number objects even when using GridSearchCV or RandomizedSearchCV. Looking at BaseSearchCV._fit, as long as the returned objects support the *=, +=, /=, and < operators, it will all still work. Yay duck typing. (Granted, this scenario seems unlikely, but see my next point.)

3. If the user's scorer returns weird objects that don't work with GridSearchCV and RandomizedSearchCV, an error will still be thrown. E.g. In my tuple example, I get an error like:

   TypeError: unsupported operand type(s) for +=: 'int' and 'tuple'

@acu192 I think this was put there particularly to throw a better error than "TypeError: unsupported operand types", probably because someone spend hours of debugging on it and then complained in the issue tracker ;)
We are actually working on scorers that return more than just a single float, and using multiple scorers, see #7388. The fix will be a bit more complicated than this, though. In particular, we want grid-search on more complex metrics.

Back in the days before scorer objects (pre scikit-learn 0.13, iirc), this kind of thing worked. We've been trying to fix it ever since. We should have support for, say, tuples of arrays of floats in 0.19 if everything works out correctly. This is, unfortunately, the wrong fix.