Travis log from the Cron job on my fork: https://travis-ci.org/lesteve/scikit-learn/builds/302533632.

Note there will still be at least one error remaining because of numpy/numpy#10026.

Travis log from the Cron job on my fork: https://travis-ci.org/lesteve/scikit-learn/builds/302533632.

Only failure is the expected one:

____________ [doctest] sklearn.metrics.pairwise.manhattan_distances ____________
498         If sum_over_features is False shape is
499         (n_samples_X * n_samples_Y, n_features) and D contains the
500         componentwise L1 pairwise-distances (ie. absolute difference),
501         else shape is (n_samples_X, n_samples_Y) and D contains
502         the pairwise L1 distances.
503 
504     Examples
505     --------
506     >>> from sklearn.metrics.pairwise import manhattan_distances
507     >>> manhattan_distances([[3]], [[3]])#doctest:+ELLIPSIS
Expected:
    array([[ 0.]])
Got:
    array([[0.]])

The sacrifice seems high, e.g., 0.849 -> 0.8, which makes me dare not to give the approval, though seems that we need to do so to pass the tests in numpy dev.

0.849 -> 0.8... is because in numpy dev you get 0.85 and in numpy 1.13 you get 0.849999999. There is no way to cover that in doctests (suggestions welcome!). astropy has a custom doctest runner to compare floats and there was some talk about implementing a more generic one in numpy but I don't think this has happened.

The killer argument for me is this: we do not and we should not rely on doctests to test scikit-learn thoroughly. The doctests are present in docstrings (and rst files) to kind of inform the user of the expected result and check that we do get the expected result.

This means that we should spend the minimal amount of time to get something working in both numpy 1.13 and numpy 1.14.

The best solution I see: when numpy 1.14 is released, we update all the docstrings to work with numpy 1.14 (without legacy mode printing of course) and we just skip the doctests in numpy <= 1.13 (for example using conftest.py).

We should probably wait to see how numpy/numpy#10029 unfolds before merging. If numpy decides that np.set_printoptions(legacy=True) should make numpy dev (aka 1.14) print/repr numpy scalars as in numpy 1.13, we do not need this PR at all.

Can use string formatting if it comes to that...

Yeah I thought about that. I would do that as last resort though: this makes the docstring (or the code in the rst file) a bit harder to follow.

Before fixing anything related to numpy's development version, we should contact the numpy developers: it has happened in the past that some projects have too early "fixed" their code for numpy-dev, when in fact a regression had been added to the master branch (see -- I believe -- @njsmith's BIDS presentation ).

Before fixing anything related to numpy's development version, we should contact the numpy developers

Agreed, that's what we generally do of course unless it is very obvious that's something that is completely our fault (ignoring DeprecationWarning until they create an error jumps to mind). In this particular case I opened numpy/numpy#10026 and numpy/numpy#10029.

Closing this one, because numpy has fixed the issues I opened. On a slightly related note I am investigating the failures with the latest Cron job. It feels like conftest.py is not taken into account (probably because we run the tests from a different folder than the checkout one) but I need to investigate a bit more to be 100% sure.