Thank you for reporting! That's not great :-/

Could you also install pytest-timeout in your env and add the option -s --timeout=10 to your call to pytest. This way we will have more info on why this is stucked.

It is weird that the test passes on appveyor if you see a deterministic failure on your machine.

Here is what I see:

>pytest -s --timeout=10 -v --pyarg sklearn.linear_model.tests.test_theil_sen::test_theil_sen_parallel

Exception in thread QueueManagerThread:

  File "<<elided>>\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 617, in _queue_management_worker
    ready = wait(readers + worker_sentinels)
  File "<<elided>>\lib\multiprocessing\connection.py", line 859, in wait
  File "<<elided>>\lib\multiprocessing\connection.py", line 791, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 129

<< lots of warnings suppressed >>

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~~~ Stack of QueueFeederThread (22640) ~~~~~~~~~~~~~~~~~~~~~~

  File "<<elided>>\lib\site-packages\sklearn\externals\joblib\externals\loky\backend\queues.py", line 138, in _feed
    nwait()
  File "<<elided>>\lib\threading.py", line 295, in wait
    waiter.acquire()

~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (24148) ~~~~~~~~~~~~~~~~~~~~~~~~~~

    testfunction(**testargs)
  File "<elided>\lib\site-packages\sklearn\linear_model\tests\test_theil_sen.py", line 267, in test_theil_sen_parallel
    max_subpopulation=2e3).fit(X, y)
  File "<elided>\site-packages\sklearn\linear_model\theil_sen.py", line 390, in fit
    for job in range(n_jobs))
  File "<<elided>>\lib\site-packages\sklearn\externals\joblib\parallel.py", line 996, in __call__
    self.retrieve()
  File "<<elided>>\lib\site-packages\sklearn\externals\joblib\parallel.py", line 899, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "<<elided>>\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 517, in wrap_future_result
    return future.result(timeout=timeout)
  File "<<elided>>\lib\concurrent\futures\_base.py", line 427, in result
    self._condition.wait(timeout)
  File "<<elided>>\lib\threading.py", line 295, in wait
    waiter.acquire()

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

It looks like you are missing a part of the trace in QueueManagerThread.
Can you confirm that we can see all the trace or update it? (the first line should show the _queue_management_worker function first.)

@tomMoral Yes, my bad. I have updated the earlier comment to include the said line.

How many workers do you have here? Are you on a server with more than 127 cores?

The machine has Intel Xeon E5-2630 v3, with 2 sockets, 8 cores per socket, HT: 2. The task manager reports 16 cores, and 32 logical cores.

When the test run hangs I do indeed see over 100 python processes in the task manager.

@oleksandr-pavlyk could you clarify whether this is resolved in master after c81e255?

I am starting a rackspace VM under windows to give it a try.

I have tried on a big Dual Intel Xeon E5-2660 v3 running Windows 2012 and scikit-learn 0.20.0 installed from pip in a dedicated conda env exactly as reported by @oleksandr-pavlyk and I could not reproduce the hangs. I repeated the pytest commands more than 20 times and each completed successfully in less than 2s. And I could see in the task manager than many cores were used. I also launched the full test suite successfully.

I tried also to build scikit-learn master and launch the full test suite twice and it completed without any hang. There are failing doctests under windows but this is expected (the way the default dtype of integer arrays is printed in doctest output is not necessarily always consistent with the output under other platforms but this is harmless).

@ogrisel @jnothman Please allow me a moment to build it and run the test. I would report in about one hour (expected build + test run time on our server)

I can still see the hang with a local build from current master (4e81949).

Upon execution of

pytest -v --pyarg sklearn.tests.test_multioutput::test_multi_output_classification_partial_fit_parallelism

There are 130 idle Python processes shown in task manager and the execution never terminates.

Pressing Ctrl+C terminates all 130 processes, which is an improvement over an earlier version of joblib.

Is there a way for me to access prebuilt wheels, or other binaries that you used to check whether the problem is fixed?

Thanks!

If you built the master branch then the problem is still there. This is weird because I could not reproduce the issue, neither on 0.20.0 nor on master.

Is there a way for me to access prebuilt wheels, or other binaries that you used to check whether the problem is fixed?

Our CI uploads the wheels built on the master branch for 64 bit Python 3.7 and 32 bit Python 2.7 on the following rackspace blob storage:
http://bc1c050a9b3a9f5bf3f5-965058070168eb2f1b8f5a23def30e14.r96.cf2.rackcdn.com/

But they should yield the same result as the one you observed by building the master branch yourself.

Thanks @ogrisel . I downloaded and installed scikit_learn-0.21.dev0+20181114100605-cp37-cp37m-win_amd64.whl and can still reproduce the hang.

retagging see #12548 (comment)

Could you please run the following on your machine :

from sklearn.externals.joblib import cpu_count
print(cpu_count())

I will try investigating that in the coming days.

@tomMoral , here it is:

(skl-0.21.dev0) tmp>ipython
Python 3.7.1 (default, Oct 28 2018, 08:39:03) [MSC v.1912 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from sklearn.externals.joblib import cpu_count
   ...: print(cpu_count())
   ...:
128

Ok so the problem is caused by loky implementation of wait, which relies on winapi.WaitForMultipleObjects which cannot wait for more than MAXIMUM_WAIT_OBJECTS=64 objects.

I opened an issue in the loky tracker (tomMoral/loky#192).

But it is still strange that you report 32 cores in your machine but get cpu_count() == 128. There seems to have another issue here. Could you please report the results of the following commands?

echo %NUMBER_OF_PROCESSORS%
python -c "import multiprocessing as mp; print('mp:', mp.cpu_count())"

Multiprocessign also reports 128:

(skl-0.21.dev0) tmp>echo %NUMBER_OF_PROCESSORS%
16

(skl-0.21.dev0) tmp>python -c "import multiprocessing as mp; print('mp:', mp.cpu_count())"
mp: 128

So there might be a bug in the way python is introspecting the system to detect the number of processors on this machine.

Actually on python 3, what matters is os.cpu_count() for loky / joblib. I don't know if it relies on the same underlying implementation as multiprocessing.

Yes, os.cpu_count() also returns 128.

In [2]: import os

In [3]: os.cpu_count()
Out[3]: 128

This might be this bug:

https://bugs.python.org/issue30581

It should be fixed in Python 3.7.1.

Perhaps @anton-malakhov might have an idea for why os.cpu_count() is returning this value.

My wish, though, is that this does not cause a hang.

Here is the fix for bpo-30581:

https://github.com/python/cpython/pull/2934/files

The hang itself is caused by https://github.com/tomMoral/loky/issues/192 but it would not be caused if python did not report a wrong number of CPUs.

@ogrisel By the way, os.cpu_count() returns 128 in Python 3.7.1

(skl-0.21.dev0) tmp>ipython
Python 3.7.1 (default, Oct 28 2018, 08:39:03) [MSC v.1912 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import os

In [2]: os.cpu_count()
Out[2]: 128

@ogrisel but more cores would still trigger the bug, even with python fixed, right? EC2 has c5.18xlarge with 72 cores for $0.70/h.

but more cores would still trigger the bug, even with python fixed, right? EC2 has c5.18xlarge with 72 cores for $0.70/h.

Note that this problem only exists on windows, so big instances are more around 5/10$ per hour.

But this should still be fixed in loky.

@tomMoral fair point ;)

I volunteer to test fixes for free :)

but more cores would still trigger the bug, even with python fixed, right? EC2 has c5.18xlarge with 72 cores for $0.70/h.

On a version of Python with the fix for cpu_count, this bug in loky can only be triggered with 128 cores or more. It's hard to find a machine like this in 2018.

By that time loky will be fixed hopefully :)

Actually no, you are right, 64 cores is enough. I misread the above conversation.

Using a non-loky joblib.parallel backend would be a sufficient workaround here?

We got bit by another one of these parallel tests, namely sklearn/decomposition/tests/test_dict_learning.py::test_dict_learning_reconstruction_parallel.

At the same time, the test sklearn/decomposition/tests/test_dict_learning.py::test_sparse_coder_parallel_mmap runs fine.

The difference between them is that the hanging one sets n_jobs=-1, while the working one is explicit n_jobs=2.

If you think it is a good idea to be explicit about the number of jobs in test suite (say n_job 4 or less), I would open a PR. Please let me know.

It seems this issue is fixed? Closing, please feel free to reopen if it's still an issue.