Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Our records indicate we have not received your CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

Hi @pitrou (or anyone with a say), can you give us a hint about the fate of this PR (even if you honestly think it does not have a very promising future).

Thanks

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

I have made the requested changes; please review again

Thanks for making the requested changes!

@pitrou: please review the changes made to this pull request.

pinging @pitrou, at least to know if the changes pointed at the right direction.

Sorry, will take a look again. Also @pablogsal you may be interested in this.

bumping up!

Are there any plans for deprecating multiprocessing? Otherwise, I think this bug should be addressed.

If the proposed fix is not the right way of fixing it, please let me know. I'll resolve the conflicts only once I know there is interest in doing so.

Thanks very much

@pierreglaser @tomMoral Would you like to take a look at this?

Yes I can have a look.

I'll have a look too.

@pitrou thanks for the prompt response!

There are multiple tests being added that make use of sleep to synchronize processes (in particular it assumes that the processes will be entered in time when sleep finishes). This is very unreliable and it will most certainly fail on the slowest buildbots.

Please, try to add some synchronization to the tests to make them more deterministic.

Note that this PR, while improving the current state of the multiprocessing.Pool, is not a full solution as it is still very easy to deadlock by calling sys.exit(0) in the function.

import sys
from multiprocessing import Pool

pool = Pool(2)
pool.apply(sys.exit, (0,))

or at unpickling time

import sys
from multiprocessing import Pool


class Failure:
    def __reduce__(self):
        return sys.exit, (0, )


pool = Pool(2)
pool.apply(id, (Failure(),))

Also, many other problems exists with multiprocessing.Pool as you can easily deadlock it (choose from failure to serialize/deserialize, flooding the queue with many tasks and one failure, segfaulting with bad timing). I did some work to try to make it fault tolerant (see class _ReusablePool in this branch ) but some design in the communication process make it tricky to fix all possible deadlocks/interpreter freeze.

Maybe a more stable solution would be to actually change the Pool to rely on a concurrent.futures executor for the parallel computations (which is now far more stable IMO), just keeping the API to reduce the maintenance burden to only one implementation of the parallel pool of workers.

This PR is stale because it has been open for 30 days with no activity.