Skip to content

[multiprocessing] Multiprocessing in spawn mode doesn't work when the target is a method in a unittest.TestCase subclass, when run either with unittest or with pytest #78065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
YoniRozenshein mannequin opened this issue Jun 17, 2018 · 3 comments
Labels
3.7 (EOL) end of life 3.8 (EOL) end of life stdlib Python modules in the Lib dir topic-multiprocessing type-bug An unexpected behavior, bug, or error

Comments

@YoniRozenshein
Copy link
Mannequin

YoniRozenshein mannequin commented Jun 17, 2018

BPO 33884
Nosy @pitrou, @applio
Files
  • mp_pickle_issues.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2018-06-17.10:45:32.688>
    labels = ['3.7', '3.8', 'type-bug', 'library', 'invalid']
    title = "[multiprocessing] Multiprocessing in spawn mode doesn't work when the target is a method in a unittest.TestCase subclass, when run either with unittest or with pytest"
    updated_at = <Date 2018-06-17.18:51:59.760>
    user = 'https://bugs.python.org/YoniRozenshein'

    bugs.python.org fields:

    activity = <Date 2018-06-17.18:51:59.760>
    actor = 'pitrou'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2018-06-17.10:45:32.688>
    creator = 'Yoni Rozenshein'
    dependencies = []
    files = ['47645']
    hgrepos = []
    issue_num = 33884
    keywords = []
    message_count = 2.0
    messages = ['319811', '319826']
    nosy_count = 3.0
    nosy_names = ['pitrou', 'davin', 'Yoni Rozenshein']
    pr_nums = []
    priority = 'normal'
    resolution = 'not a bug'
    stage = 'resolved'
    status = 'pending'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue33884'
    versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    @YoniRozenshein
    Copy link
    Mannequin Author

    YoniRozenshein mannequin commented Jun 17, 2018

    multiprocessing will attempt to pickle things using ForkingPickler when starting a new process in spawn mode (in Windows this is the only mode, in Linux this is a non-default but settable mode).

    When run within the context of a unit test, if it has to pickle a TestCase subclass, it encounters objects that can't be pickled. The attached file shows a minimum working example (uncomment the two commented lines under __main__ to run with pytest).

    When run with unittest.main(), it raises:

    TypeError: cannot serialize '_io.TextIOWrapper' object

    When run with pytest.main(), it raises:

    AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'

    Note that changing the _child_process in my example to a classmethod/staticmethod or moving it to a top-level function outside the class works around this issue (both with unittest and with pytest).

    @YoniRozenshein YoniRozenshein mannequin added 3.7 (EOL) end of life 3.8 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jun 17, 2018
    @pitrou
    Copy link
    Member

    pitrou commented Jun 17, 2018

    While this is limitation of the spawn and forkserver modes, I don't see how this is a bug. Also, as you point out, making the method a classmethod or staticmethod works around the limitation.

    @gpshead
    Copy link
    Member

    gpshead commented Dec 13, 2022

    not a bug. many things will intentionally not be serializable.

    @gpshead gpshead closed this as not planned Won't fix, can't repro, duplicate, stale Dec 13, 2022
    @gpshead gpshead moved this to Done in Unittest issues Dec 13, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 (EOL) end of life stdlib Python modules in the Lib dir topic-multiprocessing type-bug An unexpected behavior, bug, or error
    Projects
    Status: Done
    Development

    No branches or pull requests

    4 participants