Skip to content

RuntimeError on race on weakset concurrent iteration #89441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bensimner mannequin opened this issue Sep 24, 2021 · 2 comments
Closed

RuntimeError on race on weakset concurrent iteration #89441

bensimner mannequin opened this issue Sep 24, 2021 · 2 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@bensimner
Copy link
Mannequin

bensimner mannequin commented Sep 24, 2021

BPO 45278
Nosy @freddrake, @pitrou, @serhiy-storchaka, @graingert, @bensimner
Files
  • weakset_concurrent_iter_runtimeerror.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2021-09-24.11:03:08.880>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'RuntimeError on race on weakset concurrent iteration'
updated_at = <Date 2021-09-24.11:06:46.354>
user = 'https://github.com/bensimner'

    bugs.python.org fields:

    activity = <Date 2021-09-24.11:06:46.354>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-09-24.11:03:08.880>
creator = 'bjs'
dependencies = []
files = ['50302']
hgrepos = []
issue_num = 45278
keywords = []
message_count = 1.0
messages = ['402553']
nosy_count = 5.0
nosy_names = ['fdrake', 'pitrou', 'serhiy.storchaka', 'graingert', 'bjs']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45278'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
    @bensimner
    Copy link
    Mannequin Author

    bensimner mannequin commented Sep 24, 2021

    This is a very subtle race

    WeakSet uses _weakrefset.py's _IterationGuard structure to protect against the case where the elements the WeakSet refers to get cleaned up while a thread is iterating over the WeakSet.

    It defers the actual removal of any elements which get gc'd during iteration until the end of the iteration.

    the WeakSet keeps track of all the iterators, and waits until there are no more threads iterating over the set before it does the removal: https://github.com/python/cpython/blob/main/Lib/_weakrefset.py#L30

    However there is a race, if another thread begins iterating after the `if s:` check but before the _commit_removals call has ended, that iteration can get a RuntimeError.

    attached is an example script that can generate such RuntimeError's,
    although the race window here is very small and so to observe yourself you may have to tweak the magic constants around.

    As far as I'm aware nobody has reported seeing this bug happen in production, but some libraries (e.g. asyncio) do currently rely on concurrently iterating a weakset, so it's not implausible.

    @bensimner bensimner mannequin added type-bug An unexpected behavior, bug, or error labels Sep 24, 2021
    @serhiy-storchaka serhiy-storchaka added stdlib Python modules in the Lib dir 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes labels Sep 24, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @duaneg
    Copy link

    duaneg commented Apr 24, 2025

    This is a dup of #123089 (or vice versa). The fix for that has been merged for 3.14.0, and I believe this issue can also be closed.

    @kumaraditya303 kumaraditya303 closed this as not planned Won't fix, can't repro, duplicate, stale Apr 24, 2025
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    Weak reference issues
    Status: Done
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @duaneg @serhiy-storchaka @kumaraditya303