Skip to content

Resource tracker fails to track filenames with colons on Linux #98896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
maaleske opened this issue Oct 31, 2022 · 2 comments
Open

Resource tracker fails to track filenames with colons on Linux #98896

maaleske opened this issue Oct 31, 2022 · 2 comments
Labels
3.10 only security fixes 3.11 only security fixes 3.12 only security fixes topic-multiprocessing type-bug An unexpected behavior, bug, or error

Comments

@maaleske
Copy link

Bug report

multiprocessing.resource_tracker fails to track files with a colon (':') in the filename due to a parsing error, allowing resource leaks. I originally encountered this while working with filenames that include ISO8601 timestamps with colons as time separators.

Minimal example:

$ python -c "from multiprocessing.shared_memory import SharedMemory; shm = SharedMemory(create=True, size=1, name='a:b')"
$ Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/resource_tracker.py", line 199, in main
    cmd, name, rtype = line.strip().decode('ascii').split(':')
ValueError: too many values to unpack (expected 3)
$ ls /dev/shm/a\:b
/dev/shm/a:b

Without the colon, the file is correctly tracked and removed:

$ python -c "from multiprocessing.shared_memory import SharedMemory; shm = SharedMemory(create=True, size=1, name='ab')"
$ /usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
$ ls /dev/shm/ab
ls: cannot access '/dev/shm/ab': No such file or directory

Your environment

  • CPython versions tested on: 3.10.7, 3.11.0, 3.12.0a1
  • Operating system and architecture: Fedora 35 x86_64 (3.10.7 from Fedora, others with Docker images)
@maaleske maaleske added the type-bug An unexpected behavior, bug, or error label Oct 31, 2022
@kumaraditya303 kumaraditya303 added topic-multiprocessing 3.11 only security fixes 3.10 only security fixes 3.12 only security fixes labels Oct 31, 2022
@saito828koki
Copy link
Contributor

@maaleske
Thanks for your report.
In my opinion, making colons unusable in resource names is a reasonable fix for this issue. (We should also check if other parts of multiprocessing using resource_tracker like multiprocessing.synchronize.SemLock will cause the same problem. )

Are there better ways to fix this issue?

@maaleske
Copy link
Author

maaleske commented Nov 1, 2022

@saito828koki
That is poor solution, since the issue is still present if the resource is not created by the Python process, but just opened through SharedMemory (as was the case in my original encounter with it). Disallowing colons would make SharedMemory unusable in situations where Python is not in charge of the filenames, which I imagine to be a common use of shared memory resources.

I also investigated a bit more and colons are not the only problem; if the filenames contain newlines (again, valid on Linux) and colons, you can trigger other errors from the same code path, and I think there may even be a way to induce the resource tracker to unlink the wrong files by injecting an extra \ncmd:name:rtype\n in to the file through the filename, though I didn't yet find a working example.

Given that Python might not be the one responsible for the filenames, the real fix is to not parse them and store them in a way that does not require it. If you want to be rigorous, I think you would have to store each filename in a separate file, and manage the related cmd/rtype in some way.
However, there is also an issue related to incorrect behaviour of the resource tracker (#82300), with one fix proposing to remove it altogether for shared memory resources, which would make this a non-issue for SHM, but still probably relevant for semaphores.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.10 only security fixes 3.11 only security fixes 3.12 only security fixes topic-multiprocessing type-bug An unexpected behavior, bug, or error
Projects
Multiprocessing issues
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maaleske @saito828koki @kumaraditya303