SharedMemory constructor raises "cannot mmap an empty file" exception #92408

OvervCW · 2022-05-06T16:08:26Z

Bug report

I was trying to clean up a shared memory object left behind by an earlier process using the following code:

shm = SharedMemory("some_name")
shm.close()
shm.unlink()

However, the first line resulted in an exception being raised:

  File "/home/USER/.pyenv/versions/3.9.10/lib/python3.9/multiprocessing/shared_memory.py", line 114, in __init__
    self._mmap = mmap.mmap(self._fd, size)
ValueError: cannot mmap an empty file

The exception handler around that line unlinks the object in case of an OSError, but not in case of this ValueError raised by mmap.mmap:

cpython/Lib/multiprocessing/shared_memory.py

Lines 109 to 117 in 973a520

    
           try: 
        
               if create and size: 
        
                   os.ftruncate(self._fd, size) 
        
               stats = os.fstat(self._fd) 
        
               size = stats.st_size 
        
               self._mmap = mmap.mmap(self._fd, size) 
        
           except OSError: 
        
               self.unlink() 
        
               raise

This makes it effectively impossible to clean up this particular shared memory object through the standard library.

I'm not sure how the shared memory object was corrupted in the first place, but it looks like the exception is triggered because os.fstat states that it has size 0:

>>> f = _posixshmem.shm_open('some_name', os.O_RDWR, mode=0o600)
>>> os.fstat(f)
os.stat_result(st_mode=33152, st_ino=80, st_dev=27, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1651836036, st_mtime=1651836036, st_ctime=1651836036)

After which the code tries to mmap with size 0 and fails with the exception mentioned earlier.

I was only able to resolve this by calling _posixshmem.shm_unlink('some_name') manually. I think the exception handler should be extended to also unlink the file if it was truncated to 0 like this.

This scenario can be reproduced with the following code:

import _posixshmem
import os
from multiprocessing.shared_memory import SharedMemory

f = _posixshmem.shm_open('test', os.O_RDWR | os.O_CREAT | os.O_EXCL, 0o600)
os.close(f)

try:
    mem = SharedMemory("test")
finally:
    _posixshmem.shm_unlink("test")

Your environment

CPython versions tested on: 3.9.10
Operating system and architecture: Ubuntu 20.04.3 LTS

Linked PRs

gh-92408/gh-96026: SharedMemory changes #133227

The text was updated successfully, but these errors were encountered:

duaneg · 2025-04-27T00:43:36Z

Hmm. There is a quite a lot going on here.

The SharedMemory class catching OSError and missing the ValueError looks like an oversight. ISTM we should run the cleanup regardless of error. Switching to a bare except fixes that.

Which then reveals another problem: resource_tracker.unregister is being called despite register not having been called. This causes an exception in the resource tracker. We can fix that by just directly calling _posixshmem.shm_unlink instead of calling unlink.

However, that prompts a more fundamental question: why are we trying to unlink a pre-existing shared memory object we did not create? I think this is a bug: the object should only be unlinked if it was created in the constructor which subsequently raises an exception.

Fixing that leaves us back with the problem noted above: "This makes it effectively impossible to clean up this particular shared memory object through the standard library." ISTM the key problem is that we cannot use SharedMemory with zero-length shared memory objects, despite these being perfectly valid in POSIX (and Windows?). We can't mmap an empty file, but we can just skip that and assign a zero-length memoryview to buf.

Does this all sound right?

I have a set of changes to address all the above, but a) would like some feedback from others (especially the active MP maintainers @applio, @pitrou, and @gpshead) as to whether they agree and that it is worth doing, b) need to do some testing on Windows.

duaneg · 2025-04-28T08:39:25Z

See also #96026, which seems to be specifically about the second problem noted above: resources being mistakenly unregistered during error handling in the ctor.

`multiprocessing.shared_memory.SharedMemory` currently tries to unlink a POSIX shared memory object if it fails to `mmap` it after opening/creating it. However, since it only catches `OSError`, it fails to do so when opening an existing zero-length shared memory object, as `mmap.mmap` raises a `ValueError` when given a 0 size. Fix this by catching all exceptions with a bare `except`, so the cleanup is run for all errors.

OvervCW added the type-bug An unexpected behavior, bug, or error label May 6, 2022

iritkatriel added stdlib Python modules in the Lib dir topic-multiprocessing labels Nov 24, 2023

bedevere-app bot mentioned this issue Apr 30, 2025

gh-92408/gh-96026: SharedMemory changes #133227

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SharedMemory constructor raises "cannot mmap an empty file" exception #92408

SharedMemory constructor raises "cannot mmap an empty file" exception #92408

OvervCW commented May 6, 2022 •

edited by bedevere-app bot

Loading

duaneg commented Apr 27, 2025

duaneg commented Apr 28, 2025

SharedMemory constructor raises "cannot mmap an empty file" exception #92408

SharedMemory constructor raises "cannot mmap an empty file" exception #92408

Comments

OvervCW commented May 6, 2022 • edited by bedevere-app bot Loading

Linked PRs

duaneg commented Apr 27, 2025

duaneg commented Apr 28, 2025

OvervCW commented May 6, 2022 •

edited by bedevere-app bot

Loading