Skip to content

gh-74585: Fix race condition in shutil.copyfile #136836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

abadger
Copy link
Contributor

@abadger abadger commented Jul 19, 2025

This PR fixes a race condition in shutil.copyfile (there are several other shutil functions that should also be fixed. The current state is that only copyfile is fixed.) Unittests are passing but there are a few things that I'd like to look into:

Reading through the issue, the following questions were asked and should be looked into:

  • giampaolo: I wonder whether this can introduce any change in semantic though:
    >>> import os
    >>> os.stat('foobar')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    FileNotFoundError: [Errno 2] No such file or directory: 'foobar'
    >>> os.stat(333)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: [Errno 9] Bad file descriptor: 333
  • vstinner: Python doesn't use FD of directories by default because it introduces issues of FD limit with deep directory tree, issue of managing FD lifetime, etc. That's why some API have one flavor for path and another for FD.
  • Look through the copyfile changes for operations that are duplicated/should be done inside of the openers. For instance, this PR presently stats a file twice. I think that's unnecessary.
  • Look into whether there's other ways to pass the information about whether the file was created or not than via a closured variable.
  • fix other shutil functions: giampaulo: All copy* functions and move() are subjects to race conditions (the only exception is rmtree()).
  • Vstinner proposed that fixing copyfile is good in and of itself, regardless of whether we fix the rest of the shutil functions. I'll take a look at how similar they are before commenting on this.

This is a forward-port of #1659 from @pkmoore

@abadger
Copy link
Contributor Author

abadger commented Jul 20, 2025

For the WASI CI failures, it seems that WASI is setting os.name == "posix" but doesn't have fcntl. We could check whether the file is a socket using a stat call on the filename and only if it is not a socket, proceeding to the code which opened the file, which could be opened in regular blocking mode since we don't have to worry about reading a socket anymore (unless the system experiences a race in that specific scenario. I believe it would lead to the copyfile code hanging until data is sent on the socket.) We would need to stat for a second time to be sure the stat and operations for the rest of the operations were the same. I believe this could result in WASI having a potential DOS that the other platforms do not have but it would not be as large a hole as we currently have (where the race could cause files to be overwritten or private data to be read.)

@abadger abadger force-pushed the fix-race-in-shutil-74585 branch from 1500bc7 to d361a00 Compare July 20, 2025 14:53
@abadger abadger force-pushed the fix-race-in-shutil-74585 branch from d361a00 to 970fce1 Compare July 20, 2025 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant