Skip to content

bpo-43153: Don't mask PermissionError with NotADirectoryError during tempdirectory cleanup #29940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Dec 5, 2023
Merged

bpo-43153: Don't mask PermissionError with NotADirectoryError during tempdirectory cleanup #29940

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
5111fa7
Don't mask PermissionError with IsADirectoryError during tempdirector…
Fidget-Spinner Dec 6, 2021
b3da7cd
fix Window spelling
Fidget-Spinner Dec 6, 2021
0673393
improve comment
Fidget-Spinner Dec 6, 2021
98725c8
fix comments IsADirectoryError -> NotADirectoryError
Fidget-Spinner Dec 6, 2021
283df0a
use os.path.isdir instead
Fidget-Spinner Dec 6, 2021
b9f074f
add issue number to comments
Fidget-Spinner Dec 7, 2021
b658bfc
Remove windows check
Fidget-Spinner Dec 7, 2021
11a31ac
Skip test on other OSes except Windows
Fidget-Spinner Dec 7, 2021
499e424
Merge remote-tracking branch 'upstream/main' into bpo-43153
Fidget-Spinner Dec 7, 2021
bfd39ea
Address Serhiy's code review
Fidget-Spinner Dec 8, 2021
45d5dea
Merge branch 'main' into bpo-43153
serhiy-storchaka Dec 5, 2023
77f5e8a
Test for junctions on Windows.
serhiy-storchaka Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions Lib/tempfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
import io as _io
import os as _os
import shutil as _shutil
import stat as _stat
import errno as _errno
from random import Random as _Random
import sys as _sys
Expand Down Expand Up @@ -889,8 +890,31 @@ def resetperms(path):

try:
_os.unlink(path)
# PermissionError is raised on FreeBSD for directories
except (IsADirectoryError, PermissionError):
except IsADirectoryError:
cls._rmtree(path, ignore_errors=ignore_errors)
except PermissionError:
# The PermissionError handler was originally added for
# FreeBSD in directories, but it seems that it is raised
# on Windows too.
# bpo-43153: Calling _rmtree again may
# raise NotADirectoryError and mask the PermissionError.
# So we must re-raise the current PermissionError if
# path is not a directory.
try:
st = _os.lstat(path)
except OSError:
if ignore_errors:
return
raise
if (_stat.S_ISLNK(st.st_mode) or
not _stat.S_ISDIR(st.st_mode) or
(hasattr(st, 'st_file_attributes') and
st.st_file_attributes & _stat.FILE_ATTRIBUTE_REPARSE_POINT and
st.st_reparse_tag == _stat.IO_REPARSE_TAG_MOUNT_POINT)
):
if ignore_errors:
return
raise
cls._rmtree(path, ignore_errors=ignore_errors)
Comment on lines 891 to 918
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not go with an "ask-for-forgiveness-rather-than-permission" type of solution, e.g.:

try:
    _os.unlink(path)
except IsADirectoryError:
    cls._rmtree(path, ignore_errors=ignore_errors)
except PermissionError as pe:
    # PermissionError is raised on FreeBSD for directories
    # and by Windows on lock files used by other processes
    try:
        cls._rmtree(path, ignore_errors=ignore_errors)
    except NotADirectoryError:
        # NOTE: This is raised if PermissionError did not
        # correspond to a IsADirectoryError, e.g. on
        # Windows.
        if not ignore_errors:
            raise pe

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rmtree() is a complex function, and we cannot be sure whether NotADirectoryError was raised by the toplevel scandir() or somewhere deeper in the tree.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmmh...

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a responsibility inversion to me. Sure there could be a bug deep down in _rmtree that ends up bubbling an unrelated NotADirectoryError, but that would technically be a bug in _rmtree. Maybe there are valid cases where such an unrelated error would be raised?

Also I see the diff I commented on is not the final version so I'll look at the final version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessary a bug. It may be a race condition. Although the LBYL approach is also prone to race conditions, I think it has less chance to override error with a wrong exception.

I myself prefer the EAFP approach, but I do not think that it has advantages in this case.

except FileNotFoundError:
pass
Expand Down
11 changes: 11 additions & 0 deletions Lib/test/test_tempfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1641,6 +1641,17 @@ def test_explicit_cleanup_ignore_errors(self):
temp_path.exists(),
f"TemporaryDirectory {temp_path!s} exists after cleanup")

@unittest.skipUnless(os.name == "nt", "Only on Windows.")
def test_explicit_cleanup_correct_error(self):
with tempfile.TemporaryDirectory() as working_dir:
temp_dir = self.do_create(dir=working_dir)
with open(os.path.join(temp_dir.name, "example.txt"), 'wb'):
# Previously raised NotADirectoryError on some OSes
# (e.g. Windows). See bpo-43153.
with self.assertRaises(PermissionError):
temp_dir.cleanup()


@os_helper.skip_unless_symlink
def test_cleanup_with_symlink_to_a_directory(self):
# cleanup() should not follow symlinks to directories (issue #12464)
Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Library/2021-12-06-22-10-53.bpo-43153.J7mjSy.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
On Windows, ``tempfile.TemporaryDirectory`` previously masked a
``PermissionError`` with ``NotADirectoryError`` during directory cleanup. It
now correctly raises ``PermissionError`` if errors are not ignored. Patch by
Andrei Kulakov and Ken Jin.