Skip to content

gh-59999: Add option to preserve permissions in ZipFile.extract #32289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 40 commits into
base: main
Choose a base branch
from

Conversation

dignissimus
Copy link
Contributor

@dignissimus dignissimus commented Apr 3, 2022

Co-authored by Alexey Boriskin

https://bugs.python.org/issue15795

TODO

  • Documentation
    • Document constants
    • Document permission preservation parameters
    • Document new behaviour for archives created on windows
  • Testing
    • Get current tests to pass
      • Work out why extracted files have mode 777
    • Add more tests for the other stat flags
      • Explicitly state file type by applying file-type masks
      • Re-add tests for files with mode 0
    • Check the permissions of extracted files against the umask
    • Extract test archive contents to a test directory
  • Enumerations
    • Replace constants with enumerations
    • Document enumerations

@dignissimus dignissimus changed the title bpo-15795: Add option to preserve permissions in ZipFile.extract gh-59999: Add option to preserve permissions in ZipFile.extract Apr 13, 2022
@dignissimus dignissimus marked this pull request as ready for review April 13, 2022 10:26
@dignissimus dignissimus requested a review from merwok April 13, 2022 10:26
@sharkwouter
Copy link

Thanks for making this. I didn't really get back to my PR after it look a year to be reviewed.

What I did differently is that I made preserving the permissions the default. The reason for this is that the unzip command which comes with any Linux distribution, MacOS or BSD system will preserve permissions by default as well. So the user will probably expect that to be the default.

@dignissimus
Copy link
Contributor Author

Thanks for making this. I didn't really get back to my PR after it look a year to be reviewed.

It's no problem

What I did differently is that I made preserving the permissions the default. The reason for this is that the unzip command which comes with any Linux distribution, MacOS or BSD system will preserve permissions by default as well. So the user will probably expect that to be the default.

On the BPO issue (#59999) I proposed not doing this to maintain backwards compatibility

@sharkwouter
Copy link

Makes some sense, thanks for clarifying. Keep up the good work!

@MaxwellDupre
Copy link
Contributor

Running test_zipfile I get two errors:
FAIL: test_extract_preserve_none (test.test_zipfile.TestsPermissionExtraction.test_extract_preserve_none)

Traceback (most recent call last):
File "/home/dougal/Documents/GitHub/cpython/Lib/test/test_zipfile.py", line 2099, in test_extract_preserve_none
self.assertEqual(os.stat(filename).st_mode,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 33204 != 33188

FAIL: test_extractall_preserve_none (test.test_zipfile.TestsPermissionExtraction.test_extractall_preserve_none)
Traceback (most recent call last):
File "/home/dougal/Documents/GitHub/cpython/Lib/test/test_zipfile.py", line 2090, in test_extractall_preserve_none
self.assertEqual(os.stat(filename).st_mode,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 33204 != 33188

Ran 274 tests in 94.070s
FAILED (failures=2, skipped=3)

@dignissimus
Copy link
Contributor Author

Ok, interesting. What OS and file system are you running this on? And, do the other test_extract tests pass, or were some of them skipped? And, do tests still fail on 6470201?

@MaxwellDupre

@MaxwellDupre
Copy link
Contributor

I am running Fedora 35. I did a clean start (compiling python from scratch just in case) and ran the test again and got the same result.
Ran 274 tests in 78.664s
FAILED (failures=2, skipped=3)

Other test_extract tests are ok ( except
test_extract_hackers_arcnames_windows_only (test.test_zipfile.ExtractTests.test_extract_hackers_arcnames_windows_only) ... skipped 'Requires \ as path separator.'
)

I don't know what you mean by 6470201, I didn't see any tests there.

@dignissimus
Copy link
Contributor Author

I don't know what you mean by 6470201, I didn't see any tests there.

Oh of course, they didn't exist in that revision.

I've just checked and the default mode for created files differs on Fedora. On CI and my system it's 644 on Fedora it seems to be 664. I'll write the test so it checks against what the default for the system is instead of the 0o644 constant.

@dignissimus dignissimus requested a review from merwok February 15, 2025 01:38
@dignissimus dignissimus requested a review from merwok February 17, 2025 09:53
@merwok merwok requested review from ethanfurman and removed request for merwok February 17, 2025 14:02
Copy link
Member

@jaraco jaraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. I just have a few suggestions, mostly nitpicks.

Comment on lines 1883 to 1892
# Ignore permissions if the archive was created on Windows
if member.create_system == 0 or preserve_permissions == PreserveMode.NONE:
return targetpath

if preserve_permissions == PreserveMode.SAFE:
mode = (member.external_attr >> 16) & 0o777
elif preserve_permissions == PreserveMode.ALL:
mode = (member.external_attr >> 16) & 0xFFFF

os.chmod(targetpath, mode)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the diff today, it's easy to see this code is related to a single purpose. After merging, however, it becomes an inline part of a larger narrative.

For future readers, it might be clearer to have this behavior extracted as its own method.

e.g.

    def _apply_permissions(self, member, path, mode):
        if mode == PreserveMode.NONE:
            return path

        # Ignore permissions if the archive was created on Windows
        if member.create_system == 0:
            return path

        mask = {
            PreserveMode.SAFE: 0o777,
            PreserveMode.ALL: 0xFFFF,
        }
        new_mode = (member.external_attr >> 16) & mask[mode]
        os.chmod(path, new_mode)

        return path

Then in this method, return self._apply_permissions(member, targetpath, preserve_permissions).

I also took the liberty to address a couple of nitpicks in the approach, such as to align the comment with the check, and to only perform the new mode calculation in a single place. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely agree, although I changed _apply_permissions to not return anything. How do you feel about that?

with self.open(member, pwd=pwd) as source, \
open(targetpath, "wb") as target:
shutil.copyfileobj(source, target)
self._apply_permissions(member, targetpath, preserve_permissions)
return targetpath

def _apply_permissions(self, member, path, mode):
"""
Apply ZipFile permissions to a file on the filesystem with
specified PreserveMode
"""
if mode == PreserveMode.NONE:
return
# Ignore permissions if the archive was created on Windows
if member.create_system == 0:
return
mask = {
PreserveMode.SAFE: 0o777,
PreserveMode.ALL: 0xFFFF,
}
new_mode = (member.external_attr >> 16) & mask[mode]
os.chmod(path, new_mode)

@merwok
Copy link
Member

merwok commented Mar 5, 2025

(Please follow the devguide: avoid force pushes, they create notifications with broken links for reviewers, and make it harder to see changes compared to previous time. Thanks!)

@sigma67
Copy link

sigma67 commented Apr 11, 2025

Seems another look is needed from a reviewer?

@merwok merwok requested a review from jaraco April 11, 2025 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.