Skip to content

gh-121577: fix compileall always recompiling pyc files with hash based invalidation #121960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

gh-121577: fix compileall always recompiling pyc files with hash based invalidation #121960

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
35b9311
compileall now verifies whether recompilation is necessary for all in…
kulikjak Jul 18, 2024
9d47dbc
add NEWS entry
kulikjak Jul 18, 2024
f14eae5
fix NEWS entry syntax
kulikjak Jul 18, 2024
9e3b22d
fix the verification logic
kulikjak Jul 18, 2024
64d7f35
fix tests
kulikjak Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions Lib/compileall.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,12 +227,24 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
if tail == '.py':
if not force:
try:
mtime = int(os.stat(fullname).st_mtime)
expect = struct.pack('<4sLL', importlib.util.MAGIC_NUMBER,
0, mtime & 0xFFFF_FFFF)
for cfile in opt_cfiles.values():
with open(cfile, 'rb') as chandle:
actual = chandle.read(12)
header = chandle.read(16)

if header[4]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can check the magic number without reading the source file.

Also, header[4] can raise IndexError. And you need to check other bytes and bits.

Suggested change
if header[4]:
if len(header) < 16 or header[:4] != importlib.util.MAGIC_NUMBER or struct.unpack('<L', header[4:8])[0] & ~0b11:
break
if header[4] & 0b1:

# hash-based invalidation
with open(fullname, 'rb') as chandle:
source_bytes = chandle.read()
source_hash = importlib.util.source_hash(source_bytes)
actual = header
expect = struct.pack('<4sL8s', importlib.util.MAGIC_NUMBER,
header[4], source_hash)
Comment on lines +239 to +241
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can simply check the source hash:

Suggested change
actual = header
expect = struct.pack('<4sL8s', importlib.util.MAGIC_NUMBER,
header[4], source_hash)
if header[8:] != source_hash:
break

else:
# timestamp-based invalidation
mtime = int(os.stat(fullname).st_mtime)
actual = header[:12]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not check also the file size?

expect = struct.pack('<4sLL', importlib.util.MAGIC_NUMBER,
0, mtime & 0xFFFF_FFFF)
if expect != actual:
break
else:
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/test_compileall.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -845,7 +845,7 @@ def test_pyc_invalidation_mode(self):
with open(pyc, 'rb') as fp:
data = fp.read()
self.assertEqual(int.from_bytes(data[4:8], 'little'), 0b11)
self.assertRunOK('--invalidation-mode=unchecked-hash', self.pkgdir)
self.assertRunOK('--invalidation-mode=unchecked-hash', '-f', self.pkgdir)
with open(pyc, 'rb') as fp:
data = fp.read()
self.assertEqual(int.from_bytes(data[4:8], 'little'), 0b01)
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2024-07-18-13-48-18.gh-issue-121577.CAHKTY.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
:mod:`compileall` now verifies whether recompilation is necessary with all
``.pyc`` invalidation modes. Patch by Jakub Kulik.
Loading