Skip to content

compileall: -s surprisingly makes path relative #133503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mgorny opened this issue May 6, 2025 · 1 comment · Fixed by #134756
Closed

compileall: -s surprisingly makes path relative #133503

mgorny opened this issue May 6, 2025 · 1 comment · Fixed by #134756
Labels
docs Documentation in the Doc dir

Comments

@mgorny
Copy link
Contributor

mgorny commented May 6, 2025 •
edited by bedevere-app bot
Loading

Documentation

(Filing as a documentation issue, since I suspect the behavior is intentional and has its valid use cases.)

In Gentoo, we install packages into a temporary directory whose contents resemble the actual filesystem prefix. For some packages that don't use PEP517 build systems, we invoke python -m compileall to byte-compile the installed modules.

Today I wanted to switch our code to start using the -s option, and I've noticed that in addition to the path specified, it also strips the path separator following it.

For example, consider the following example:

$ mkdir -p /tmp/destdir/usr/lib/python3.13/site-packages
$ > /tmp/destdir/usr/lib/python3.13/site-packages/foo.py
$ python3.13 -m compileall -s /tmp/destdir /tmp/destdir/usr/lib/python3.13/site-packages/foo.py
Compiling '/tmp/destdir/usr/lib/python3.13/site-packages/foo.py'...
$ strings /tmp/destdir/usr/lib/python3.13/site-packages/__pycache__/foo.cpython-313.pyc 
'usr/lib/python3.13/site-packages/foo.py
<module>r

Note that while I expeced it to strip /tmp/destdir, leaving /usr/lib/python3.13/site-packages/foo.py, it also stripped the following slash, making the path relative.

Now, I can achieve the desired behavior by adding -p /:

$ rm /tmp/destdir/usr/lib/python3.13/site-packages/__pycache__/foo.cpython-313.pyc
$ python3.13 -m compileall -s /tmp/destdir -p / /tmp/destdir/usr/lib/python3.13/site-packages/foo.py
Compiling '/tmp/destdir/usr/lib/python3.13/site-packages/foo.py'...
$ strings /tmp/destdir/usr/lib/python3.13/site-packages/__pycache__/foo.cpython-313.pyc 
(/usr/lib/python3.13/site-packages/foo.py
<module>r

However, I found that behavior surprising and I think it should be documented better. Particularly, the current documentation states:

cpython/Doc/library/compileall.rst

Lines 58 to 63 in 120c9d4

.. option:: -s strip_prefix
.. option:: -p prepend_prefix
Remove (``-s``) or append (``-p``) the given prefix of paths
recorded in the ``.pyc`` files.
Cannot be combined with ``-d``.

which to me sounds like it would remove the specified string, rather than making the path relative. Perhaps it would make sense to specify explicitly that:

  1. Removing the prefix makes paths relative to it.
  2. -s and -p can be used simultaneously.
  3. -p / can be used to make the path absolute.

Linked PRs
@mgorny mgorny added the docs Documentation in the Doc dir label May 6, 2025
@thesamesam thesamesam mentioned this issue May 7, 2025
Closed
@sharktide
Copy link
Contributor

I'll do something about this

sharktide added a commit to sharktide/cpython that referenced this issue May 26, 2025
@sharktide
pythongh-133503 Update compileall.rst's documentation of -s and `…
8f2db41 
…-p` for clarity
@sharktide sharktide mentioned this issue May 26, 2025
Merged
sharktide added a commit to sharktide/cpython that referenced this issue May 26, 2025
@sharktide
pythongh-133503: Remove note attribute and document -s separately
5e7aad9
sharktide added a commit to sharktide/cpython that referenced this issue May 26, 2025
@sharktide
pythongh-133503 Do not reference s too much in p
30de5e7
sharktide added a commit to sharktide/cpython that referenced this issue May 27, 2025
@sharktide
pythongh-133503: Remove any and all references of s and p together
5442260
sharktide added a commit to sharktide/cpython that referenced this issue May 27, 2025
@sharktide
pythongh-133503 Update compileall.rst again for clarity
6be87f8
sharktide added a commit to sharktide/cpython that referenced this issue May 27, 2025
@sharktide @picnixz
pythongh-133503 Fix wrong words - Append to Prepend
812c25c 
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
picnixz added a commit that referenced this issue Jun 1, 2025
@sharktide @picnixz
gh-133503: clarify compileall -s/-p docs (#134756)
fe6f8a3 
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 1, 2025
@sharktide @picnixz @miss-islington
pythongh-133503: clarify compileall -s/-p docs (pythonGH-134756)
e6c6f9e 
(cherry picked from commit fe6f8a3)

Co-authored-by: Rihaan Meher <meherrihaan@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
@bedevere-app bedevere-app bot mentioned this issue Jun 1, 2025
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 1, 2025
@sharktide @picnixz @miss-islington
pythongh-133503: clarify compileall -s/-p docs (pythonGH-134756)
44fb854 
(cherry picked from commit fe6f8a3)

Co-authored-by: Rihaan Meher <meherrihaan@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
@bedevere-app bedevere-app bot mentioned this issue Jun 1, 2025
Merged
picnixz added a commit that referenced this issue Jun 1, 2025
@miss-islington @sharktide @picnixz
[3.14] gh-133503: clarify compileall -s/-p docs (GH-134756) (#134997)
5d07d16 
gh-133503: clarify `compileall -s/-p` docs (GH-134756)
(cherry picked from commit fe6f8a3)

Co-authored-by: Rihaan Meher <meherrihaan@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
picnixz added a commit that referenced this issue Jun 1, 2025
@miss-islington @sharktide @picnixz
[3.13] gh-133503: clarify compileall -s/-p docs (GH-134756) (#134996)
0446628 
gh-133503: clarify `compileall -s/-p` docs (GH-134756)
(cherry picked from commit fe6f8a3)

Co-authored-by: Rihaan Meher <meherrihaan@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
docs Documentation in the Doc dir
Projects
docs issues
Status: Todo
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

gh-133503: Update compileall.rst's documentation of -s and -p for clarity, sharktide/cpython
2 participants
@mgorny @sharktide