Skip to content

Parts of strings that look like comments being stripped out of nested strings with debug specifier #135148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
tom-pytel opened this issue Jun 4, 2025 · 6 comments
Open

Parts of strings that look like comments being stripped out of nested strings with debug specifier #135148

tom-pytel opened this issue Jun 4, 2025 · 6 comments
Labels
3.12 only security fixes 3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes type-bug An unexpected behavior, bug, or error

Comments

@tom-pytel
Copy link
Contributor

tom-pytel commented Jun 4, 2025 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

Is this desired behavior?

$ ./python -m ast
f'{""" # this is part of the string, it shouldn't be stripped out
"""=}'
Module(
   body=[
      Expr(
         value=JoinedStr(
            values=[
               Constant(value='""" \n"""='),
               FormattedValue(
                  value=Constant(value=" # this is part of the string, it shouldn't be stripped out\n"),
                  conversion=114)]))])

$ ./python -m ast
t'{""" # this is part of the string, it shouldn't be stripped out
"""=}'
Module(
   body=[
      Expr(
         value=TemplateStr(
            values=[
               Constant(value='""" \n"""='),
               Interpolation(
                  value=Constant(value=" # this is part of the string, it shouldn't be stripped out\n"),
                  str='""" \n"""',
                  conversion=114)]))])

Even if they are not actually multiline:

$ python -m ast
t'{" # nooo "=}'
Module(
   body=[
      Expr(
         value=TemplateStr(
            values=[
               Constant(value='" '),
               Interpolation(
                  value=Constant(value=' # nooo '),
                  str='"',
                  conversion=114)]))])

Seems to go back to 3.12

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

No response

Linked PRs
@tom-pytel tom-pytel added the type-bug An unexpected behavior, bug, or error label Jun 4, 2025
@JelleZijlstra
Copy link
Member

Yeah this is clearly wrong. cc @pablogsal @lysnikolaou

@JelleZijlstra JelleZijlstra changed the title Parts of strings that look like comments being stripped out of multiline strings Parts of strings that look like comments being stripped out of nested strings with debug specifier Jun 5, 2025
@JelleZijlstra
Copy link
Member

This is actually a regression in 3.12, likely due to PEP-701:

$ uv run --python=3.12 python
Python 3.12.2 (main, May  6 2024, 20:50:52) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f'{" # no "=}'
'" \' # no \''
>>> 
$ uv run --python=3.11 python
Python 3.11.11 (main, Feb 12 2025, 15:06:01) [Clang 19.1.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> f'{" # no "=}'
'" # no "=\' # no \''

Removing the release blocker label as it's old.

@JelleZijlstra JelleZijlstra added 3.12 only security fixes 3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes and removed release-blocker labels Jun 5, 2025
@tom-pytel
Copy link
Contributor Author

tom-pytel commented Jun 5, 2025 •
edited
Loading

Not just debug specifier, note the .str field:

$ ./python -m ast
t'{" # no "}'
Module(
   body=[
      Expr(
         value=TemplateStr(
            values=[
               Interpolation(
                  value=Constant(value=' # no '),
                  str='"',
                  conversion=-1)]))])

Could this be problematic?

>>> t'{" ! important information ! "}'.interpolations
(Interpolation(' ! important information ! ', '" ! important information ! "', None, ''),)

>>> t'{" # important information # "}'.interpolations
(Interpolation(' # important information # ', '"', None, ''),)

@lysnikolaou
Copy link
Member

Oof, yeah, this was probably introduced in #112284. We'll need to fix it, but the fix is not exactly straight-forward. I'll take a deeper look tomorrow.

@tom-pytel
Copy link
Contributor Author

the fix is not exactly straight-forward.

Not sure if this would apply in the context of the parser but worked for me before I realized it wasn't matching up.

  1. In range of interest exclude all lines which have multiline strings which do not end on the line.

  2. For each remaining line store maximum end column of expressions that end on that line and restrict comment checks to past this column.

pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2025
@pablogsal
pythongh-135148: Correctly handle f/t strings with comments and debug…
9e34669 
… expressions
@bedevere-app bedevere-app bot mentioned this issue Jun 6, 2025
Open
@pablogsal
Copy link
Member

pablogsal commented Jun 6, 2025 •
edited
Loading

Oof, yeah, this was probably introduced in #112284. We'll need to fix it, but the fix is not exactly straight-forward. I'll take a deeper look tomorrow.

I have give this a go but is not pretty: #135198 I don't think we can do it simpler than this to be honest :S

pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2025
@pablogsal
pythongh-135148: Correctly handle f/t strings with comments and debug…
bd49471 
… expressions
pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2025
@pablogsal
pythongh-135148: Correctly handle f/t strings with comments and debug…
5f8456e 
… expressions
pablogsal added a commit to pablogsal/cpython that referenced this issue Jun 6, 2025
@pablogsal
pythongh-135148: Correctly handle f/t strings with comments and debug…
ef9d22f 
… expressions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 only security fixes 3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes type-bug An unexpected behavior, bug, or error
Projects
Release and Deferred blockers 🚫
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
4 participants
@JelleZijlstra @pablogsal @lysnikolaou @tom-pytel