Skip to content

tokenize.generate_tokens no longer reports lineno for indentation errors #104741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nedbat opened this issue May 22, 2023 · 2 comments
Closed

tokenize.generate_tokens no longer reports lineno for indentation errors #104741

nedbat opened this issue May 22, 2023 · 2 comments
Assignees
@pablogsal
Labels
release-blocker type-bug An unexpected behavior, bug, or error

Comments

@nedbat
Copy link
Member

nedbat commented May 22, 2023 •
edited by bedevere-bot
Loading

tokenize.generate_tokens raises an exception for indentation errors. It used to populate the .lineno attribute of the exception, but as of yesterday, .lineno is None.

Test file:

# parsebug.py
import io
import sys
import tokenize

text = """\
0 spaces
  2
 1
"""

print(sys.version)
readline = io.StringIO(text).readline
try:
    list(tokenize.generate_tokens(readline))
except Exception as exc:
    print(f"{exc.lineno = }")
    raise

With 3.12.0a7:

% .tox/py312/bin/python parsebug.py
3.12.0a7 (main, Apr  5 2023, 05:51:58) [Clang 14.0.3 (clang-1403.0.22.14.1)]
exc.lineno = 3
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/parsebug.py", line 15, in <module>
    list(tokenize.generate_tokens(readline))
  File "/usr/local/pyenv/pyenv/versions/3.12.0a7/lib/python3.12/tokenize.py", line 516, in _tokenize
    raise IndentationError(
  File "<tokenize>", line 3
    1
IndentationError: unindent does not match any outer indentation level

With a nightly 3.12 build:

% .tox/anypy/bin/python parsebug.py
3.12.0a7+ (heads/main:9bc80dac47, May 22 2023, 05:34:19) [Clang 14.0.3 (clang-1403.0.22.14.1)]
exc.lineno = None
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/parsebug.py", line 15, in <module>
    list(tokenize.generate_tokens(readline))
  File "/usr/local/cpython/lib/python3.12/tokenize.py", line 451, in _tokenize
    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):
  File "/usr/local/cpython/lib/python3.12/tokenize.py", line 542, in _generate_tokens_from_c_tokenizer
    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
IndentationError: unindent does not match any outer indentation level (<tokenize>, line 3)

The line number is clearly known, since it's reported. Is there another way I should be getting the line number from the exception?

/cc @mgmacias95

Linked PRs
@nedbat nedbat added the type-bug An unexpected behavior, bug, or error label May 22, 2023
@pablogsal pablogsal self-assigned this May 22, 2023
@pablogsal
Copy link
Member

Is there another way I should be getting the line number from the exception?

Nop, this is something we need to fix on our side. Thanks for raising it with us!

@mgmacias95 will make a PR shortly!

@bedevere-bot bedevere-bot mentioned this issue May 22, 2023
Merged
pablogsal pushed a commit that referenced this issue May 22, 2023
@mgmacias95
gh-104741: Add line number attribute to indentation error exception (#…
729b252 
…104743)
@nedbat
Copy link
Member Author

nedbat commented May 22, 2023

I can confirm that #104743 fixes this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pablogsal pablogsal

Labels
release-blocker type-bug An unexpected behavior, bug, or error
Projects
Release and Deferred blockers 🚫
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nedbat @Yhg1s @pablogsal