I worry about some of the rather verbose comments in the grammar file, and the various error rules.

I can try to eliminate alternatives if they start with "invalid_" but I think it will not be super resilient :(

Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed).

Would you like to bring lookaheads back?

And maybe the ordering and formatting of some of the rules could be better.

Not much we can do here with a custom lexer unless we touch the original file.

PS. Your initial comments still shows variables (a=...) but the script correctly erases those.

Yeah, I removed the variables in the last amend after I realized that we wanted to get rid of them.

I worry about some of the rather verbose comments in the grammar file, and the various error rules.

I can try to eliminate alternatives if they start with "invalid_" but I think it will not be super resilient :(

Plus, you'd have to eliminate both the alternatives and the rules. But these really do not deserve to be listed in the reference manual.

Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed).

Would you like to bring lookaheads back?

Only the ones that are significant -- and those in formal parameters are significant (without them, def foo(bar baz) would be legal)

And maybe the ordering and formatting of some of the rules could be better.

Not much we can do here with a custom lexer unless we touch the original file.

Yeah, we can do that later.

Only the ones that are significant -- and those in formal parameters are significant (without them, def foo(bar baz) would be legal)

Hmmm.... is there any obvious automatic way (using regular expressions) to detect the ones we want from the ones we don't want? Otherwise, we may need to maintain a copy of the grammar for displaying here and manually trim what we need.

Hmmm.... is there any obvious automatic way (using regular expressions) to detect the ones we want from the ones we don't want?

A heuristic that is currently 100% correct is that a lookahead is only used for optimisation and can be skipped if and only if it is a positive lookahead and it appears first in its alternative. Although this is currently correct, it seems like something that could very easily be invalidated in the future, so I don't know if it can be used in the script.

I think I'm going to do something with this after all. Let's see if we can't make the highlighter also remove invalid_* rules and comments.

Okay, @lysnikolaou we may need your review because now it's both Pablo's and my code.

Should we backport this to 3.9? (Minus the Grammar/Grammar delete. :-)

I think it makes sense to do the backport. What do you think @lysnikolaou ?

Okay then @pablogsal what do you think of my version?

Should we backport this to 3.9? (Minus the Grammar/Grammar delete. :-)

I think it makes sense to do the backport. What do you think @lysnikolaou ?

I also think that backporting this to 3.9 makes sense.

The new version I pushed has a longer intro in grammar.rst, both to explain (some of) what's omitted and to clarify the notation. It also restores most lookaheads, except for the positive ones that are purely optimizations (as a heuristic I use | followed by a positive lookahead). It makes the grammar a little uglier but reduces the possibility of looking like an ambiguity.

Thanks @pablogsal for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

Sorry, @pablogsal and @gvanrossum, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 72cabb2aa636272e608285f5a6ba83b62be9be4e 3.9

GH-21641 is a backport of this pull request to the 3.9 branch.