As far as the "splitting at the braces thing" goes: I would not rely on this not changing. Back when I wrote the original f-string tokenizer, this was just an optimization so I could play C games with pointers to null-terminated strings. I'd temporarily change "this: {{braces}} done" to be these strings, in turn:

"this: {\0"
"braces}\0"
" done\0"

After I was done, I'd replace the '\0' with whatever was originally there. Doing it this way, I didn't have to allocate space for a new string without the doubled braces. I can easily image a future where this tradeoff changes, or is only used for strings longer than some temporary fixed size buffer, or something like that.

I don't know if the PEP 701 tokenizer kept this behavior deliberately for compatibility, or if it was just easier for them, too.

For the rest of your questions: @pablogsal

Ned, you can sign up in .github/CODEOWNERS to be notified (emailed) when a PR is submitted that changes particular files (at least those that are tracked).

We kept this behavior for compatibility. We also had to deal with this in the untokenizer as well:

We could explore changing this if everyone agrees as this also was kind of a problem in the REPL:

On the other hand the change would be backwards incompatible....so I am not sure what's the best thing to do here

Thanks for all the details. I've adjusted coverage.py for the tokenization as it is now, and don't depend on the tokens breaking at the braces. So no need for change on my behalf. If you do make a change, my tests should alert me!

are you ok if we close this issue? What are your thoughts @ericvsmith ?