gh-102856: Python tokenizer implementation for PEP 701#104323

Merged

pablogsal merged 20 commits intopython:mainfrom

mgmacias95:python_tokenizer

May 21, 2023

Commits on May 18, 2023

First iteration

mgmacias95
authored and
pablogsal
committed
Handle escaping {

mgmacias95
authored and
pablogsal
committed
nested expressions

mgmacias95
authored and
pablogsal
committed
Recursive expression tokenization

mgmacias95
authored and
pablogsal
committed
Remove intermediate token created for dev purposes

mgmacias95
authored and
pablogsal
committed
More improvements

mgmacias95
authored and
pablogsal
committed
fix handling of } tokens

mgmacias95
authored and
pablogsal
committed
other tokenizer
pablogsal
committed
Some progress
pablogsal
committed
Fix more bugs
pablogsal
committed
Fix more problems
pablogsal
committed
Use IA to clean code
pablogsal
committed
Remove lel
pablogsal
committed
Remove whitespace
pablogsal
committed
Fix docs
mgmacias95
committed

Commits on May 19, 2023

Commits on May 20, 2023