py: Implement decorator syntax relaxation from CPython 3.9. #8503
Conversation
tests/basics/decorator_full.py
Outdated
| print("TypeError") | ||
|
|
||
| decs = [ | ||
| "1.+2j", |
There was a problem hiding this comment.
Can this test be modified so it runs on miminal targets, ie so that there doesn't need to be special handling in the test suite runners to exclude this full decorator test from certain ports?
Can this complex number be replaced with something else (or just removed)? Is it testing something (providing test coverage) that can't be tested in any other way? If so it would be good to move just this complex-number based test to another file and put it in the tests/float/ directory so it's automatically skipped.
There was a problem hiding this comment.
I removed the complex number test and cleaned up the test runners in the latest commit.
51f36ba to
76a8b0c
Compare
|
Amended the commit message to the correct format. |
|
First commit is 'py: Implements ....', that should be 'py: Implement ....' i.e. commits are in the imperative mood. |
76a8b0c to
e3b7d1d
Compare
Squashed and pushed with an updated commit message. |
e3b7d1d to
be8df3d
Compare
py/grammar.h: The grammar file hosts the core change associated with
this pull request. The decorator rule is changed to match a
namedexpr_test rule rather than a dotted_name rule. However, since the
compiler currently relies on the dotted_name rule to easily detect
built-in decorators (e.g. @micropython.bytecode), a new way of detecting
these built-ins needed to be found:
1. One possibility is to have the grammar try to match a dotted_name
first and a namedexpr_test second, but since the parser does not
bactrack through lexical tokens, it is not obvious to me how to
accomplish this.
2. Another possiblity is to write code in the compiler to visit the
AST rooted at the namedexpr_test rule of a given decorator and
determine whether said decorator is a built-in that way. This
could potentially be an expensive operation and/or lead to a
significant increase in code size.
3. [SELECTED] A third option is to introduce a grammar rule
built_in_decorator that tries to match a "micropython" string,
and backtracks into a namedexpr_test rule if there is no match.
This relies on the fact that all built-in decorators start with
"micropython". One possible concern is that any decorator
starting with the "micropython" NAME token must match with a
built-in decorator, otherwise a syntax error is issued. This is,
however, congruent with current behavior: this is exactly what
the compile_built_in_decorator function would do (before this
commit).
py/parse.c: Add a check to attempt to match the "micropython"
string to a NAME token when parsing a built_in_decrator rule. The parser
backtracks out of the rule if there is no match.
py/compile.c: Since the compile_built_in_decorator function is not used
anywhere else, I took the liberty to simplify this function to match the
new grammar.
tests/basics: Amend the decorator.py test with additional tests for
syntax, type and name errors. Adds an expected output file so that this
test works out-of-the-box with older versions of CPython than 3.9.
tests/cmdline/cmd_parsetree.py.exp: Update the expected rule IDs in the
expected output since the grammar is changed.
Signed-off-by: Rayane Chatrieux <rayane.chatrieux@gmail.com>
be8df3d to
808fdbb
Compare
| // decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE | ||
| // decorator: '@' decorator_arg NEWLINE | ||
| // decorator_arg: built_in_decorator | namedexpr_test | ||
| // built_in_decorator: 'micropython' '.' NAME |
There was a problem hiding this comment.
I think it might be cleaner/simpler to define the grammar like this:
decorator: '@' namedexpr_test NEWLINE
and then handle detection and compilation of the builtin decorator in py/compile.c (that way it can potentially support more complex builtin decorators, eg @micropython.bytecode(opt=2)).
|
This is an automated heads-up that we've just merged a Pull Request See #13763 A search suggests this PR might apply the STATIC macro to some C code. If it Although this is an automated message, feel free to @-reply to me directly if |
Synopsis: Implements the decorator syntax relaxation added to CPython 3.9 (any expression can be a decorator argument, syntactically).
Details: Details and rationale are provided below, on a per file change basis.
py/grammar.h: The grammar file hosts the core change associated with this pull request. The decorator rule is changed to match a
namedexpr_testrule rather than adotted_namerule. However, since the compiler currently relies on thedotted_namerule to easily detect built-in decorators (e.g.@micropython.bytecode), a new way of detecting these built-ins needed to be found:dotted_name(or proxy thereof) first and anamedexpr_testsecond, but since the parser does not backtrack through lexical tokens, it is not obvious to me how to accomplish this.namedexpr_testrule of a given decorator and determine whether said decorator is a built-in that way. This could potentially be an expensive operation and/or lead to a significant increase in code size.built_in_decoratorthat tries to match a"micropython"string, and backtracks into anamedexpr_testrule if there is no match. This relies on the fact that all built-in decorators start with"micropython". One possible concern is that any decorator starting with the"micropython"NAME token must match with a built-in decorator, otherwise a syntax error is issued. This is, however, congruent with current behavior: this is exactly what thecompile_built_in_decoratorfunction would do (before this commit).py/parse.c: A check is added to attempt to match the
"micropython"string to a NAME token when parsing abuilt_in_decratorrule.py/compile.c: Since the
compile_built_in_decoratorfunction is not used anywhere else, this function can be simplified to match the new grammar.tests/basics:
The old decorator.py test is renamed to decorator_minimal.py. A new set of tests are added in decorator_full.py.The decorator.py test is amended with additional tests for syntax, type and name errors. Adecorator_full.py.expdecorator.py.exp file is added to work out of the box with CPython versions older than 3.9.tests/cmdline/cmd_parsetree.py.exp: Updated the expected rule IDs since the grammar is changed.
tools/ci.sh: Excludes the decorator_full.py test from the minimal build since it uses several unsupported features (e.g. decimal/complex numbers).ports/qemu_arm/Makefile.test: Excludes the decorator_full.py test here as well.Signed-off-by: Rayane Chatrieux rayane.chatrieux@gmail.com