Rewrite exported function #21
Conversation
lysnikolaou
commented
Mar 24, 2020
- It now initializes the keywords field so that the keywords variable can be static.
- It also accepts an argument, in order to be able to run the parser with different start rules.
* It now initializes the keywords field so that the keywords
variable can be static.
* It also accepts an argument, in order to be able to
run the parser with different start rules.
There was a problem hiding this comment.
Seems that the pegen tests are failing because you need to get rid of the extern const int n_keyword_lists in peg_parser/pegen.h:
python3.8 -m pegen -q -c data/python.gram -o peg_parser/parse.c --compile-extension
peg_parser/parse.c:3:18: error: static declaration of ‘n_keyword_lists’ follows non-static declaration
static const int n_keyword_lists = 9;
^~~~~~~~~~~~~~~
In file included from peg_parser/parse.c:2:0:
peg_parser/pegen.h:84:18: note: previous declaration of ‘n_keyword_lists’ was here
extern const int n_keyword_lists;
^~~~~~~~~~~~~~~
peg_parser/parse.c:4:22: error: static declaration of ‘reserved_keywords’ follows non-static declaration
static KeywordToken *reserved_keywords[] = {
^~~~~~~~~~~~~~~~~
In file included from peg_parser/parse.c:2:0:
peg_parser/pegen.h:85:22: note: previous declaration of ‘reserved_keywords’ was here
extern KeywordToken *reserved_keywords[];
^~~~~~~~~~~~~~~~~
| @@ -14,6 +14,11 @@ enum INPUT_MODE { | |||
| }; | |||
| typedef enum INPUT_MODE INPUT_MODE; | |||
|
|
|||
| enum START_RULE { | |||
There was a problem hiding this comment.
We'll probably add more, we we need to choose between start rules. I'm currently working on maybe changing the start rule for f-string to expressions_rule and for this I added another field EXPRESSIONS to this enum.
There was a problem hiding this comment.
(Probably for a separate PR.)
Maybe we could pass a string literal instead? By convention we could have NULL -> start and a few other names could be recognized based on metadata given in the .gram file, e.g. [1]
@alternate_starts "single_input file_input eval_input"
which would generate a little table at the end
static struct { char *, mod_ty (*)(Parser *) } [] alternate_starts = {
{"single_input", single_input_rule},
...
NULL,
};
and then the dispatch function could just look up the argument in the table. (This is a negligible cost given everything else that goes into parsing even one line.)
[1] We'd need to add a few new rules named single_input, file_input, eval_input that correspond to those targets in Grammar/Grammar.
I intentionally chose to ignore these failures, since we are about to remove all the C code from Tools/peg_generator/peg_parser and only rely on the C code in Parser/pegen. See #14 and we-like-parsers/pegen_experiments#235. |
Grammar/python.gram
Outdated
| @@ -1,12 +1,24 @@ | |||
| # Simplified grammar for Python | |||
|
|
|||
| @bytecode True | |||
| @modulename 'peg_parser' # Non needed for now, but might be needed later | |||
| @modulename 'peg_parser' # Not needed for now, but might be needed later | |||
There was a problem hiding this comment.
Honestly I'd take it out until we find we need it again.
| @@ -14,6 +14,11 @@ enum INPUT_MODE { | |||
| }; | |||
| typedef enum INPUT_MODE INPUT_MODE; | |||
|
|
|||
| enum START_RULE { | |||
There was a problem hiding this comment.
(Probably for a separate PR.)
Maybe we could pass a string literal instead? By convention we could have NULL -> start and a few other names could be recognized based on metadata given in the .gram file, e.g. [1]
@alternate_starts "single_input file_input eval_input"
which would generate a little table at the end
static struct { char *, mod_ty (*)(Parser *) } [] alternate_starts = {
{"single_input", single_input_rule},
...
NULL,
};
and then the dispatch function could just look up the argument in the table. (This is a negligible cost given everything else that goes into parsing even one line.)
[1] We'd need to add a few new rules named single_input, file_input, eval_input that correspond to those targets in Grammar/Grammar.