Skip to content

gh-126835: Rename AST optimization related stuff after moving const folding to the peephole optimizier #131830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
May 4, 2025
Merged

gh-126835: Rename AST optimization related stuff after moving const folding to the peephole optimizier #131830

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
678b085
rename ast_opt.c -> ast_process.c
WolframAlph Mar 28, 2025
e080e27
_PyASTOptimizeState -> _PyASTProcessState
WolframAlph Mar 28, 2025
5153ea5
_PyAST_Optimize -> _PyAST_Process
WolframAlph Mar 28, 2025
ad5d080
_PyCompile_AstOptimize -> _PyCompile_AstProcess
WolframAlph Mar 28, 2025
3a6abef
add news
WolframAlph Mar 28, 2025
e2a3cbc
address review
WolframAlph Mar 28, 2025
2d1bc6f
mention full file path
WolframAlph Mar 28, 2025
3b43460
rename process to preprocess
WolframAlph May 4, 2025
72cca4f
_PyASTProcessState -> _PyASTPreprocessState
WolframAlph May 4, 2025
8a59650
update comment
WolframAlph May 4, 2025
a30846d
Merge branch 'main' into rename-ast-opt
Eclips4 May 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ Include/internal/pycore_time.h @pganssle @abalkin

# AST
Python/ast.c @isidentical @JelleZijlstra @eclips4
Python/ast_opt.c @isidentical @eclips4
Python/ast_preprocess.c @isidentical @eclips4
Parser/asdl.py @isidentical @JelleZijlstra @eclips4
Parser/asdl_c.py @isidentical @JelleZijlstra @eclips4
Lib/ast.py @isidentical @JelleZijlstra @eclips4
Expand Down
6 changes: 3 additions & 3 deletions Include/internal/pycore_compile.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,16 @@ PyAPI_FUNC(PyCodeObject*) _PyAST_Compile(
int optimize,
struct _arena *arena);

/* AST optimizations */
extern int _PyCompile_AstOptimize(
/* AST preprocessing */
extern int _PyCompile_AstPreprocess(
struct _mod *mod,
PyObject *filename,
PyCompilerFlags *flags,
int optimize,
struct _arena *arena,
int syntax_check_only);

extern int _PyAST_Optimize(
extern int _PyAST_Preprocess(
struct _mod *,
struct _arena *arena,
PyObject *filename,
Expand Down
4 changes: 2 additions & 2 deletions InternalDocs/compiler.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -505,8 +505,8 @@ Important files
* [Python/ast.c](../Python/ast.c):
Used for validating the AST.

* [Python/ast_opt.c](../Python/ast_opt.c):
Optimizes the AST.
* [Python/ast_preprocess.c](../Python/ast_preprocess.c):
Preprocesses the AST before compiling.

* [Python/ast_unparse.c](../Python/ast_unparse.c):
Converts the AST expression node back into a string (for string annotations).
Expand Down
2 changes: 1 addition & 1 deletion Makefile.pre.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,7 @@ PYTHON_OBJS= \
Python/asdl.o \
Python/assemble.o \
Python/ast.o \
Python/ast_opt.o \
Python/ast_preprocess.o \
Python/ast_unparse.o \
Python/bltinmodule.o \
Python/brc.o \
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2025-03-28-15-06-55.gh-issue-126835.IpcMTn.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Move constant folding to the peephole optimizer. Rename AST optimization
related files (Python/ast_opt.c -> Python/ast_preprocess.c), structs (_PyASTOptimizeState -> _PyASTPreprocessState)
and functions (_PyAST_Optimize -> _PyAST_Preprocess, _PyCompile_AstOptimize -> _PyCompile_AstPreprocess).
2 changes: 1 addition & 1 deletion PCbuild/_freeze_module.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@
<ClCompile Include="..\Python\asdl.c" />
<ClCompile Include="..\Python\assemble.c" />
<ClCompile Include="..\Python\ast.c" />
<ClCompile Include="..\Python\ast_opt.c" />
<ClCompile Include="..\Python\ast_preprocess.c" />
<ClCompile Include="..\Python\ast_unparse.c" />
<ClCompile Include="..\Python\bltinmodule.c" />
<ClCompile Include="..\Python\brc.c" />
Expand Down
2 changes: 1 addition & 1 deletion PCbuild/_freeze_module.vcxproj.filters
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
<ClCompile Include="..\Python\ast.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_opt.c">
<ClCompile Include="..\Python\ast_preprocess.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_unparse.c">
Expand Down
2 changes: 1 addition & 1 deletion PCbuild/pythoncore.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -580,7 +580,7 @@
<ClCompile Include="..\Python\asdl.c" />
<ClCompile Include="..\Python\assemble.c" />
<ClCompile Include="..\Python\ast.c" />
<ClCompile Include="..\Python\ast_opt.c" />
<ClCompile Include="..\Python\ast_preprocess.c" />
<ClCompile Include="..\Python\ast_unparse.c" />
<ClCompile Include="..\Python\bltinmodule.c" />
<ClCompile Include="..\Python\bootstrap_hash.c" />
Expand Down
2 changes: 1 addition & 1 deletion PCbuild/pythoncore.vcxproj.filters
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1325,7 +1325,7 @@
<ClCompile Include="..\Python\ast.c">
<Filter>Python</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_opt.c">
<ClCompile Include="..\Python\ast_preprocess.c">
<Filter>Python</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_unparse.c">
Expand Down
78 changes: 39 additions & 39 deletions Python/ast_opt.c → Python/ast_preprocess.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/* AST Optimizer */
/* AST pre-processing */
#include "Python.h"
#include "pycore_ast.h" // _PyAST_GetDocString()
#include "pycore_c_array.h" // _Py_CArray_EnsureCapacity()
Expand All @@ -22,7 +22,7 @@ typedef struct {

_Py_c_array_t cf_finally; /* context for PEP 765 check */
int cf_finally_used;
} _PyASTOptimizeState;
} _PyASTPreprocessState;

#define ENTER_RECURSIVE() \
if (Py_EnterRecursiveCall(" during compilation")) { \
Expand All @@ -32,14 +32,14 @@ if (Py_EnterRecursiveCall(" during compilation")) { \
#define LEAVE_RECURSIVE() Py_LeaveRecursiveCall();

static ControlFlowInFinallyContext*
get_cf_finally_top(_PyASTOptimizeState *state)
get_cf_finally_top(_PyASTPreprocessState *state)
{
int idx = state->cf_finally_used;
return ((ControlFlowInFinallyContext*)state->cf_finally.array) + idx;
}

static int
push_cf_context(_PyASTOptimizeState *state, stmt_ty node, bool finally, bool funcdef, bool loop)
push_cf_context(_PyASTPreprocessState *state, stmt_ty node, bool finally, bool funcdef, bool loop)
{
if (_Py_CArray_EnsureCapacity(&state->cf_finally, state->cf_finally_used+1) < 0) {
return 0;
Expand All @@ -55,14 +55,14 @@ push_cf_context(_PyASTOptimizeState *state, stmt_ty node, bool finally, bool fun
}

static void
pop_cf_context(_PyASTOptimizeState *state)
pop_cf_context(_PyASTPreprocessState *state)
{
assert(state->cf_finally_used > 0);
state->cf_finally_used--;
}

static int
control_flow_in_finally_warning(const char *kw, stmt_ty n, _PyASTOptimizeState *state)
control_flow_in_finally_warning(const char *kw, stmt_ty n, _PyASTPreprocessState *state)
{
PyObject *msg = PyUnicode_FromFormat("'%s' in a 'finally' block", kw);
if (msg == NULL) {
Expand All @@ -76,7 +76,7 @@ control_flow_in_finally_warning(const char *kw, stmt_ty n, _PyASTOptimizeState *
}

static int
before_return(_PyASTOptimizeState *state, stmt_ty node_)
before_return(_PyASTPreprocessState *state, stmt_ty node_)
{
if (state->cf_finally_used > 0) {
ControlFlowInFinallyContext *ctx = get_cf_finally_top(state);
Expand All @@ -90,7 +90,7 @@ before_return(_PyASTOptimizeState *state, stmt_ty node_)
}

static int
before_loop_exit(_PyASTOptimizeState *state, stmt_ty node_, const char *kw)
before_loop_exit(_PyASTPreprocessState *state, stmt_ty node_, const char *kw)
{
if (state->cf_finally_used > 0) {
ControlFlowInFinallyContext *ctx = get_cf_finally_top(state);
Expand Down Expand Up @@ -365,7 +365,7 @@ optimize_format(expr_ty node, PyObject *fmt, asdl_expr_seq *elts, PyArena *arena
}

static int
fold_binop(expr_ty node, PyArena *arena, _PyASTOptimizeState *state)
fold_binop(expr_ty node, PyArena *arena, _PyASTPreprocessState *state)
{
if (state->syntax_check_only) {
return 1;
Expand All @@ -389,18 +389,18 @@ fold_binop(expr_ty node, PyArena *arena, _PyASTOptimizeState *state)
return 1;
}

static int astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);
static int astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTPreprocessState *state);

#define CALL(FUNC, TYPE, ARG) \
if (!FUNC((ARG), ctx_, state)) \
Expand Down Expand Up @@ -436,7 +436,7 @@ stmt_seq_remove_item(asdl_stmt_seq *stmts, Py_ssize_t idx)
}

static int
astfold_body(asdl_stmt_seq *stmts, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_body(asdl_stmt_seq *stmts, PyArena *ctx_, _PyASTPreprocessState *state)
{
int docstring = _PyAST_GetDocString(stmts) != NULL;
if (docstring && (state->optimize >= 2)) {
Expand Down Expand Up @@ -466,7 +466,7 @@ astfold_body(asdl_stmt_seq *stmts, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
switch (node_->kind) {
case Module_kind:
Expand All @@ -488,7 +488,7 @@ astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
ENTER_RECURSIVE();
switch (node_->kind) {
Expand Down Expand Up @@ -613,14 +613,14 @@ astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
CALL(astfold_expr, expr_ty, node_->value);
return 1;
}

static int
astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
CALL(astfold_expr, expr_ty, node_->target);
CALL(astfold_expr, expr_ty, node_->iter);
Expand All @@ -629,7 +629,7 @@ astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTOptimizeState
}

static int
astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
CALL_SEQ(astfold_arg, arg, node_->posonlyargs);
CALL_SEQ(astfold_arg, arg, node_->args);
Expand All @@ -642,7 +642,7 @@ astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
if (!(state->ff_features & CO_FUTURE_ANNOTATIONS)) {
CALL_OPT(astfold_expr, expr_ty, node_->annotation);
Expand All @@ -651,7 +651,7 @@ astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
ENTER_RECURSIVE();
switch (node_->kind) {
Expand Down Expand Up @@ -806,7 +806,7 @@ astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
switch (node_->kind) {
case ExceptHandler_kind:
Expand All @@ -820,15 +820,15 @@ astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTOptimizeState
}

static int
astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
CALL(astfold_expr, expr_ty, node_->context_expr);
CALL_OPT(astfold_expr, expr_ty, node_->optional_vars);
return 1;
}

static int
fold_const_match_patterns(expr_ty node, PyArena *ctx_, _PyASTOptimizeState *state)
fold_const_match_patterns(expr_ty node, PyArena *ctx_, _PyASTPreprocessState *state)
{
if (state->syntax_check_only) {
return 1;
Expand Down Expand Up @@ -869,7 +869,7 @@ fold_const_match_patterns(expr_ty node, PyArena *ctx_, _PyASTOptimizeState *stat
}

static int
astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
// Currently, this is really only used to form complex/negative numeric
// constants in MatchValue and MatchMapping nodes
Expand Down Expand Up @@ -911,7 +911,7 @@ astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
CALL(astfold_pattern, expr_ty, node_->pattern);
CALL_OPT(astfold_expr, expr_ty, node_->guard);
Expand All @@ -920,7 +920,7 @@ astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTOptimizeState *stat
}

static int
astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
{
switch (node_->kind) {
case TypeVar_kind:
Expand All @@ -942,11 +942,11 @@ astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTOptimizeState *stat
#undef CALL_SEQ

int
_PyAST_Optimize(mod_ty mod, PyArena *arena, PyObject *filename, int optimize,
int ff_features, int syntax_check_only)
_PyAST_Preprocess(mod_ty mod, PyArena *arena, PyObject *filename, int optimize,
int ff_features, int syntax_check_only)
{
_PyASTOptimizeState state;
memset(&state, 0, sizeof(_PyASTOptimizeState));
_PyASTPreprocessState state;
memset(&state, 0, sizeof(_PyASTPreprocessState));
state.filename = filename;
state.optimize = optimize;
state.ff_features = ff_features;
Expand Down
2 changes: 1 addition & 1 deletion Python/bltinmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -845,7 +845,7 @@ builtin_compile_impl(PyObject *module, PyObject *source, PyObject *filename,
goto error;
}
int syntax_check_only = ((flags & PyCF_OPTIMIZED_AST) == PyCF_ONLY_AST); /* unoptiomized AST */
if (_PyCompile_AstOptimize(mod, filename, &cf, optimize,
if (_PyCompile_AstPreprocess(mod, filename, &cf, optimize,
arena, syntax_check_only) < 0) {
_PyArena_Free(arena);
goto error;
Expand Down
8 changes: 4 additions & 4 deletions Python/compile.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ compiler_setup(compiler *c, mod_ty mod, PyObject *filename,
c->c_optimize = (optimize == -1) ? _Py_GetConfig()->optimization_level : optimize;
c->c_save_nested_seqs = false;

if (!_PyAST_Optimize(mod, arena, filename, c->c_optimize, merged, 0)) {
if (!_PyAST_Preprocess(mod, arena, filename, c->c_optimize, merged, 0)) {
return ERROR;
}
c->c_st = _PySymtable_Build(mod, filename, &c->c_future);
Expand Down Expand Up @@ -1481,8 +1481,8 @@ _PyAST_Compile(mod_ty mod, PyObject *filename, PyCompilerFlags *pflags,
}

int
_PyCompile_AstOptimize(mod_ty mod, PyObject *filename, PyCompilerFlags *cf,
int optimize, PyArena *arena, int no_const_folding)
_PyCompile_AstPreprocess(mod_ty mod, PyObject *filename, PyCompilerFlags *cf,
int optimize, PyArena *arena, int no_const_folding)
{
_PyFutureFeatures future;
if (!_PyFuture_FromAST(mod, filename, &future)) {
Expand All @@ -1492,7 +1492,7 @@ _PyCompile_AstOptimize(mod_ty mod, PyObject *filename, PyCompilerFlags *cf,
if (optimize == -1) {
optimize = _Py_GetConfig()->optimization_level;
}
if (!_PyAST_Optimize(mod, arena, filename, optimize, flags, no_const_folding)) {
if (!_PyAST_Preprocess(mod, arena, filename, optimize, flags, no_const_folding)) {
return -1;
}
return 0;
Expand Down
2 changes: 1 addition & 1 deletion Python/pythonrun.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1498,7 +1498,7 @@ Py_CompileStringObject(const char *str, PyObject *filename, int start,
}
if (flags && (flags->cf_flags & PyCF_ONLY_AST)) {
int syntax_check_only = ((flags->cf_flags & PyCF_OPTIMIZED_AST) == PyCF_ONLY_AST); /* unoptiomized AST */
if (_PyCompile_AstOptimize(mod, filename, flags, optimize, arena, syntax_check_only) < 0) {
if (_PyCompile_AstPreprocess(mod, filename, flags, optimize, arena, syntax_check_only) < 0) {
_PyArena_Free(arena);
return NULL;
}
Expand Down
1 change: 0 additions & 1 deletion Tools/c-analyzer/cpython/ignored.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,6 @@ Objects/unicodeobject.c unicode_translate_call_errorhandler argparse -
Parser/parser.c - reserved_keywords -
Parser/parser.c - soft_keywords -
Parser/lexer/lexer.c - type_comment_prefix -
Python/ast_opt.c fold_unaryop ops -
Python/ceval.c - _PyEval_BinaryOps -
Python/ceval.c - _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS -
Python/codecs.c - Py_hexdigits -
Expand Down
Loading