Skip to content

bpo-42202: Store func annotations as single tuple at bytecode level #23316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
Nov 25, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
b4937e9
Add co_annotations field to codeobject
uriyyo Nov 16, 2020
7ff803e
Update Objects/codeobject.c
uriyyo Nov 16, 2020
b3bca0e
Update Objects/codeobject.c
uriyyo Nov 16, 2020
ec0e565
Update Objects/funcobject.c
uriyyo Nov 16, 2020
6abc7e7
Update Python/compile.c
uriyyo Nov 16, 2020
4c09965
Add PyCode_NewWithAnnotations function in order to create CodeObject …
uriyyo Nov 16, 2020
5e03c3e
Use co_annotations for module and class annotations
uriyyo Nov 16, 2020
1fc24f4
Change format of co_annotations to single tuple of string
uriyyo Nov 17, 2020
da237df
Regenerate importlib
uriyyo Nov 17, 2020
201a816
Simplify compiler_add_annotation function
uriyyo Nov 17, 2020
9f7a4eb
Merge remote-tracking branch 'upstream/master' into fix-issue-42202
uriyyo Nov 18, 2020
6742c42
Remove co_annotations. Store func annotations as single tuple at byte…
uriyyo Nov 18, 2020
a71ad66
Update Python/compile.c
uriyyo Nov 18, 2020
3ebf232
Update Objects/funcobject.c
uriyyo Nov 18, 2020
fde6f39
Update Objects/funcobject.c
uriyyo Nov 18, 2020
bb21767
Update Objects/funcobject.c
uriyyo Nov 18, 2020
be73760
Update Objects/funcobject.c
uriyyo Nov 18, 2020
b273dc8
Change local variable name from annotations to anns
uriyyo Nov 18, 2020
b54f9ff
Use correct variable name
uriyyo Nov 18, 2020
34b1c4f
Update Objects/funcobject.c
methane Nov 18, 2020
fa67386
Replace PyTuple_Size with PyTuple_GET_SIZE
uriyyo Nov 19, 2020
9e61756
Fix bug with partially inited annotations dict
uriyyo Nov 20, 2020
0a3ddda
Update docs and bytecode MAGIC_NUMBER
uriyyo Nov 23, 2020
52bda16
Merge branch 'master' into fix-issue-42202
uriyyo Nov 23, 2020
97f51f4
Regenerate importlib
uriyyo Nov 23, 2020
f39234a
Update Doc/library/dis.rst
methane Nov 24, 2020
cd12313
Simplify annotations tuple creation
uriyyo Nov 24, 2020
c12fc1c
Remove unnecessary goto, cleanup code
uriyyo Nov 24, 2020
b233f2c
Update Doc/whatsnew/3.10.rst
uriyyo Nov 24, 2020
85b354b
Remove trailing whitespaces
uriyyo Nov 24, 2020
5409fad
Add Inada to list of contributors
uriyyo Nov 25, 2020
1bc9561
Update _bootstrap_external.py
methane Nov 25, 2020
957ce6d
Regenerate importlib
uriyyo Nov 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Doc/library/dis.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1144,11 +1144,13 @@ All of the following opcodes use their arguments.
* ``0x01`` a tuple of default values for positional-only and
positional-or-keyword parameters in positional order
* ``0x02`` a dictionary of keyword-only parameters' default values
* ``0x04`` an annotation dictionary
* ``0x04`` a tuple of strings containing parameters' annotations
* ``0x08`` a tuple containing cells for free variables, making a closure
* the code associated with the function (at TOS1)
* the :term:`qualified name` of the function (at TOS)

.. versionchanged:: 3.10
Flag value ``0x04`` is a tuple of strings instead of dictionary

.. opcode:: BUILD_SLICE (argc)

Expand Down
11 changes: 11 additions & 0 deletions Doc/whatsnew/3.10.rst
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,11 @@ Optimizations
for more details. (Contributed by Victor Stinner and Pablo Galindo in
:issue:`38980`)

* Function parameters and their annotations are no longer computed at runtime,
but rather at compilation time. They are stored as a tuple of strings at the
bytecode level. It is now around 100% faster to create a function with parameter
annotations. (Contributed by Yurii Karabas and Inada Naoki in :issue:`42202`)

Deprecated
==========

Expand Down Expand Up @@ -457,6 +462,12 @@ Changes in the Python API
have been renamed to *exc*.
(Contributed by Zackery Spytz and Matthias Bussonnier in :issue:`26389`.)

CPython bytecode changes
========================

* The ``MAKE_FUNCTION`` instruction accepts tuple of strings as annotations
instead of dictionary.
(Contributed by Yurii Karabas and Inada Naoki in :issue:`42202`)

Build Changes
=============
Expand Down
3 changes: 2 additions & 1 deletion Lib/importlib/_bootstrap_external.py
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,7 @@ def _write_atomic(path, data, mode=0o666):
# Python 3.9a2 3425 (simplify bytecodes for **value unpacking)
# Python 3.10a1 3430 (Make 'annotations' future by default)
# Python 3.10a1 3431 (New line number table format -- PEP 626)
# Python 3.10a2 3432 (Function annotation for MAKE_FUNCTION is changed from dict to tuple bpo-42202)

#
# MAGIC must change whenever the bytecode emitted by the compiler may no
Expand All @@ -320,7 +321,7 @@ def _write_atomic(path, data, mode=0o666):
# Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
# in PC/launcher.c must also be updated.

MAGIC_NUMBER = (3431).to_bytes(2, 'little') + b'\r\n'
MAGIC_NUMBER = (3432).to_bytes(2, 'little') + b'\r\n'
_RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c

_PYCACHE = '__pycache__'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Change function parameters annotations internal representation to tuple
of strings. Patch provided by Yurii Karabas.
19 changes: 19 additions & 0 deletions Objects/funcobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,25 @@ func_get_annotations(PyFunctionObject *op, void *Py_UNUSED(ignored))
if (op->func_annotations == NULL)
return NULL;
}
if (PyTuple_CheckExact(op->func_annotations)) {
PyObject *ann_tuple = op->func_annotations;
PyObject *ann_dict = PyDict_New();
if (ann_dict == NULL) {
return NULL;
}

assert(PyTuple_GET_SIZE(ann_tuple) % 2 == 0);

for (Py_ssize_t i = 0; i < PyTuple_GET_SIZE(ann_tuple); i += 2) {
int err = PyDict_SetItem(ann_dict,
PyTuple_GET_ITEM(ann_tuple, i),
PyTuple_GET_ITEM(ann_tuple, i + 1));

if (err < 0)
return NULL;
}
Py_SETREF(op->func_annotations, ann_dict);
}
Py_INCREF(op->func_annotations);
return op->func_annotations;
}
Expand Down
2 changes: 1 addition & 1 deletion Python/ceval.c
Original file line number Diff line number Diff line change
Expand Up @@ -3880,7 +3880,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, PyFrameObject *f, int throwflag)
func ->func_closure = POP();
}
if (oparg & 0x04) {
assert(PyDict_CheckExact(TOP()));
assert(PyTuple_CheckExact(TOP()));
func->func_annotations = POP();
}
if (oparg & 0x02) {
Expand Down
70 changes: 27 additions & 43 deletions Python/compile.c
Original file line number Diff line number Diff line change
Expand Up @@ -2027,26 +2027,24 @@ compiler_visit_annexpr(struct compiler *c, expr_ty annotation)

static int
compiler_visit_argannotation(struct compiler *c, identifier id,
expr_ty annotation, PyObject *names)
expr_ty annotation, Py_ssize_t *annotations_len)
{
if (annotation) {
PyObject *mangled;
VISIT(c, annexpr, annotation);
mangled = _Py_Mangle(c->u->u_private, id);
PyObject *mangled = _Py_Mangle(c->u->u_private, id);
if (!mangled)
return 0;
if (PyList_Append(names, mangled) < 0) {
Py_DECREF(mangled);
return 0;
}

ADDOP_LOAD_CONST(c, mangled);
Py_DECREF(mangled);
VISIT(c, annexpr, annotation);
*annotations_len += 2;
}
return 1;
}

static int
compiler_visit_argannotations(struct compiler *c, asdl_arg_seq* args,
PyObject *names)
Py_ssize_t *annotations_len)
{
int i;
for (i = 0; i < asdl_seq_LEN(args); i++) {
Expand All @@ -2055,7 +2053,7 @@ compiler_visit_argannotations(struct compiler *c, asdl_arg_seq* args,
c,
arg->arg,
arg->annotation,
names))
annotations_len))
return 0;
}
return 1;
Expand All @@ -2065,58 +2063,44 @@ static int
compiler_visit_annotations(struct compiler *c, arguments_ty args,
expr_ty returns)
{
/* Push arg annotation dict.
/* Push arg annotation names and values.
The expressions are evaluated out-of-order wrt the source code.

Return 0 on error, -1 if no dict pushed, 1 if a dict is pushed.
Return 0 on error, -1 if no annotations pushed, 1 if a annotations is pushed.
*/
static identifier return_str;
PyObject *names;
Py_ssize_t len;
names = PyList_New(0);
if (!names)
return 0;
Py_ssize_t annotations_len = 0;

if (!compiler_visit_argannotations(c, args->args, names))
goto error;
if (!compiler_visit_argannotations(c, args->posonlyargs, names))
goto error;
if (!compiler_visit_argannotations(c, args->args, &annotations_len))
return 0;
if (!compiler_visit_argannotations(c, args->posonlyargs, &annotations_len))
return 0;
if (args->vararg && args->vararg->annotation &&
!compiler_visit_argannotation(c, args->vararg->arg,
args->vararg->annotation, names))
goto error;
if (!compiler_visit_argannotations(c, args->kwonlyargs, names))
goto error;
args->vararg->annotation, &annotations_len))
return 0;
if (!compiler_visit_argannotations(c, args->kwonlyargs, &annotations_len))
return 0;
if (args->kwarg && args->kwarg->annotation &&
!compiler_visit_argannotation(c, args->kwarg->arg,
args->kwarg->annotation, names))
goto error;
args->kwarg->annotation, &annotations_len))
return 0;

if (!return_str) {
return_str = PyUnicode_InternFromString("return");
if (!return_str)
goto error;
return 0;
}
if (!compiler_visit_argannotation(c, return_str, returns, names)) {
goto error;
if (!compiler_visit_argannotation(c, return_str, returns, &annotations_len)) {
return 0;
}

len = PyList_GET_SIZE(names);
if (len) {
PyObject *keytuple = PyList_AsTuple(names);
Py_DECREF(names);
ADDOP_LOAD_CONST_NEW(c, keytuple);
ADDOP_I(c, BUILD_CONST_KEY_MAP, len);
if (annotations_len) {
ADDOP_I(c, BUILD_TUPLE, annotations_len);
return 1;
}
else {
Py_DECREF(names);
return -1;
}

error:
Py_DECREF(names);
return 0;
return -1;
}

static int
Expand Down
Loading