Skip to content

Commit f93a54c

Browse files
gvanrossumgvanrossum
committed
Merge branch 'master' into fix-traceback-syntax-error
2 parents 31641ff + 16ab070 commit f93a54c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+1118
-1793
lines changed

‎.github/workflows/build.yml

+24
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,27 @@ on:
1616
- 3.7
1717

1818
jobs:
19+
check_source:
20+
name: 'Check for source changes'
21+
runs-on: ubuntu-latest
22+
outputs:
23+
run_tests: ${{ steps.check.outputs.run_tests }}
24+
steps:
25+
- uses: actions/checkout@v2
26+
- name: Check for source changes
27+
id: check
28+
run: |
29+
if [ -z "GITHUB_BASE_REF" ]; then
30+
echo '::set-output name=run_tests::true'
31+
else
32+
git fetch origin $GITHUB_BASE_REF --depth=1
33+
git diff --name-only origin/$GITHUB_BASE_REF... | grep -qvE '(\.rst$|^Doc|^Misc)' && echo '::set-output name=run_tests::true'
34+
fi
1935
build_win32:
2036
name: 'Windows (x86)'
2137
runs-on: windows-latest
38+
needs: check_source
39+
if: needs.check_source.outputs.run_tests == 'true'
2240
steps:
2341
- uses: actions/checkout@v1
2442
- name: Build CPython
@@ -31,6 +49,8 @@ jobs:
3149
build_win_amd64:
3250
name: 'Windows (x64)'
3351
runs-on: windows-latest
52+
needs: check_source
53+
if: needs.check_source.outputs.run_tests == 'true'
3454
steps:
3555
- uses: actions/checkout@v1
3656
- name: Build CPython
@@ -43,6 +63,8 @@ jobs:
4363
build_macos:
4464
name: 'macOS'
4565
runs-on: macos-latest
66+
needs: check_source
67+
if: needs.check_source.outputs.run_tests == 'true'
4668
steps:
4769
- uses: actions/checkout@v1
4870
- name: Configure CPython
@@ -57,6 +79,8 @@ jobs:
5779
build_ubuntu:
5880
name: 'Ubuntu'
5981
runs-on: ubuntu-latest
82+
needs: check_source
83+
if: needs.check_source.outputs.run_tests == 'true'
6084
env:
6185
OPENSSL_VER: 1.1.1f
6286
steps:

‎Doc/library/code.rst

+2-2
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,8 @@ build applications which provide an interactive interpreter prompt.
5656

5757
*source* is the source string; *filename* is the optional filename from which
5858
source was read, defaulting to ``'<input>'``; and *symbol* is the optional
59-
grammar start symbol, which should be either ``'single'`` (the default) or
60-
``'eval'``.
59+
grammar start symbol, which should be ``'single'`` (the default), ``'eval'``
60+
or ``'exec'``.
6161

6262
Returns a code object (the same as ``compile(source, filename, symbol)``) if the
6363
command is complete and valid; ``None`` if the command is incomplete; raises

‎Doc/library/codeop.rst

+3-2
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,9 @@ To do just the former:
4343
:exc:`OverflowError` or :exc:`ValueError` if there is an invalid literal.
4444

4545
The *symbol* argument determines whether *source* is compiled as a statement
46-
(``'single'``, the default) or as an :term:`expression` (``'eval'``). Any
47-
other value will cause :exc:`ValueError` to be raised.
46+
(``'single'``, the default), as a sequence of statements (``'exec'``) or
47+
as an :term:`expression` (``'eval'``). Any other value will
48+
cause :exc:`ValueError` to be raised.
4849

4950
.. note::
5051

‎Doc/library/compileall.rst

+16-5
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,11 @@ compile Python sources.
113113

114114
Ignore symlinks pointing outside the given directory.
115115

116+
.. cmdoption:: --hardlink-dupes
117+
118+
If two ``.pyc`` files with different optimization level have
119+
the same content, use hard links to consolidate duplicate files.
120+
116121
.. versionchanged:: 3.2
117122
Added the ``-i``, ``-b`` and ``-h`` options.
118123

@@ -125,7 +130,7 @@ compile Python sources.
125130
Added the ``--invalidation-mode`` option.
126131

127132
.. versionchanged:: 3.9
128-
Added the ``-s``, ``-p``, ``-e`` options.
133+
Added the ``-s``, ``-p``, ``-e`` and ``--hardlink-dupes`` options.
129134
Raised the default recursion limit from 10 to
130135
:py:func:`sys.getrecursionlimit()`.
131136
Added the possibility to specify the ``-o`` option multiple times.
@@ -143,7 +148,7 @@ runtime.
143148
Public functions
144149
----------------
145150

146-
.. function:: compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, \*, stripdir=None, prependdir=None, limit_sl_dest=None)
151+
.. function:: compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, \*, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)
147152

148153
Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
149154
files along the way. Return a true value if all the files compiled successfully,
@@ -193,6 +198,9 @@ Public functions
193198
the ``-s``, ``-p`` and ``-e`` options described above.
194199
They may be specified as ``str``, ``bytes`` or :py:class:`os.PathLike`.
195200

201+
If *hardlink_dupes* is true and two ``.pyc`` files with different optimization
202+
level have the same content, use hard links to consolidate duplicate files.
203+
196204
.. versionchanged:: 3.2
197205
Added the *legacy* and *optimize* parameter.
198206

@@ -219,9 +227,9 @@ Public functions
219227
Setting *workers* to 0 now chooses the optimal number of cores.
220228

221229
.. versionchanged:: 3.9
222-
Added *stripdir*, *prependdir* and *limit_sl_dest* arguments.
230+
Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments.
223231

224-
.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, \*, stripdir=None, prependdir=None, limit_sl_dest=None)
232+
.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, \*, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)
225233

226234
Compile the file with path *fullname*. Return a true value if the file
227235
compiled successfully, and a false value otherwise.
@@ -257,6 +265,9 @@ Public functions
257265
the ``-s``, ``-p`` and ``-e`` options described above.
258266
They may be specified as ``str``, ``bytes`` or :py:class:`os.PathLike`.
259267

268+
If *hardlink_dupes* is true and two ``.pyc`` files with different optimization
269+
level have the same content, use hard links to consolidate duplicate files.
270+
260271
.. versionadded:: 3.2
261272

262273
.. versionchanged:: 3.5
@@ -273,7 +284,7 @@ Public functions
273284
The *invalidation_mode* parameter's default value is updated to None.
274285

275286
.. versionchanged:: 3.9
276-
Added *stripdir*, *prependdir* and *limit_sl_dest* arguments.
287+
Added *stripdir*, *prependdir*, *limit_sl_dest* and *hardlink_dupes* arguments.
277288

278289
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=None)
279290

‎Doc/whatsnew/3.9.rst

+13
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,16 @@ that schedules a shutdown for the default executor that waits on the
245245
Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher
246246
implementation that polls process file descriptors. (:issue:`38692`)
247247

248+
compileall
249+
----------
250+
251+
Added new possibility to use hardlinks for duplicated ``.pyc`` files: *hardlink_dupes* parameter and --hardlink-dupes command line option.
252+
(Contributed by Lumír 'Frenzy' Balhar in :issue:`40495`.)
253+
254+
Added new options for path manipulation in resulting ``.pyc`` files: *stripdir*, *prependdir*, *limit_sl_dest* parameters and -s, -p, -e command line options.
255+
Added the possibility to specify the option for an optimization level multiple times.
256+
(Contributed by Lumír 'Frenzy' Balhar in :issue:`38112`.)
257+
248258
concurrent.futures
249259
------------------
250260

@@ -964,3 +974,6 @@ Removed
964974
* ``PyTuple_ClearFreeList()``
965975
* ``PyUnicode_ClearFreeList()``: the Unicode free list has been removed in
966976
Python 3.3.
977+
978+
* Remove ``_PyUnicode_ClearStaticStrings()`` function.
979+
(Contributed by Victor Stinner in :issue:`39465`.)

‎Grammar/python.gram

+19-11
Original file line numberDiff line numberDiff line change
@@ -89,12 +89,12 @@ assignment[stmt_ty]:
8989
"Variable annotation syntax is",
9090
_Py_AnnAssign(CHECK(_PyPegen_set_expr_context(p, a, Store)), b, c, 1, EXTRA)
9191
) }
92-
| a=('(' b=inside_paren_ann_assign_target ')' { b }
93-
| ann_assign_subscript_attribute_target) ':' b=expression c=['=' d=annotated_rhs { d }] {
92+
| a=('(' b=single_target ')' { b }
93+
| single_subscript_attribute_target) ':' b=expression c=['=' d=annotated_rhs { d }] {
9494
CHECK_VERSION(6, "Variable annotations syntax is", _Py_AnnAssign(a, b, c, 0, EXTRA)) }
9595
| a=(z=star_targets '=' { z })+ b=(yield_expr | star_expressions) tc=[TYPE_COMMENT] {
9696
_Py_Assign(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA) }
97-
| a=target b=augassign c=(yield_expr | star_expressions) {
97+
| a=single_target b=augassign c=(yield_expr | star_expressions) {
9898
_Py_AugAssign(a, b->kind, c, EXTRA) }
9999
| invalid_assignment
100100

@@ -185,7 +185,7 @@ try_stmt[stmt_ty]:
185185
| 'try' ':' b=block f=finally_block { _Py_Try(b, NULL, NULL, f, EXTRA) }
186186
| 'try' ':' b=block ex=except_block+ el=[else_block] f=[finally_block] { _Py_Try(b, ex, el, f, EXTRA) }
187187
except_block[excepthandler_ty]:
188-
| 'except' e=expression t=['as' z=target { z }] ':' b=block {
188+
| 'except' e=expression t=['as' z=NAME { z }] ':' b=block {
189189
_Py_ExceptHandler(e, (t) ? ((expr_ty) t)->v.Name.id : NULL, b, EXTRA) }
190190
| 'except' ':' b=block { _Py_ExceptHandler(NULL, NULL, b, EXTRA) }
191191
finally_block[asdl_seq*]: 'finally' ':' a=block { a }
@@ -573,12 +573,11 @@ star_atom[expr_ty]:
573573
| '(' a=[star_targets_seq] ')' { _Py_Tuple(a, Store, EXTRA) }
574574
| '[' a=[star_targets_seq] ']' { _Py_List(a, Store, EXTRA) }
575575

576-
inside_paren_ann_assign_target[expr_ty]:
577-
| ann_assign_subscript_attribute_target
576+
single_target[expr_ty]:
577+
| single_subscript_attribute_target
578578
| a=NAME { _PyPegen_set_expr_context(p, a, Store) }
579-
| '(' a=inside_paren_ann_assign_target ')' { a }
580-
581-
ann_assign_subscript_attribute_target[expr_ty]:
579+
| '(' a=single_target ')' { a }
580+
single_subscript_attribute_target[expr_ty]:
582581
| a=t_primary '.' b=NAME !t_lookahead { _Py_Attribute(a, b->v.Name.id, Store, EXTRA) }
583582
| a=t_primary '[' b=slices ']' !t_lookahead { _Py_Subscript(a, b, Store, EXTRA) }
584583

@@ -641,8 +640,17 @@ invalid_assignment:
641640
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "only single target (not tuple) can be annotated") }
642641
| a=expression ':' expression ['=' annotated_rhs] {
643642
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "illegal target for annotation") }
644-
| a=expression ('=' | augassign) (yield_expr | star_expressions) {
645-
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot assign to %s", _PyPegen_get_expr_name(a)) }
643+
| a=star_expressions '=' (yield_expr | star_expressions) {
644+
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(
645+
_PyPegen_get_invalid_target(a),
646+
"cannot assign to %s", _PyPegen_get_expr_name(_PyPegen_get_invalid_target(a))) }
647+
| a=star_expressions augassign (yield_expr | star_expressions) {
648+
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(
649+
a,
650+
"'%s' is an illegal expression for augmented assignment",
651+
_PyPegen_get_expr_name(a)
652+
)}
653+
646654
invalid_block:
647655
| NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block") }
648656
invalid_comprehension:

‎Include/cpython/object.h

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ PyAPI_FUNC(Py_ssize_t) _Py_GetRefTotal(void);
3636
3737
PyId_foo is a static variable, either on block level or file level. On first
3838
usage, the string "foo" is interned, and the structures are linked. On interpreter
39-
shutdown, all strings are released (through _PyUnicode_ClearStaticStrings).
39+
shutdown, all strings are released.
4040
4141
Alternatively, _Py_static_string allows choosing the variable name.
4242
_PyUnicode_FromId returns a borrowed reference to the interned string.

‎Include/cpython/unicodeobject.h

-2
Original file line numberDiff line numberDiff line change
@@ -1215,8 +1215,6 @@ Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE*) PyUnicode_AsUnicodeCopy(
12151215

12161216
/* Return an interned Unicode object for an Identifier; may fail if there is no memory.*/
12171217
PyAPI_FUNC(PyObject*) _PyUnicode_FromId(_Py_Identifier*);
1218-
/* Clear all static strings. */
1219-
PyAPI_FUNC(void) _PyUnicode_ClearStaticStrings(void);
12201218

12211219
/* Fast equality check when the inputs are known to be exact unicode types
12221220
and where the hash values are equal (i.e. a very probable match) */

‎Include/internal/pycore_hashtable.h

+12-11
Original file line numberDiff line numberDiff line change
@@ -48,18 +48,18 @@ typedef _Py_hashtable_entry_t* (*_Py_hashtable_get_entry_func)(_Py_hashtable_t *
4848
const void *key);
4949

5050
typedef struct {
51-
/* allocate a memory block */
51+
// Allocate a memory block
5252
void* (*malloc) (size_t size);
5353

54-
/* release a memory block */
54+
// Release a memory block
5555
void (*free) (void *ptr);
5656
} _Py_hashtable_allocator_t;
5757

5858

5959
/* _Py_hashtable: table */
6060
struct _Py_hashtable_t {
61-
size_t num_buckets;
62-
size_t entries; /* Total number of entries in the table. */
61+
size_t nentries; // Total number of entries in the table
62+
size_t nbuckets;
6363
_Py_slist_t *buckets;
6464

6565
_Py_hashtable_get_entry_func get_entry_func;
@@ -70,10 +70,10 @@ struct _Py_hashtable_t {
7070
_Py_hashtable_allocator_t alloc;
7171
};
7272

73-
/* hash a pointer (void*) */
73+
/* Hash a pointer (void*) */
7474
PyAPI_FUNC(Py_uhash_t) _Py_hashtable_hash_ptr(const void *key);
7575

76-
/* comparison using memcmp() */
76+
/* Comparison using memcmp() */
7777
PyAPI_FUNC(int) _Py_hashtable_compare_direct(
7878
const void *key1,
7979
const void *key2);
@@ -129,13 +129,14 @@ _Py_hashtable_get_entry(_Py_hashtable_t *ht, const void *key)
129129
130130
Use _Py_hashtable_get_entry() to distinguish entry value equal to NULL
131131
and entry not found. */
132-
extern void *_Py_hashtable_get(_Py_hashtable_t *ht, const void *key);
132+
PyAPI_FUNC(void*) _Py_hashtable_get(_Py_hashtable_t *ht, const void *key);
133133

134134

135-
// Remove a key and its associated value without calling key and value destroy
136-
// functions.
137-
// Return the removed value if the key was found.
138-
// Return NULL if the key was not found.
135+
/* Remove a key and its associated value without calling key and value destroy
136+
functions.
137+
138+
Return the removed value if the key was found.
139+
Return NULL if the key was not found. */
139140
PyAPI_FUNC(void*) _Py_hashtable_steal(
140141
_Py_hashtable_t *ht,
141142
const void *key);

‎Include/internal/pycore_interp.h

+14-8
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,19 @@ struct _ceval_state {
5151
#endif
5252
};
5353

54+
/* fs_codec.encoding is initialized to NULL.
55+
Later, it is set to a non-NULL string by _PyUnicode_InitEncodings(). */
56+
struct _Py_unicode_fs_codec {
57+
char *encoding; // Filesystem encoding (encoded to UTF-8)
58+
int utf8; // encoding=="utf-8"?
59+
char *errors; // Filesystem errors (encoded to UTF-8)
60+
_Py_error_handler error_handler;
61+
};
62+
63+
struct _Py_unicode_state {
64+
struct _Py_unicode_fs_codec fs_codec;
65+
};
66+
5467

5568
/* interpreter state */
5669

@@ -97,14 +110,7 @@ struct _is {
97110
PyObject *codec_error_registry;
98111
int codecs_initialized;
99112

100-
/* fs_codec.encoding is initialized to NULL.
101-
Later, it is set to a non-NULL string by _PyUnicode_InitEncodings(). */
102-
struct {
103-
char *encoding; /* Filesystem encoding (encoded to UTF-8) */
104-
int utf8; /* encoding=="utf-8"? */
105-
char *errors; /* Filesystem errors (encoded to UTF-8) */
106-
_Py_error_handler error_handler;
107-
} fs_codec;
113+
struct _Py_unicode_state unicode;
108114

109115
PyConfig config;
110116
#ifdef HAVE_DLOPEN

‎Lib/codeop.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,8 @@ def compile_command(source, filename="<input>", symbol="single"):
112112
source -- the source string; may contain \n characters
113113
filename -- optional filename from which source was read; default
114114
"<input>"
115-
symbol -- optional grammar start symbol; "single" (default) or "eval"
115+
symbol -- optional grammar start symbol; "single" (default), "exec"
116+
or "eval"
116117
117118
Return value / exceptions raised:
118119

0 commit comments

Comments
 (0)