Skip to content

Commit 1503ae7

Browse files
Merge branch 'main' into htmlparser-comment
2 parents c8d9740 + b582d75 commit 1503ae7

17 files changed

+431
-262
lines changed

Doc/library/compression.zstd.rst

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -523,8 +523,14 @@ Advanced parameter control
523523
.. attribute:: compression_level
524524

525525
A high-level means of setting other compression parameters that affect
526-
the speed and ratio of compressing data. Setting the level to zero uses
527-
:attr:`COMPRESSION_LEVEL_DEFAULT`.
526+
the speed and ratio of compressing data.
527+
528+
Regular compression levels are greater than ``0``. Values greater than
529+
``20`` are considered "ultra" compression and require more memory than
530+
other levels. Negative values can be used to trade off faster compression
531+
for worse compression ratios.
532+
533+
Setting the level to zero uses :attr:`COMPRESSION_LEVEL_DEFAULT`.
528534

529535
.. attribute:: window_log
530536

Doc/library/io.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -719,6 +719,9 @@ than raw I/O does.
719719
The optional argument *initial_bytes* is a :term:`bytes-like object` that
720720
contains initial data.
721721

722+
Methods may be used from multiple threads without external locking in
723+
:term:`free threading` builds.
724+
722725
:class:`BytesIO` provides or overrides these methods in addition to those
723726
from :class:`BufferedIOBase` and :class:`IOBase`:
724727

Doc/library/shutil.rst

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -618,7 +618,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
618618
*format* is the archive format: one of
619619
"zip" (if the :mod:`zlib` module is available), "tar", "gztar" (if the
620620
:mod:`zlib` module is available), "bztar" (if the :mod:`bz2` module is
621-
available), or "xztar" (if the :mod:`lzma` module is available).
621+
available), "xztar" (if the :mod:`lzma` module is available), or "zstdtar"
622+
(if the :mod:`compression.zstd` module is available).
622623

623624
*root_dir* is a directory that will be the root directory of the
624625
archive, all paths in the archive will be relative to it; for example,
@@ -673,6 +674,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
673674
- *gztar*: gzip'ed tar-file (if the :mod:`zlib` module is available).
674675
- *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available).
675676
- *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available).
677+
- *zstdtar*: Zstandard compressed tar-file (if the :mod:`compression.zstd`
678+
module is available).
676679

677680
You can register new formats or provide your own archiver for any existing
678681
formats, by using :func:`register_archive_format`.
@@ -716,8 +719,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
716719
*extract_dir* is the name of the target directory where the archive is
717720
unpacked. If not provided, the current working directory is used.
718721

719-
*format* is the archive format: one of "zip", "tar", "gztar", "bztar", or
720-
"xztar". Or any other format registered with
722+
*format* is the archive format: one of "zip", "tar", "gztar", "bztar",
723+
"xztar", or "zstdtar". Or any other format registered with
721724
:func:`register_unpack_format`. If not provided, :func:`unpack_archive`
722725
will use the archive file name extension and see if an unpacker was
723726
registered for that extension. In case none is found,
@@ -789,6 +792,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
789792
- *gztar*: gzip'ed tar-file (if the :mod:`zlib` module is available).
790793
- *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available).
791794
- *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available).
795+
- *zstdtar*: Zstandard compressed tar-file (if the :mod:`compression.zstd`
796+
module is available).
792797

793798
You can register new formats or provide your own unpacker for any existing
794799
formats, by using :func:`register_unpack_format`.

Doc/library/sys.monitoring.rst

Lines changed: 36 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,8 @@ The following events are supported:
137137

138138
.. monitoring-event:: PY_UNWIND
139139

140-
Exit from a Python function during exception unwinding.
140+
Exit from a Python function during exception unwinding. This includes exceptions raised directly within the
141+
function and that are allowed to continue to propagate.
141142

142143
.. monitoring-event:: PY_YIELD
143144

@@ -171,7 +172,7 @@ events, use the expression ``PY_RETURN | PY_START``.
171172
if get_events(DEBUGGER_ID) == NO_EVENTS:
172173
...
173174

174-
Events are divided into three groups:
175+
Setting this event deactivates all events.
175176

176177
.. _monitoring-event-local:
177178

@@ -243,20 +244,23 @@ raise an exception unless it would be visible to other code.
243244

244245
To allow tools to monitor for real exceptions without slowing down generators
245246
and coroutines, the :monitoring-event:`STOP_ITERATION` event is provided.
246-
:monitoring-event:`STOP_ITERATION` can be locally disabled, unlike :monitoring-event:`RAISE`.
247+
:monitoring-event:`STOP_ITERATION` can be locally disabled, unlike
248+
:monitoring-event:`RAISE`.
247249

248-
Note that the :monitoring-event:`STOP_ITERATION` event and the :monitoring-event:`RAISE`
249-
event for a :exc:`StopIteration` exception are equivalent, and are treated as interchangeable
250-
when generating events. Implementations will favor :monitoring-event:`STOP_ITERATION` for
251-
performance reasons, but may generate a :monitoring-event:`RAISE` event with a :exc:`StopIteration`.
250+
Note that the :monitoring-event:`STOP_ITERATION` event and the
251+
:monitoring-event:`RAISE` event for a :exc:`StopIteration` exception are
252+
equivalent, and are treated as interchangeable when generating events.
253+
Implementations will favor :monitoring-event:`STOP_ITERATION` for performance
254+
reasons, but may generate a :monitoring-event:`RAISE` event with a
255+
:exc:`StopIteration`.
252256

253257
Turning events on and off
254258
-------------------------
255259

256260
In order to monitor an event, it must be turned on and a corresponding callback
257-
must be registered.
258-
Events can be turned on or off by setting the events either globally or
259-
for a particular code object.
261+
must be registered. Events can be turned on or off by setting the events either
262+
globally and/or for a particular code object. An event will trigger only once,
263+
even if it is turned on both globally and locally.
260264

261265

262266
Setting events globally
@@ -292,10 +296,6 @@ in Python (see :ref:`c-api-monitoring`).
292296
Activates all the local events for *code* which are set in *event_set*.
293297
Raises a :exc:`ValueError` if *tool_id* is not in use.
294298

295-
Local events add to global events, but do not mask them.
296-
In other words, all global events will trigger for a code object,
297-
regardless of the local events.
298-
299299

300300
Disabling events
301301
''''''''''''''''
@@ -325,8 +325,6 @@ except for a few breakpoints.
325325
Registering callback functions
326326
------------------------------
327327

328-
To register a callable for events call
329-
330328
.. function:: register_callback(tool_id: int, event: int, func: Callable | None, /) -> Callable | None
331329

332330
Registers the callable *func* for the *event* with the given *tool_id*
@@ -335,12 +333,16 @@ To register a callable for events call
335333
it is unregistered and returned.
336334
Otherwise :func:`register_callback` returns ``None``.
337335

338-
339336
Functions can be unregistered by calling
340337
``sys.monitoring.register_callback(tool_id, event, None)``.
341338

342339
Callback functions can be registered and unregistered at any time.
343340

341+
Callbacks are called only once regardless if the event is turned on both
342+
globally and locally. As such, if an event could be turned on for both global
343+
and local events by your code then the callback needs to be written to handle
344+
either trigger.
345+
344346
Registering or unregistering a callback function will generate a :func:`sys.audit` event.
345347

346348

@@ -353,37 +355,46 @@ Callback function arguments
353355
that there are no arguments to the call.
354356

355357
When an active event occurs, the registered callback function is called.
358+
Callback functions returning an object other than :data:`DISABLE` will have no effect.
356359
Different events will provide the callback function with different arguments, as follows:
357360

358361
* :monitoring-event:`PY_START` and :monitoring-event:`PY_RESUME`::
359362

360-
func(code: CodeType, instruction_offset: int) -> DISABLE | Any
363+
func(code: CodeType, instruction_offset: int) -> object
361364

362365
* :monitoring-event:`PY_RETURN` and :monitoring-event:`PY_YIELD`::
363366

364-
func(code: CodeType, instruction_offset: int, retval: object) -> DISABLE | Any
367+
func(code: CodeType, instruction_offset: int, retval: object) -> object
365368

366-
* :monitoring-event:`CALL`, :monitoring-event:`C_RAISE` and :monitoring-event:`C_RETURN`::
369+
* :monitoring-event:`CALL`, :monitoring-event:`C_RAISE` and :monitoring-event:`C_RETURN`
370+
(*arg0* can be :data:`MISSING` specifically)::
367371

368-
func(code: CodeType, instruction_offset: int, callable: object, arg0: object | MISSING) -> DISABLE | Any
372+
func(code: CodeType, instruction_offset: int, callable: object, arg0: object) -> object
369373

374+
*code* represents the code object where the call is being made, while
375+
*callable* is the object that is about to be called (and thus
376+
triggered the event).
370377
If there are no arguments, *arg0* is set to :data:`sys.monitoring.MISSING`.
371378

379+
For instance methods, *callable* will be the function object as found on the
380+
class with *arg0* set to the instance (i.e. the ``self`` argument to the
381+
method).
382+
372383
* :monitoring-event:`RAISE`, :monitoring-event:`RERAISE`, :monitoring-event:`EXCEPTION_HANDLED`,
373384
:monitoring-event:`PY_UNWIND`, :monitoring-event:`PY_THROW` and :monitoring-event:`STOP_ITERATION`::
374385

375-
func(code: CodeType, instruction_offset: int, exception: BaseException) -> DISABLE | Any
386+
func(code: CodeType, instruction_offset: int, exception: BaseException) -> object
376387

377388
* :monitoring-event:`LINE`::
378389

379-
func(code: CodeType, line_number: int) -> DISABLE | Any
390+
func(code: CodeType, line_number: int) -> object
380391

381392
* :monitoring-event:`BRANCH_LEFT`, :monitoring-event:`BRANCH_RIGHT` and :monitoring-event:`JUMP`::
382393

383-
func(code: CodeType, instruction_offset: int, destination_offset: int) -> DISABLE | Any
394+
func(code: CodeType, instruction_offset: int, destination_offset: int) -> object
384395

385396
Note that the *destination_offset* is where the code will next execute.
386397

387398
* :monitoring-event:`INSTRUCTION`::
388399

389-
func(code: CodeType, instruction_offset: int) -> DISABLE | Any
400+
func(code: CodeType, instruction_offset: int) -> object

Doc/library/tarfile.rst

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@ higher-level functions in :ref:`shutil <archiving-operations>`.
1818

1919
Some facts and figures:
2020

21-
* reads and writes :mod:`gzip`, :mod:`bz2` and :mod:`lzma` compressed archives
22-
if the respective modules are available.
21+
* reads and writes :mod:`gzip`, :mod:`bz2`, :mod:`compression.zstd`, and
22+
:mod:`lzma` compressed archives if the respective modules are available.
2323

2424
* read/write support for the POSIX.1-1988 (ustar) format.
2525

@@ -47,6 +47,10 @@ Some facts and figures:
4747
or paths outside of the destination. Previously, the filter strategy
4848
was equivalent to :func:`fully_trusted <fully_trusted_filter>`.
4949

50+
.. versionchanged:: 3.14
51+
52+
Added support for Zstandard compression using :mod:`compression.zstd`.
53+
5054
.. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, **kwargs)
5155

5256
Return a :class:`TarFile` object for the pathname *name*. For detailed
@@ -71,6 +75,8 @@ Some facts and figures:
7175
+------------------+---------------------------------------------+
7276
| ``'r:xz'`` | Open for reading with lzma compression. |
7377
+------------------+---------------------------------------------+
78+
| ``'r:zst'`` | Open for reading with Zstandard compression.|
79+
+------------------+---------------------------------------------+
7480
| ``'x'`` or | Create a tarfile exclusively without |
7581
| ``'x:'`` | compression. |
7682
| | Raise a :exc:`FileExistsError` exception |
@@ -88,6 +94,10 @@ Some facts and figures:
8894
| | Raise a :exc:`FileExistsError` exception |
8995
| | if it already exists. |
9096
+------------------+---------------------------------------------+
97+
| ``'x:zst'`` | Create a tarfile with Zstandard compression.|
98+
| | Raise a :exc:`FileExistsError` exception |
99+
| | if it already exists. |
100+
+------------------+---------------------------------------------+
91101
| ``'a' or 'a:'`` | Open for appending with no compression. The |
92102
| | file is created if it does not exist. |
93103
+------------------+---------------------------------------------+
@@ -99,6 +109,8 @@ Some facts and figures:
99109
+------------------+---------------------------------------------+
100110
| ``'w:xz'`` | Open for lzma compressed writing. |
101111
+------------------+---------------------------------------------+
112+
| ``'w:zst'`` | Open for Zstandard compressed writing. |
113+
+------------------+---------------------------------------------+
102114

103115
Note that ``'a:gz'``, ``'a:bz2'`` or ``'a:xz'`` is not possible. If *mode*
104116
is not suitable to open a certain (compressed) file for reading,
@@ -115,6 +127,15 @@ Some facts and figures:
115127
For modes ``'w:xz'``, ``'x:xz'`` and ``'w|xz'``, :func:`tarfile.open` accepts the
116128
keyword argument *preset* to specify the compression level of the file.
117129

130+
For modes ``'w:zst'``, ``'x:zst'`` and ``'w|zst'``, :func:`tarfile.open`
131+
accepts the keyword argument *level* to specify the compression level of
132+
the file. The keyword argument *options* may also be passed, providing
133+
advanced Zstandard compression parameters described by
134+
:class:`~compression.zstd.CompressionParameter`. The keyword argument
135+
*zstd_dict* can be passed to provide a :class:`~compression.zstd.ZstdDict`,
136+
a Zstandard dictionary used to improve compression of smaller amounts of
137+
data.
138+
118139
For special purposes, there is a second format for *mode*:
119140
``'filemode|[compression]'``. :func:`tarfile.open` will return a :class:`TarFile`
120141
object that processes its data as a stream of blocks. No random seeking will
@@ -146,6 +167,9 @@ Some facts and figures:
146167
| ``'r|xz'`` | Open an lzma compressed *stream* for |
147168
| | reading. |
148169
+-------------+--------------------------------------------+
170+
| ``'r|zst'`` | Open a Zstandard compressed *stream* for |
171+
| | reading. |
172+
+-------------+--------------------------------------------+
149173
| ``'w|'`` | Open an uncompressed *stream* for writing. |
150174
+-------------+--------------------------------------------+
151175
| ``'w|gz'`` | Open a gzip compressed *stream* for |
@@ -157,6 +181,9 @@ Some facts and figures:
157181
| ``'w|xz'`` | Open an lzma compressed *stream* for |
158182
| | writing. |
159183
+-------------+--------------------------------------------+
184+
| ``'w|zst'`` | Open a Zstandard compressed *stream* for |
185+
| | writing. |
186+
+-------------+--------------------------------------------+
160187

161188
.. versionchanged:: 3.5
162189
The ``'x'`` (exclusive creation) mode was added.

Doc/library/zipfile.rst

Lines changed: 32 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -129,14 +129,28 @@ The module defines the following items:
129129

130130
.. versionadded:: 3.3
131131

132+
.. data:: ZIP_ZSTANDARD
133+
134+
The numeric constant for Zstandard compression. This requires the
135+
:mod:`compression.zstd` module.
136+
132137
.. note::
133138

134-
The ZIP file format specification has included support for bzip2 compression
135-
since 2001, and for LZMA compression since 2006. However, some tools
136-
(including older Python releases) do not support these compression
137-
methods, and may either refuse to process the ZIP file altogether,
138-
or fail to extract individual files.
139+
In APPNOTE 6.3.7, the method ID ``20`` was assigned to Zstandard
140+
compression. This was changed in APPNOTE 6.3.8 to method ID ``93`` to
141+
avoid conflicts, with method ID ``20`` being deprecated. For
142+
compatibility, the :mod:`!zipfile` module reads both method IDs but will
143+
only write data with method ID ``93``.
144+
145+
.. versionadded:: 3.14
146+
147+
.. note::
139148

149+
The ZIP file format specification has included support for bzip2 compression
150+
since 2001, for LZMA compression since 2006, and Zstandard compression since
151+
2020. However, some tools (including older Python releases) do not support
152+
these compression methods, and may either refuse to process the ZIP file
153+
altogether, or fail to extract individual files.
140154

141155
.. seealso::
142156

@@ -176,10 +190,11 @@ ZipFile Objects
176190

177191
*compression* is the ZIP compression method to use when writing the archive,
178192
and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`,
179-
:const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized
180-
values will cause :exc:`NotImplementedError` to be raised. If
181-
:const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified
182-
but the corresponding module (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not
193+
:const:`ZIP_BZIP2`, :const:`ZIP_LZMA`, or :const:`ZIP_ZSTANDARD`;
194+
unrecognized values will cause :exc:`NotImplementedError` to be raised. If
195+
:const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2`, :const:`ZIP_LZMA`, or
196+
:const:`ZIP_ZSTANDARD` is specified but the corresponding module
197+
(:mod:`zlib`, :mod:`bz2`, :mod:`lzma`, or :mod:`compression.zstd`) is not
183198
available, :exc:`RuntimeError` is raised. The default is :const:`ZIP_STORED`.
184199

185200
If *allowZip64* is ``True`` (the default) zipfile will create ZIP files that
@@ -194,6 +209,10 @@ ZipFile Objects
194209
(see :class:`zlib <zlib.compressobj>` for more information).
195210
When using :const:`ZIP_BZIP2` integers ``1`` through ``9`` are accepted
196211
(see :class:`bz2 <bz2.BZ2File>` for more information).
212+
When using :const:`ZIP_ZSTANDARD` integers ``-131072`` through ``22`` are
213+
commonly accepted (see
214+
:attr:`CompressionParameter.compression_level <compression.zstd.CompressionParameter.compression_level>`
215+
for more on retrieving valid values and their meaning).
197216

198217
The *strict_timestamps* argument, when set to ``False``, allows to
199218
zip files older than 1980-01-01 at the cost of setting the
@@ -415,9 +434,10 @@ ZipFile Objects
415434
read or append. *pwd* is the password used for encrypted files as a :class:`bytes`
416435
object and, if specified, overrides the default password set with :meth:`setpassword`.
417436
Calling :meth:`read` on a ZipFile that uses a compression method other than
418-
:const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or
419-
:const:`ZIP_LZMA` will raise a :exc:`NotImplementedError`. An error will also
420-
be raised if the corresponding compression module is not available.
437+
:const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2`,
438+
:const:`ZIP_LZMA`, or :const:`ZIP_ZSTANDARD` will raise a
439+
:exc:`NotImplementedError`. An error will also be raised if the
440+
corresponding compression module is not available.
421441

422442
.. versionchanged:: 3.6
423443
Calling :meth:`read` on a closed ZipFile will raise a :exc:`ValueError`.

0 commit comments

Comments
 (0)