IDLE has two main window types, the Shell window and the Editor window. It is
possible to have multiple editor windows simultaneously. On Windows and
Linux, each has its own top menu. Each menu documented below indicates
@@ -131,20 +232,20 @@
Reformat the current blank-line-delimited paragraph in comment block or
+multiline string or selected line in a string. All lines in the
+paragraph will be formatted to less than N columns, where N defaults to 72.
+
Indent Region
Shift selected lines right by the indent width (default 4 spaces).
Dedent Region
Shift selected lines left by the indent width (default 4 spaces).
Do Check Module. If no error, restart the shell to clean the
environment, then execute the module. Output is displayed in the Shell
@@ -278,9 +384,9 @@
Look on the current line. with the cursor, and the line above for a filename
and line number. If found, open the file if not already open, and show the
@@ -315,9 +421,9 @@
Open a configuration dialog and change preferences for the following:
fonts, indentation, keybindings, text color themes, startup windows and
@@ -347,14 +453,14 @@
IDLE may open editor windows when it starts, depending on settings
and how you start IDLE. Thereafter, use the File menu. There can be only
one open editor window for a given file.
After a block-opening statement, the next line is indented by 4 spaces (in the
Python Shell window by one tab). After certain keywords (break, return etc.)
the next line is dedented. In leading indentation, Backspace deletes up
@@ -463,9 +569,16 @@
Any selection becomes a search target. However, only selections within
+a line work because searches are only performed within lines with the
+terminal newline removed. If [x]Regularexpresion is checked, the
+target is interpreted according to the Python re module.
Completions are supplied, when requested and available, for module
names, attributes of classes or functions, or filenames. Each request
method displays a completion box with existing names. (See tab
@@ -487,7 +600,7 @@
A calltip is shown automatically when one types ( after the name
of an accessible function. A function name expression may include
dots and subscripts. A calltip remains until it is clicked, the cursor
@@ -533,9 +646,9 @@
Within an editor window containing Python code, code context can be toggled
in order to show or hide a pane at the top of the window. When shown, this
pane freezes the opening lines for block code, such as those beginning with
@@ -548,47 +661,63 @@
In IDLE’s Shell, enter, edit, and recall complete statements. (Most
+consoles and terminals only work with a single physical line at a time).
+
Submit a single-line statement for execution by hitting Return
+with the cursor anywhere on the line. If a line is extended with
+Backslash (\), the cursor must be on the last physical line.
+Submit a multi-line compound statement by entering a blank line after
+the statement.
When one pastes code into Shell, it is not compiled and possibly executed
-until one hits Return. One may edit pasted code first.
-If one pastes more that one statement into Shell, the result will be a
+until one hits Return, as specified above.
+One may edit pasted code first.
+If one pastes more than one statement into Shell, the result will be a
SyntaxError when multiple statements are compiled as if they were one.
+
Lines containing RESTART mean that the user execution process has been
+re-started. This occurs when the user execution process has crashed,
+when one requests a restart on the Shell menu, or when one runs code
+in an editor window.
The editing features described in previous subsections work when entering
code interactively. IDLE’s Shell window also responds to the following keys.
-
C-c interrupts executing command
-
C-d sends end-of-file; closes window if typed at a >>> prompt
-
Alt-/ (Expand word) is also useful to reduce typing
+
C-c interrupts executing command
+
C-d sends end-of-file; closes window if typed at a >>> prompt
+
Alt-/ (Expand word) is also useful to reduce typing
Command history
-
Alt-p retrieves previous command matching what you have typed. On
-macOS use C-p.
-
Alt-n retrieves next. On macOS use C-n.
-
Return while on any previous command retrieves that command
+
Alt-p retrieves previous command matching what you have typed. On
+macOS use C-p.
+
Alt-n retrieves next. On macOS use C-n.
+
Return while the cursor is on any previous command
+retrieves that command
Idle defaults to black on white text, but colors text with special meanings.
For the shell, these are shell output, shell error, user output, and
user error. For Python code, at the shell prompt or in an editor, these are
keywords, builtin class and function names, names following class and
def, strings, and comments. For any text window, these are the cursor (when
present), found text (when possible), and selected text.
+
IDLE also highlights the soft keywordsmatch,
+case, and _ in
+pattern-matching statements. However, this highlighting is not perfect and
+will be incorrect in some rare cases, including some _-s in case
+patterns.
Text coloring is done in the background, so uncolorized text is occasionally
visible. To change the color scheme, use the Configure IDLE dialog
Highlighting tab. The marking of debugger breakpoint lines in the editor and
text in popups and dialogs is not user-configurable.
Upon startup with the -s option, IDLE will execute the file referenced by
the environment variables IDLESTARTUP or PYTHONSTARTUP.
IDLE first checks for IDLESTARTUP; if IDLESTARTUP is present the file
@@ -601,8 +730,8 @@
IDLE uses a socket to communicate between the IDLE GUI process and the user
code execution process. A connection must be established whenever the Shell
starts or restarts. (The latter is indicated by a divider line that says
@@ -677,15 +806,15 @@
With rare exceptions, the result of executing Python code with IDLE is
intended to be the same as executing the same code by the default method,
directly with Python in a text-mode system console or terminal window.
However, the different interface and operation occasionally affect
visible results. For instance, sys.modules starts with more entries,
-and threading.activeCount() returns 2 instead of 1.
+and threading.active_count() returns 2 instead of 1.
By default, IDLE runs user code in a separate OS process rather than in
the user interface process that runs the shell and editor. In the execution
process, it replaces sys.stdin, sys.stdout, and sys.stderr
@@ -704,7 +833,9 @@
When a program outputs text, the result is determined by the
corresponding output device. When IDLE executes user code, sys.stdout
and sys.stderr are connected to the display area of IDLE’s Shell. Some of
@@ -773,9 +904,9 @@
IDLE is intentionally different from standard Python in order to
facilitate development of tkinter programs. Enter importtkinterastk;root=tk.Tk() in standard Python and nothing appears. Enter the same
@@ -793,12 +924,12 @@
By default, IDLE executes user code in a separate subprocess via a socket,
which uses the internal loopback interface. This connection is not
-externally visible and no data is sent to or received from the Internet.
+externally visible and no data is sent to or received from the internet.
If firewall software complains anyway, you can ignore it.
If the attempt to make the socket connection fails, Idle will notify you.
Such failures are sometimes transient, but if persistent, the problem
@@ -818,12 +949,12 @@
Help menu entry “IDLE Help” displays a formatted html version of the
IDLE chapter of the Library Reference. The result, in a read-only
tkinter text window, is close to what one sees in a web browser.
@@ -838,9 +969,9 @@
The font preferences, highlighting, keys, and general preferences can be
changed via Configure IDLE on the Option menu.
Non-default user settings are saved in a .idlerc directory in the user’s
@@ -856,23 +987,35 @@
Under System Preferences: Dock, one can set “Prefer tabs when opening
documents” to “Always”. This setting is not compatible with the tk/tkinter
GUI framework used by IDLE, and it breaks a few IDLE features.
IDLE contains an extension facility. Preferences for extensions can be
changed with the Extensions tab of the preferences dialog. See the
beginning of config-extensions.def in the idlelib directory for further
information. The only current default extension is zzdummy, an example
also used for testing.
The Lib/idlelib package implements the IDLE application. See the rest
+of this page for how to use IDLE.
+
The files in idlelib are described in idlelib/README.txt. Access it
+either in idlelib or click Help => About IDLE on the IDLE menu. This
+file also maps IDLE menu items to the code that implements the item.
+Except for files listed under ‘Startup’, the idlelib code is ‘private’ in
+sense that feature changes can be backported (see PEP 434).
diff --git a/Lib/idlelib/help.py b/Lib/idlelib/help.py
index f420d40fb9ea40..cc027b9cef4f5b 100644
--- a/Lib/idlelib/help.py
+++ b/Lib/idlelib/help.py
@@ -76,9 +76,9 @@ def handle_starttag(self, tag, attrs):
if a == 'class':
class_ = v
s = ''
- if tag == 'div' and class_ == 'section':
+ if tag == 'section' and attrs == [('id', 'idle')]:
self.show = True # Start main content.
- elif tag == 'div' and class_ == 'sphinxsidebar':
+ elif tag == 'div' and class_ == 'clearer':
self.show = False # End main content.
elif tag == 'p' and self.prevtag and not self.prevtag[0]:
# Begin a new block for
tags after a closed tag.
diff --git a/Lib/idlelib/help_about.py b/Lib/idlelib/help_about.py
index 64b13ac2abb3b2..710f2ac71074ba 100644
--- a/Lib/idlelib/help_about.py
+++ b/Lib/idlelib/help_about.py
@@ -3,6 +3,7 @@
"""
import os
import sys
+import webbrowser
from platform import python_version, architecture
from tkinter import Toplevel, Frame, Label, Button, PhotoImage
@@ -10,6 +11,8 @@
from idlelib import textview
+version = python_version()
+
def build_bits():
"Return bits for platform."
@@ -42,7 +45,7 @@ def __init__(self, parent, title=None, *, _htest=False, _utest=False):
self.create_widgets()
self.resizable(height=False, width=False)
self.title(title or
- f'About IDLE {python_version()} ({build_bits()} bit)')
+ f'About IDLE {version} ({build_bits()} bit)')
self.transient(parent)
self.grab_set()
self.protocol("WM_DELETE_WINDOW", self.ok)
@@ -88,17 +91,19 @@ def create_widgets(self):
email = Label(frame_background, text='email: idle-dev@python.org',
justify=LEFT, fg=self.fg, bg=self.bg)
email.grid(row=6, column=0, columnspan=2, sticky=W, padx=10, pady=0)
- docs = Label(frame_background, text='https://docs.python.org/' +
- python_version()[:3] + '/library/idle.html',
+ docs_url = ("https://docs.python.org/%d.%d/library/idle.html" %
+ sys.version_info[:2])
+ docs = Label(frame_background, text=docs_url,
justify=LEFT, fg=self.fg, bg=self.bg)
docs.grid(row=7, column=0, columnspan=2, sticky=W, padx=10, pady=0)
+ docs.bind("", lambda event: webbrowser.open(docs['text']))
Frame(frame_background, borderwidth=1, relief=SUNKEN,
height=2, bg=self.bg).grid(row=8, column=0, sticky=EW,
columnspan=3, padx=5, pady=5)
pyver = Label(frame_background,
- text='Python version: ' + python_version(),
+ text='Python version: ' + version,
fg=self.fg, bg=self.bg)
pyver.grid(row=9, column=0, sticky=W, padx=10, pady=0)
tkver = Label(frame_background, text='Tk version: ' + tk_patchlevel,
@@ -124,7 +129,7 @@ def create_widgets(self):
columnspan=3, padx=5, pady=5)
idlever = Label(frame_background,
- text='IDLE version: ' + python_version(),
+ text='IDLE version: ' + version,
fg=self.fg, bg=self.bg)
idlever.grid(row=12, column=0, sticky=W, padx=10, pady=0)
idle_buttons = Frame(frame_background, bg=self.bg)
diff --git a/Lib/idlelib/history.py b/Lib/idlelib/history.py
index 7ce09253eff5c9..621c5812fc25c9 100644
--- a/Lib/idlelib/history.py
+++ b/Lib/idlelib/history.py
@@ -65,7 +65,7 @@ def fetch(self, reverse):
self.text.bell()
return
nprefix = len(prefix)
- while 1:
+ while True:
pointer += -1 if reverse else 1
if pointer < 0 or pointer >= nhist:
self.text.bell()
diff --git a/Lib/idlelib/hyperparser.py b/Lib/idlelib/hyperparser.py
index 77baca782b3fdc..76144ee8fb30f5 100644
--- a/Lib/idlelib/hyperparser.py
+++ b/Lib/idlelib/hyperparser.py
@@ -237,9 +237,9 @@ def get_expression(self):
last_identifier_pos = pos
postdot_phase = True
- while 1:
+ while True:
# Eat whitespaces, comments, and if postdot_phase is False - a dot
- while 1:
+ while True:
if pos>brck_limit and rawtext[pos-1] in self._whitespace_chars:
# Eat a whitespace
pos -= 1
diff --git a/Lib/idlelib/idle_test/example_noext b/Lib/idlelib/idle_test/example_noext
new file mode 100644
index 00000000000000..7d2510e30bc0ae
--- /dev/null
+++ b/Lib/idlelib/idle_test/example_noext
@@ -0,0 +1,4 @@
+#!usr/bin/env python
+
+def example_function(some_argument):
+ pass
diff --git a/Lib/idlelib/idle_test/example_stub.pyi b/Lib/idlelib/idle_test/example_stub.pyi
new file mode 100644
index 00000000000000..a9811a78d10a6e
--- /dev/null
+++ b/Lib/idlelib/idle_test/example_stub.pyi
@@ -0,0 +1,2 @@
+class Example:
+ def method(self, argument1: str, argument2: list[int]) -> None: ...
diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
index 1373b7642a6ea9..d297f8aa0094ee 100644
--- a/Lib/idlelib/idle_test/htest.py
+++ b/Lib/idlelib/idle_test/htest.py
@@ -77,8 +77,8 @@ def _wrapper(parent): # htest #
'kwds': {'title': 'help_about test',
'_htest': True,
},
- 'msg': "Test every button. Ensure Python, TK and IDLE versions "
- "are correctly displayed.\n [Close] to exit.",
+ 'msg': "Click on URL to open in default browser.\n"
+ "Verify x.y.z versions and test each button, including Close.\n "
}
# TODO implement ^\; adding '' to function does not work.
@@ -246,7 +246,7 @@ def _wrapper(parent): # htest #
_object_browser_spec = {
'file': 'debugobj',
'kwds': {},
- 'msg': "Double click on items upto the lowest level.\n"
+ 'msg': "Double click on items up to the lowest level.\n"
"Attributes of the objects and related information "
"will be displayed side-by-side at each level."
}
@@ -255,7 +255,7 @@ def _wrapper(parent): # htest #
'file': 'pathbrowser',
'kwds': {},
'msg': "Test for correct display of all paths in sys.path.\n"
- "Toggle nested items upto the lowest level.\n"
+ "Toggle nested items up to the lowest level.\n"
"Double clicking on an item prints a traceback\n"
"for an exception that is ignored."
}
@@ -341,7 +341,7 @@ def _wrapper(parent): # htest #
'file': 'tree',
'kwds': {},
'msg': "The canvas is scrollable.\n"
- "Click on folders upto to the lowest level."
+ "Click on folders up to to the lowest level."
}
_undo_delegator_spec = {
diff --git a/Lib/idlelib/idle_test/mock_tk.py b/Lib/idlelib/idle_test/mock_tk.py
index db583553838fb3..8304734b847a83 100644
--- a/Lib/idlelib/idle_test/mock_tk.py
+++ b/Lib/idlelib/idle_test/mock_tk.py
@@ -79,7 +79,7 @@ def tearDownClass(cls):
---
For 'ask' functions, set func.result return value before calling the method
that uses the message function. When messagebox functions are the
- only gui alls in a method, this replacement makes the method gui-free,
+ only GUI calls in a method, this replacement makes the method GUI-free,
"""
askokcancel = Mbox_func() # True or False
askquestion = Mbox_func() # 'yes' or 'no'
diff --git a/Lib/idlelib/idle_test/test_autocomplete.py b/Lib/idlelib/idle_test/test_autocomplete.py
index 642bb5db64dc34..a811363c18d04e 100644
--- a/Lib/idlelib/idle_test/test_autocomplete.py
+++ b/Lib/idlelib/idle_test/test_autocomplete.py
@@ -218,6 +218,11 @@ def make_acw(): return self.dummy_acw()
self.assertTrue(acp.open_completions(ac.TAB))
self.text.delete('1.0', 'end')
+ def test_completion_kwds(self):
+ self.assertIn('and', ac.completion_kwds)
+ self.assertIn('case', ac.completion_kwds)
+ self.assertNotIn('None', ac.completion_kwds)
+
def test_fetch_completions(self):
# Test that fetch_completions returns 2 lists:
# For attribute completion, a large list containing all variables, and
diff --git a/Lib/idlelib/idle_test/test_browser.py b/Lib/idlelib/idle_test/test_browser.py
index 03a50f22ca1e82..6cfea3888cd6a9 100644
--- a/Lib/idlelib/idle_test/test_browser.py
+++ b/Lib/idlelib/idle_test/test_browser.py
@@ -5,6 +5,7 @@
import unittest
from unittest import mock
from idlelib.idle_test.mock_idle import Func
+from idlelib.util import py_extensions
from collections import deque
import os.path
@@ -57,6 +58,15 @@ def test_close(self):
self.assertTrue(mb.node.destroy.called)
del mb.top.destroy, mb.node.destroy
+ def test_is_browseable_extension(self):
+ path = "/path/to/file"
+ for ext in py_extensions:
+ with self.subTest(ext=ext):
+ filename = f'{path}{ext}'
+ actual = browser.is_browseable_extension(filename)
+ expected = ext not in browser.browseable_extension_blocklist
+ self.assertEqual(actual, expected)
+
# Nested tree same as in test_pyclbr.py except for supers on C0. C1.
mb = pyclbr
@@ -160,8 +170,7 @@ def test_ondoubleclick(self, fopen):
with mock.patch('os.path.exists', return_value=True):
mbt.OnDoubleClick()
- fopen.assert_called()
- fopen.called_with(fname)
+ fopen.assert_called_once_with(fname)
class ChildBrowserTreeItemTest(unittest.TestCase):
diff --git a/Lib/idlelib/idle_test/test_calltip.py b/Lib/idlelib/idle_test/test_calltip.py
index b23915c5ab7849..e8d2bd17cb6811 100644
--- a/Lib/idlelib/idle_test/test_calltip.py
+++ b/Lib/idlelib/idle_test/test_calltip.py
@@ -106,7 +106,7 @@ def test_signature_wrap(self):
If you want to completely replace the main wrapping algorithm,
you\'ll probably have to override _wrap_chunks().''')
- def test_properly_formated(self):
+ def test_properly_formatted(self):
def foo(s='a'*100):
pass
diff --git a/Lib/idlelib/idle_test/test_colorizer.py b/Lib/idlelib/idle_test/test_colorizer.py
index c31c49236ca0b9..308bc389384d33 100644
--- a/Lib/idlelib/idle_test/test_colorizer.py
+++ b/Lib/idlelib/idle_test/test_colorizer.py
@@ -1,11 +1,12 @@
-"Test colorizer, coverage 93%."
-
+"Test colorizer, coverage 99%."
from idlelib import colorizer
from test.support import requires
import unittest
from unittest import mock
+from idlelib.idle_test.tkinter_testing_utils import run_in_tk_mainloop
from functools import partial
+import textwrap
from tkinter import Tk, Text
from idlelib import config
from idlelib.percolator import Percolator
@@ -19,15 +20,38 @@
'extensions': config.IdleUserConfParser(''),
}
-source = (
- "if True: int ('1') # keyword, builtin, string, comment\n"
- "elif False: print(0) # 'string' in comment\n"
- "else: float(None) # if in comment\n"
- "if iF + If + IF: 'keyword matching must respect case'\n"
- "if'': x or'' # valid string-keyword no-space combinations\n"
- "async def f(): await g()\n"
- "'x', '''x''', \"x\", \"\"\"x\"\"\"\n"
- )
+source = textwrap.dedent("""\
+ if True: int ('1') # keyword, builtin, string, comment
+ elif False: print(0) # 'string' in comment
+ else: float(None) # if in comment
+ if iF + If + IF: 'keyword matching must respect case'
+ if'': x or'' # valid keyword-string no-space combinations
+ async def f(): await g()
+ # Strings should be entirely colored, including quotes.
+ 'x', '''x''', "x", \"""x\"""
+ 'abc\\
+ def'
+ '''abc\\
+ def'''
+ # All valid prefixes for unicode and byte strings should be colored.
+ r'x', u'x', R'x', U'x', f'x', F'x'
+ fr'x', Fr'x', fR'x', FR'x', rf'x', rF'x', Rf'x', RF'x'
+ b'x',B'x', br'x',Br'x',bR'x',BR'x', rb'x', rB'x',Rb'x',RB'x'
+ # Invalid combinations of legal characters should be half colored.
+ ur'x', ru'x', uf'x', fu'x', UR'x', ufr'x', rfu'x', xf'x', fx'x'
+ match point:
+ case (x, 0) as _:
+ print(f"X={x}")
+ case [_, [_], "_",
+ _]:
+ pass
+ case _ if ("a" if _ else set()): pass
+ case _:
+ raise ValueError("Not a point _")
+ '''
+ case _:'''
+ "match x:"
+ """)
def setUpModule():
@@ -107,7 +131,7 @@ def setUpClass(cls):
requires('gui')
root = cls.root = Tk()
root.withdraw()
- text = cls.text = Text(root)
+ cls.text = Text(root)
@classmethod
def tearDownClass(cls):
@@ -152,7 +176,7 @@ def setUpClass(cls):
@classmethod
def tearDownClass(cls):
- cls.percolator.redir.close()
+ cls.percolator.close()
del cls.percolator, cls.text
cls.root.update_idletasks()
cls.root.destroy()
@@ -364,8 +388,21 @@ def test_recolorize_main(self, mock_notify):
('4.0', ('KEYWORD',)), ('4.3', ()), ('4.6', ()),
('5.2', ('STRING',)), ('5.8', ('KEYWORD',)), ('5.10', ('STRING',)),
('6.0', ('KEYWORD',)), ('6.10', ('DEFINITION',)), ('6.11', ()),
- ('7.0', ('STRING',)), ('7.4', ()), ('7.5', ('STRING',)),
- ('7.12', ()), ('7.14', ('STRING',)),
+ ('8.0', ('STRING',)), ('8.4', ()), ('8.5', ('STRING',)),
+ ('8.12', ()), ('8.14', ('STRING',)),
+ ('19.0', ('KEYWORD',)),
+ ('20.4', ('KEYWORD',)), ('20.16', ('KEYWORD',)),# ('20.19', ('KEYWORD',)),
+ #('22.4', ('KEYWORD',)), ('22.10', ('KEYWORD',)), ('22.14', ('KEYWORD',)), ('22.19', ('STRING',)),
+ #('23.12', ('KEYWORD',)),
+ ('24.8', ('KEYWORD',)),
+ ('25.4', ('KEYWORD',)), ('25.9', ('KEYWORD',)),
+ ('25.11', ('KEYWORD',)), ('25.15', ('STRING',)),
+ ('25.19', ('KEYWORD',)), ('25.22', ()),
+ ('25.24', ('KEYWORD',)), ('25.29', ('BUILTIN',)), ('25.37', ('KEYWORD',)),
+ ('26.4', ('KEYWORD',)), ('26.9', ('KEYWORD',)),# ('26.11', ('KEYWORD',)), ('26.14', (),),
+ ('27.25', ('STRING',)), ('27.38', ('STRING',)),
+ ('29.0', ('STRING',)),
+ ('30.1', ('STRING',)),
# SYNC at the end of every line.
('1.55', ('SYNC',)), ('2.50', ('SYNC',)), ('3.34', ('SYNC',)),
)
@@ -391,11 +428,173 @@ def test_recolorize_main(self, mock_notify):
eq(text.tag_nextrange('COMMENT', '2.0'), ('2.22', '2.43'))
eq(text.tag_nextrange('SYNC', '2.0'), ('2.43', '3.0'))
eq(text.tag_nextrange('STRING', '2.0'), ('4.17', '4.53'))
- eq(text.tag_nextrange('STRING', '7.0'), ('7.0', '7.3'))
- eq(text.tag_nextrange('STRING', '7.3'), ('7.5', '7.12'))
- eq(text.tag_nextrange('STRING', '7.12'), ('7.14', '7.17'))
- eq(text.tag_nextrange('STRING', '7.17'), ('7.19', '7.26'))
- eq(text.tag_nextrange('SYNC', '7.0'), ('7.26', '9.0'))
+ eq(text.tag_nextrange('STRING', '8.0'), ('8.0', '8.3'))
+ eq(text.tag_nextrange('STRING', '8.3'), ('8.5', '8.12'))
+ eq(text.tag_nextrange('STRING', '8.12'), ('8.14', '8.17'))
+ eq(text.tag_nextrange('STRING', '8.17'), ('8.19', '8.26'))
+ eq(text.tag_nextrange('SYNC', '8.0'), ('8.26', '9.0'))
+ eq(text.tag_nextrange('SYNC', '30.0'), ('30.10', '32.0'))
+
+ def _assert_highlighting(self, source, tag_ranges):
+ """Check highlighting of a given piece of code.
+
+ This inserts just this code into the Text widget. It will then
+ check that the resulting highlighting tag ranges exactly match
+ those described in the given `tag_ranges` dict.
+
+ Note that the irrelevant tags 'sel', 'TODO' and 'SYNC' are
+ ignored.
+ """
+ text = self.text
+
+ with mock.patch.object(colorizer.ColorDelegator, 'notify_range'):
+ text.delete('1.0', 'end-1c')
+ text.insert('insert', source)
+ text.tag_add('TODO', '1.0', 'end-1c')
+ self.color.recolorize_main()
+
+ # Make a dict with highlighting tag ranges in the Text widget.
+ text_tag_ranges = {}
+ for tag in set(text.tag_names()) - {'sel', 'TODO', 'SYNC'}:
+ indexes = [rng.string for rng in text.tag_ranges(tag)]
+ for index_pair in zip(indexes[::2], indexes[1::2]):
+ text_tag_ranges.setdefault(tag, []).append(index_pair)
+
+ self.assertEqual(text_tag_ranges, tag_ranges)
+
+ with mock.patch.object(colorizer.ColorDelegator, 'notify_range'):
+ text.delete('1.0', 'end-1c')
+
+ def test_def_statement(self):
+ # empty def
+ self._assert_highlighting('def', {'KEYWORD': [('1.0', '1.3')]})
+
+ # def followed by identifier
+ self._assert_highlighting('def foo:', {'KEYWORD': [('1.0', '1.3')],
+ 'DEFINITION': [('1.4', '1.7')]})
+
+ # def followed by partial identifier
+ self._assert_highlighting('def fo', {'KEYWORD': [('1.0', '1.3')],
+ 'DEFINITION': [('1.4', '1.6')]})
+
+ # def followed by non-keyword
+ self._assert_highlighting('def ++', {'KEYWORD': [('1.0', '1.3')]})
+
+ def test_match_soft_keyword(self):
+ # empty match
+ self._assert_highlighting('match', {'KEYWORD': [('1.0', '1.5')]})
+
+ # match followed by partial identifier
+ self._assert_highlighting('match fo', {'KEYWORD': [('1.0', '1.5')]})
+
+ # match followed by identifier and colon
+ self._assert_highlighting('match foo:', {'KEYWORD': [('1.0', '1.5')]})
+
+ # match followed by keyword
+ self._assert_highlighting('match and', {'KEYWORD': [('1.6', '1.9')]})
+
+ # match followed by builtin with keyword prefix
+ self._assert_highlighting('match int:', {'KEYWORD': [('1.0', '1.5')],
+ 'BUILTIN': [('1.6', '1.9')]})
+
+ # match followed by non-text operator
+ self._assert_highlighting('match^', {})
+ self._assert_highlighting('match @', {})
+
+ # match followed by colon
+ self._assert_highlighting('match :', {})
+
+ # match followed by comma
+ self._assert_highlighting('match\t,', {})
+
+ # match followed by a lone underscore
+ self._assert_highlighting('match _:', {'KEYWORD': [('1.0', '1.5')]})
+
+ def test_case_soft_keyword(self):
+ # empty case
+ self._assert_highlighting('case', {'KEYWORD': [('1.0', '1.4')]})
+
+ # case followed by partial identifier
+ self._assert_highlighting('case fo', {'KEYWORD': [('1.0', '1.4')]})
+
+ # case followed by identifier and colon
+ self._assert_highlighting('case foo:', {'KEYWORD': [('1.0', '1.4')]})
+
+ # case followed by keyword
+ self._assert_highlighting('case and', {'KEYWORD': [('1.5', '1.8')]})
+
+ # case followed by builtin with keyword prefix
+ self._assert_highlighting('case int:', {'KEYWORD': [('1.0', '1.4')],
+ 'BUILTIN': [('1.5', '1.8')]})
+
+ # case followed by non-text operator
+ self._assert_highlighting('case^', {})
+ self._assert_highlighting('case @', {})
+
+ # case followed by colon
+ self._assert_highlighting('case :', {})
+
+ # case followed by comma
+ self._assert_highlighting('case\t,', {})
+
+ # case followed by a lone underscore
+ self._assert_highlighting('case _:', {'KEYWORD': [('1.0', '1.4'),
+ ('1.5', '1.6')]})
+
+ def test_long_multiline_string(self):
+ source = textwrap.dedent('''\
+ """a
+ b
+ c
+ d
+ e"""
+ ''')
+ self._assert_highlighting(source, {'STRING': [('1.0', '5.4')]})
+
+ @run_in_tk_mainloop(delay=50)
+ def test_incremental_editing(self):
+ text = self.text
+ eq = self.assertEqual
+
+ # Simulate typing 'inte'. During this, the highlighting should
+ # change from normal to keyword to builtin to normal.
+ text.insert('insert', 'i')
+ yield
+ eq(text.tag_nextrange('BUILTIN', '1.0'), ())
+ eq(text.tag_nextrange('KEYWORD', '1.0'), ())
+
+ text.insert('insert', 'n')
+ yield
+ eq(text.tag_nextrange('BUILTIN', '1.0'), ())
+ eq(text.tag_nextrange('KEYWORD', '1.0'), ('1.0', '1.2'))
+
+ text.insert('insert', 't')
+ yield
+ eq(text.tag_nextrange('BUILTIN', '1.0'), ('1.0', '1.3'))
+ eq(text.tag_nextrange('KEYWORD', '1.0'), ())
+
+ text.insert('insert', 'e')
+ yield
+ eq(text.tag_nextrange('BUILTIN', '1.0'), ())
+ eq(text.tag_nextrange('KEYWORD', '1.0'), ())
+
+ # Simulate deleting three characters from the end of 'inte'.
+ # During this, the highlighting should change from normal to
+ # builtin to keyword to normal.
+ text.delete('insert-1c', 'insert')
+ yield
+ eq(text.tag_nextrange('BUILTIN', '1.0'), ('1.0', '1.3'))
+ eq(text.tag_nextrange('KEYWORD', '1.0'), ())
+
+ text.delete('insert-1c', 'insert')
+ yield
+ eq(text.tag_nextrange('BUILTIN', '1.0'), ())
+ eq(text.tag_nextrange('KEYWORD', '1.0'), ('1.0', '1.2'))
+
+ text.delete('insert-1c', 'insert')
+ yield
+ eq(text.tag_nextrange('BUILTIN', '1.0'), ())
+ eq(text.tag_nextrange('KEYWORD', '1.0'), ())
@mock.patch.object(colorizer.ColorDelegator, 'recolorize')
@mock.patch.object(colorizer.ColorDelegator, 'notify_range')
diff --git a/Lib/idlelib/idle_test/test_config_key.py b/Lib/idlelib/idle_test/test_config_key.py
index b7fe7fd6b5ec10..32f878b842b276 100644
--- a/Lib/idlelib/idle_test/test_config_key.py
+++ b/Lib/idlelib/idle_test/test_config_key.py
@@ -2,7 +2,7 @@
Coverage is effectively 100%. Tkinter dialog is mocked, Mac-only line
may be skipped, and dummy function in bind test should not be called.
-Not tested: exit with 'self.advanced or self.keys_ok(keys)) ...' False.
+Not tested: exit with 'self.advanced or self.keys_ok(keys) ...' False.
"""
from idlelib import config_key
@@ -13,15 +13,13 @@
from idlelib.idle_test.mock_idle import Func
from idlelib.idle_test.mock_tk import Mbox_func
-gkd = config_key.GetKeysDialog
-
class ValidationTest(unittest.TestCase):
"Test validation methods: ok, keys_ok, bind_ok."
- class Validator(gkd):
+ class Validator(config_key.GetKeysFrame):
def __init__(self, *args, **kwargs):
- config_key.GetKeysDialog.__init__(self, *args, **kwargs)
+ super().__init__(*args, **kwargs)
class list_keys_final:
get = Func()
self.list_keys_final = list_keys_final
@@ -34,15 +32,14 @@ def setUpClass(cls):
cls.root = Tk()
cls.root.withdraw()
keylist = [[''], ['', '']]
- cls.dialog = cls.Validator(
- cls.root, 'Title', '<>', keylist, _utest=True)
+ cls.dialog = cls.Validator(cls.root, '<>', keylist)
@classmethod
def tearDownClass(cls):
- cls.dialog.cancel()
+ del cls.dialog
cls.root.update_idletasks()
cls.root.destroy()
- del cls.dialog, cls.root
+ del cls.root
def setUp(self):
self.dialog.showerror.message = ''
@@ -111,14 +108,14 @@ def setUpClass(cls):
requires('gui')
cls.root = Tk()
cls.root.withdraw()
- cls.dialog = gkd(cls.root, 'Title', '<>', [], _utest=True)
+ cls.dialog = config_key.GetKeysFrame(cls.root, '<>', [])
@classmethod
def tearDownClass(cls):
- cls.dialog.cancel()
+ del cls.dialog
cls.root.update_idletasks()
cls.root.destroy()
- del cls.dialog, cls.root
+ del cls.root
def test_toggle_level(self):
dialog = self.dialog
@@ -130,7 +127,7 @@ def stackorder():
this can be used to check whether a frame is above or
below another one.
"""
- for index, child in enumerate(dialog.frame.winfo_children()):
+ for index, child in enumerate(dialog.winfo_children()):
if child._name == 'keyseq_basic':
basic = index
if child._name == 'keyseq_advanced':
@@ -161,7 +158,7 @@ def stackorder():
class KeySelectionTest(unittest.TestCase):
"Test selecting key on Basic frames."
- class Basic(gkd):
+ class Basic(config_key.GetKeysFrame):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
class list_keys_final:
@@ -179,14 +176,14 @@ def setUpClass(cls):
requires('gui')
cls.root = Tk()
cls.root.withdraw()
- cls.dialog = cls.Basic(cls.root, 'Title', '<>', [], _utest=True)
+ cls.dialog = cls.Basic(cls.root, '<>', [])
@classmethod
def tearDownClass(cls):
- cls.dialog.cancel()
+ del cls.dialog
cls.root.update_idletasks()
cls.root.destroy()
- del cls.dialog, cls.root
+ del cls.root
def setUp(self):
self.dialog.clear_key_seq()
@@ -206,7 +203,7 @@ def test_get_modifiers(self):
dialog.modifier_checkbuttons['foo'].invoke()
eq(gm(), ['BAZ'])
- @mock.patch.object(gkd, 'get_modifiers')
+ @mock.patch.object(config_key.GetKeysFrame, 'get_modifiers')
def test_build_key_string(self, mock_modifiers):
dialog = self.dialog
key = dialog.list_keys_final
@@ -227,7 +224,7 @@ def test_build_key_string(self, mock_modifiers):
dialog.build_key_string()
eq(string(), '')
- @mock.patch.object(gkd, 'get_modifiers')
+ @mock.patch.object(config_key.GetKeysFrame, 'get_modifiers')
def test_final_key_selected(self, mock_modifiers):
dialog = self.dialog
key = dialog.list_keys_final
@@ -240,7 +237,7 @@ def test_final_key_selected(self, mock_modifiers):
eq(string(), '')
-class CancelTest(unittest.TestCase):
+class CancelWindowTest(unittest.TestCase):
"Simulate user clicking [Cancel] button."
@classmethod
@@ -248,21 +245,89 @@ def setUpClass(cls):
requires('gui')
cls.root = Tk()
cls.root.withdraw()
- cls.dialog = gkd(cls.root, 'Title', '<>', [], _utest=True)
+ cls.dialog = config_key.GetKeysWindow(
+ cls.root, 'Title', '<>', [], _utest=True)
@classmethod
def tearDownClass(cls):
cls.dialog.cancel()
+ del cls.dialog
cls.root.update_idletasks()
cls.root.destroy()
- del cls.dialog, cls.root
+ del cls.root
- def test_cancel(self):
+ @mock.patch.object(config_key.GetKeysFrame, 'ok')
+ def test_cancel(self, mock_frame_ok):
self.assertEqual(self.dialog.winfo_class(), 'Toplevel')
self.dialog.button_cancel.invoke()
with self.assertRaises(TclError):
self.dialog.winfo_class()
self.assertEqual(self.dialog.result, '')
+ mock_frame_ok.assert_not_called()
+
+
+class OKWindowTest(unittest.TestCase):
+ "Simulate user clicking [OK] button."
+
+ @classmethod
+ def setUpClass(cls):
+ requires('gui')
+ cls.root = Tk()
+ cls.root.withdraw()
+ cls.dialog = config_key.GetKeysWindow(
+ cls.root, 'Title', '<>', [], _utest=True)
+
+ @classmethod
+ def tearDownClass(cls):
+ cls.dialog.cancel()
+ del cls.dialog
+ cls.root.update_idletasks()
+ cls.root.destroy()
+ del cls.root
+
+ @mock.patch.object(config_key.GetKeysFrame, 'ok')
+ def test_ok(self, mock_frame_ok):
+ self.assertEqual(self.dialog.winfo_class(), 'Toplevel')
+ self.dialog.button_ok.invoke()
+ with self.assertRaises(TclError):
+ self.dialog.winfo_class()
+ mock_frame_ok.assert_called()
+
+
+class WindowResultTest(unittest.TestCase):
+ "Test window result get and set."
+
+ @classmethod
+ def setUpClass(cls):
+ requires('gui')
+ cls.root = Tk()
+ cls.root.withdraw()
+ cls.dialog = config_key.GetKeysWindow(
+ cls.root, 'Title', '<>', [], _utest=True)
+
+ @classmethod
+ def tearDownClass(cls):
+ cls.dialog.cancel()
+ del cls.dialog
+ cls.root.update_idletasks()
+ cls.root.destroy()
+ del cls.root
+
+ def test_result(self):
+ dialog = self.dialog
+ eq = self.assertEqual
+
+ dialog.result = ''
+ eq(dialog.result, '')
+ eq(dialog.frame.result,'')
+
+ dialog.result = 'bar'
+ eq(dialog.result,'bar')
+ eq(dialog.frame.result,'bar')
+
+ dialog.frame.result = 'foo'
+ eq(dialog.result, 'foo')
+ eq(dialog.frame.result,'foo')
class HelperTest(unittest.TestCase):
diff --git a/Lib/idlelib/idle_test/test_configdialog.py b/Lib/idlelib/idle_test/test_configdialog.py
index 98ddc67afdcc08..e5d5b4013fca57 100644
--- a/Lib/idlelib/idle_test/test_configdialog.py
+++ b/Lib/idlelib/idle_test/test_configdialog.py
@@ -73,13 +73,13 @@ def test_click_ok(self):
def test_click_apply(self):
d = dialog
deactivate = d.deactivate_current_config = mock.Mock()
- save_ext = d.save_all_changed_extensions = mock.Mock()
+ save_ext = d.extpage.save_all_changed_extensions = mock.Mock()
activate = d.activate_config_changes = mock.Mock()
d.buttons['Apply'].invoke()
deactivate.assert_called_once()
save_ext.assert_called_once()
activate.assert_called_once()
- del d.save_all_changed_extensions
+ del d.extpage.save_all_changed_extensions
del d.activate_config_changes, d.deactivate_current_config
def test_click_cancel(self):
@@ -260,27 +260,6 @@ def test_set_samples(self):
d.set_samples = Func() # Re-mask for other tests.
-class IndentTest(unittest.TestCase):
-
- @classmethod
- def setUpClass(cls):
- cls.page = dialog.fontpage
- cls.page.update()
-
- def test_load_tab_cfg(self):
- d = self.page
- d.space_num.set(16)
- d.load_tab_cfg()
- self.assertEqual(d.space_num.get(), 4)
-
- def test_indent_scale(self):
- d = self.page
- changes.clear()
- d.indent_scale.set(20)
- self.assertEqual(d.space_num.get(), 16)
- self.assertEqual(mainpage, {'Indent': {'num-spaces': '16'}})
-
-
class HighPageTest(unittest.TestCase):
"""Test that highlight tab widgets enable users to make changes.
@@ -975,8 +954,8 @@ def test_set_keys_type(self):
def test_get_new_keys(self):
eq = self.assertEqual
d = self.page
- orig_getkeysdialog = configdialog.GetKeysDialog
- gkd = configdialog.GetKeysDialog = Func(return_self=True)
+ orig_getkeysdialog = configdialog.GetKeysWindow
+ gkd = configdialog.GetKeysWindow = Func(return_self=True)
gnkn = d.get_new_keys_name = Func()
d.button_new_keys.state(('!disabled',))
@@ -1018,7 +997,7 @@ def test_get_new_keys(self):
eq(d.keybinding.get(), '')
del d.get_new_keys_name
- configdialog.GetKeysDialog = orig_getkeysdialog
+ configdialog.GetKeysWindow = orig_getkeysdialog
def test_get_new_keys_name(self):
orig_sectionname = configdialog.SectionName
@@ -1203,49 +1182,32 @@ def test_delete_custom_keys(self):
del d.askyesno
-class GenPageTest(unittest.TestCase):
+class WinPageTest(unittest.TestCase):
"""Test that general tab widgets enable users to make changes.
Test that widget actions set vars, that var changes add
- options to changes and that helplist works correctly.
+ options to changes.
"""
@classmethod
def setUpClass(cls):
- page = cls.page = dialog.genpage
+ page = cls.page = dialog.winpage
dialog.note.select(page)
- page.set = page.set_add_delete_state = Func()
- page.upc = page.update_help_changes = Func()
page.update()
- @classmethod
- def tearDownClass(cls):
- page = cls.page
- del page.set, page.set_add_delete_state
- del page.upc, page.update_help_changes
- page.helplist.delete(0, 'end')
- page.user_helplist.clear()
-
def setUp(self):
changes.clear()
- def test_load_general_cfg(self):
+ def test_load_windows_cfg(self):
# Set to wrong values, load, check right values.
eq = self.assertEqual
d = self.page
d.startup_edit.set(1)
- d.autosave.set(1)
d.win_width.set(1)
d.win_height.set(1)
- d.helplist.insert('end', 'bad')
- d.user_helplist = ['bad', 'worse']
- idleConf.SetOption('main', 'HelpFiles', '1', 'name;file')
- d.load_general_cfg()
+ d.load_windows_cfg()
eq(d.startup_edit.get(), 0)
- eq(d.autosave.get(), 0)
eq(d.win_width.get(), '80')
eq(d.win_height.get(), '40')
- eq(d.helplist.get(0, 'end'), ('name',))
- eq(d.user_helplist, [('name', 'file', '1')])
def test_startup(self):
d = self.page
@@ -1267,6 +1229,12 @@ def test_editor_size(self):
d.win_width_int.insert(0, '11')
self.assertEqual(mainpage, {'EditorWindow': {'width': '11'}})
+ def test_indent_spaces(self):
+ d = self.page
+ d.indent_chooser.set(6)
+ self.assertEqual(d.indent_spaces.get(), '6')
+ self.assertEqual(mainpage, {'Indent': {'num-spaces': '6'}})
+
def test_cursor_blink(self):
self.page.cursor_blink_bool.invoke()
self.assertEqual(mainpage, {'EditorWindow': {'cursor-blink': 'False'}})
@@ -1289,6 +1257,35 @@ def test_parenmatch(self):
d.bell_on.invoke()
eq(extpage, {'ParenMatch': {'bell': 'False'}})
+ def test_paragraph(self):
+ self.page.format_width_int.delete(0, 'end')
+ self.page.format_width_int.insert(0, '11')
+ self.assertEqual(extpage, {'FormatParagraph': {'max-width': '11'}})
+
+
+class ShedPageTest(unittest.TestCase):
+ """Test that shed tab widgets enable users to make changes.
+
+ Test that widget actions set vars, that var changes add
+ options to changes.
+ """
+ @classmethod
+ def setUpClass(cls):
+ page = cls.page = dialog.shedpage
+ dialog.note.select(page)
+ page.update()
+
+ def setUp(self):
+ changes.clear()
+
+ def test_load_shelled_cfg(self):
+ # Set to wrong values, load, check right values.
+ eq = self.assertEqual
+ d = self.page
+ d.autosave.set(1)
+ d.load_shelled_cfg()
+ eq(d.autosave.get(), 0)
+
def test_autosave(self):
d = self.page
d.save_auto_on.invoke()
@@ -1296,21 +1293,58 @@ def test_autosave(self):
d.save_ask_on.invoke()
self.assertEqual(mainpage, {'General': {'autosave': '0'}})
- def test_paragraph(self):
- self.page.format_width_int.delete(0, 'end')
- self.page.format_width_int.insert(0, '11')
- self.assertEqual(extpage, {'FormatParagraph': {'max-width': '11'}})
-
def test_context(self):
self.page.context_int.delete(0, 'end')
self.page.context_int.insert(0, '1')
self.assertEqual(extpage, {'CodeContext': {'maxlines': '1'}})
+
+#unittest.skip("Nothing here yet TODO")
+class ExtPageTest(unittest.TestCase):
+ """Test that the help source list works correctly."""
+ @classmethod
+ def setUpClass(cls):
+ page = dialog.extpage
+ dialog.note.select(page)
+
+
+class HelpSourceTest(unittest.TestCase):
+ """Test that the help source list works correctly."""
+ @classmethod
+ def setUpClass(cls):
+ page = dialog.extpage
+ dialog.note.select(page)
+ frame = cls.frame = page.frame_help
+ frame.set = frame.set_add_delete_state = Func()
+ frame.upc = frame.update_help_changes = Func()
+ frame.update()
+
+ @classmethod
+ def tearDownClass(cls):
+ frame = cls.frame
+ del frame.set, frame.set_add_delete_state
+ del frame.upc, frame.update_help_changes
+ frame.helplist.delete(0, 'end')
+ frame.user_helplist.clear()
+
+ def setUp(self):
+ changes.clear()
+
+ def test_load_helplist(self):
+ eq = self.assertEqual
+ fr = self.frame
+ fr.helplist.insert('end', 'bad')
+ fr.user_helplist = ['bad', 'worse']
+ idleConf.SetOption('main', 'HelpFiles', '1', 'name;file')
+ fr.load_helplist()
+ eq(fr.helplist.get(0, 'end'), ('name',))
+ eq(fr.user_helplist, [('name', 'file', '1')])
+
def test_source_selected(self):
- d = self.page
- d.set = d.set_add_delete_state
- d.upc = d.update_help_changes
- helplist = d.helplist
+ fr = self.frame
+ fr.set = fr.set_add_delete_state
+ fr.upc = fr.update_help_changes
+ helplist = fr.helplist
dex = 'end'
helplist.insert(dex, 'source')
helplist.activate(dex)
@@ -1321,38 +1355,38 @@ def test_source_selected(self):
x, y, dx, dy = helplist.bbox(dex)
x += dx // 2
y += dy // 2
- d.set.called = d.upc.called = 0
+ fr.set.called = fr.upc.called = 0
helplist.event_generate('', x=0, y=0)
helplist.event_generate('', x=x, y=y)
helplist.event_generate('', x=x, y=y)
helplist.event_generate('', x=x, y=y)
self.assertEqual(helplist.get('anchor'), 'source')
- self.assertTrue(d.set.called)
- self.assertFalse(d.upc.called)
+ self.assertTrue(fr.set.called)
+ self.assertFalse(fr.upc.called)
def test_set_add_delete_state(self):
# Call with 0 items, 1 unselected item, 1 selected item.
eq = self.assertEqual
- d = self.page
- del d.set_add_delete_state # Unmask method.
- sad = d.set_add_delete_state
- h = d.helplist
+ fr = self.frame
+ del fr.set_add_delete_state # Unmask method.
+ sad = fr.set_add_delete_state
+ h = fr.helplist
h.delete(0, 'end')
sad()
- eq(d.button_helplist_edit.state(), ('disabled',))
- eq(d.button_helplist_remove.state(), ('disabled',))
+ eq(fr.button_helplist_edit.state(), ('disabled',))
+ eq(fr.button_helplist_remove.state(), ('disabled',))
h.insert(0, 'source')
sad()
- eq(d.button_helplist_edit.state(), ('disabled',))
- eq(d.button_helplist_remove.state(), ('disabled',))
+ eq(fr.button_helplist_edit.state(), ('disabled',))
+ eq(fr.button_helplist_remove.state(), ('disabled',))
h.selection_set(0)
sad()
- eq(d.button_helplist_edit.state(), ())
- eq(d.button_helplist_remove.state(), ())
- d.set_add_delete_state = Func() # Mask method.
+ eq(fr.button_helplist_edit.state(), ())
+ eq(fr.button_helplist_remove.state(), ())
+ fr.set_add_delete_state = Func() # Mask method.
def test_helplist_item_add(self):
# Call without and twice with HelpSource result.
@@ -1360,25 +1394,25 @@ def test_helplist_item_add(self):
eq = self.assertEqual
orig_helpsource = configdialog.HelpSource
hs = configdialog.HelpSource = Func(return_self=True)
- d = self.page
- d.helplist.delete(0, 'end')
- d.user_helplist.clear()
- d.set.called = d.upc.called = 0
+ fr = self.frame
+ fr.helplist.delete(0, 'end')
+ fr.user_helplist.clear()
+ fr.set.called = fr.upc.called = 0
hs.result = ''
- d.helplist_item_add()
- self.assertTrue(list(d.helplist.get(0, 'end')) ==
- d.user_helplist == [])
- self.assertFalse(d.upc.called)
+ fr.helplist_item_add()
+ self.assertTrue(list(fr.helplist.get(0, 'end')) ==
+ fr.user_helplist == [])
+ self.assertFalse(fr.upc.called)
hs.result = ('name1', 'file1')
- d.helplist_item_add()
+ fr.helplist_item_add()
hs.result = ('name2', 'file2')
- d.helplist_item_add()
- eq(d.helplist.get(0, 'end'), ('name1', 'name2'))
- eq(d.user_helplist, [('name1', 'file1'), ('name2', 'file2')])
- eq(d.upc.called, 2)
- self.assertFalse(d.set.called)
+ fr.helplist_item_add()
+ eq(fr.helplist.get(0, 'end'), ('name1', 'name2'))
+ eq(fr.user_helplist, [('name1', 'file1'), ('name2', 'file2')])
+ eq(fr.upc.called, 2)
+ self.assertFalse(fr.set.called)
configdialog.HelpSource = orig_helpsource
@@ -1387,58 +1421,58 @@ def test_helplist_item_edit(self):
eq = self.assertEqual
orig_helpsource = configdialog.HelpSource
hs = configdialog.HelpSource = Func(return_self=True)
- d = self.page
- d.helplist.delete(0, 'end')
- d.helplist.insert(0, 'name1')
- d.helplist.selection_set(0)
- d.helplist.selection_anchor(0)
- d.user_helplist.clear()
- d.user_helplist.append(('name1', 'file1'))
- d.set.called = d.upc.called = 0
+ fr = self.frame
+ fr.helplist.delete(0, 'end')
+ fr.helplist.insert(0, 'name1')
+ fr.helplist.selection_set(0)
+ fr.helplist.selection_anchor(0)
+ fr.user_helplist.clear()
+ fr.user_helplist.append(('name1', 'file1'))
+ fr.set.called = fr.upc.called = 0
hs.result = ''
- d.helplist_item_edit()
+ fr.helplist_item_edit()
hs.result = ('name1', 'file1')
- d.helplist_item_edit()
- eq(d.helplist.get(0, 'end'), ('name1',))
- eq(d.user_helplist, [('name1', 'file1')])
- self.assertFalse(d.upc.called)
+ fr.helplist_item_edit()
+ eq(fr.helplist.get(0, 'end'), ('name1',))
+ eq(fr.user_helplist, [('name1', 'file1')])
+ self.assertFalse(fr.upc.called)
hs.result = ('name2', 'file2')
- d.helplist_item_edit()
- eq(d.helplist.get(0, 'end'), ('name2',))
- eq(d.user_helplist, [('name2', 'file2')])
- self.assertTrue(d.upc.called == d.set.called == 1)
+ fr.helplist_item_edit()
+ eq(fr.helplist.get(0, 'end'), ('name2',))
+ eq(fr.user_helplist, [('name2', 'file2')])
+ self.assertTrue(fr.upc.called == fr.set.called == 1)
configdialog.HelpSource = orig_helpsource
def test_helplist_item_remove(self):
eq = self.assertEqual
- d = self.page
- d.helplist.delete(0, 'end')
- d.helplist.insert(0, 'name1')
- d.helplist.selection_set(0)
- d.helplist.selection_anchor(0)
- d.user_helplist.clear()
- d.user_helplist.append(('name1', 'file1'))
- d.set.called = d.upc.called = 0
-
- d.helplist_item_remove()
- eq(d.helplist.get(0, 'end'), ())
- eq(d.user_helplist, [])
- self.assertTrue(d.upc.called == d.set.called == 1)
+ fr = self.frame
+ fr.helplist.delete(0, 'end')
+ fr.helplist.insert(0, 'name1')
+ fr.helplist.selection_set(0)
+ fr.helplist.selection_anchor(0)
+ fr.user_helplist.clear()
+ fr.user_helplist.append(('name1', 'file1'))
+ fr.set.called = fr.upc.called = 0
+
+ fr.helplist_item_remove()
+ eq(fr.helplist.get(0, 'end'), ())
+ eq(fr.user_helplist, [])
+ self.assertTrue(fr.upc.called == fr.set.called == 1)
def test_update_help_changes(self):
- d = self.page
- del d.update_help_changes
- d.user_helplist.clear()
- d.user_helplist.append(('name1', 'file1'))
- d.user_helplist.append(('name2', 'file2'))
+ fr = self.frame
+ del fr.update_help_changes
+ fr.user_helplist.clear()
+ fr.user_helplist.append(('name1', 'file1'))
+ fr.user_helplist.append(('name2', 'file2'))
- d.update_help_changes()
+ fr.update_help_changes()
self.assertEqual(mainpage['HelpFiles'],
{'1': 'name1;file1', '2': 'name2;file2'})
- d.update_help_changes = Func()
+ fr.update_help_changes = Func()
class VarTraceTest(unittest.TestCase):
diff --git a/Lib/idlelib/idle_test/test_debugger_r.py b/Lib/idlelib/idle_test/test_debugger_r.py
index 638ebd36a7405d..cf8af05fe27e77 100644
--- a/Lib/idlelib/idle_test/test_debugger_r.py
+++ b/Lib/idlelib/idle_test/test_debugger_r.py
@@ -2,28 +2,21 @@
from idlelib import debugger_r
import unittest
-from test.support import requires
-from tkinter import Tk
-
-
-class Test(unittest.TestCase):
+# Boilerplate likely to be needed for future test classes.
+##from test.support import requires
+##from tkinter import Tk
+##class Test(unittest.TestCase):
## @classmethod
## def setUpClass(cls):
## requires('gui')
## cls.root = Tk()
-##
## @classmethod
## def tearDownClass(cls):
## cls.root.destroy()
-## del cls.root
-
- def test_init(self):
- self.assertTrue(True) # Get coverage of import
-
-# Classes GUIProxy, IdbAdapter, FrameProxy, CodeProxy, DictProxy,
-# GUIAdapter, IdbProxy plus 7 module functions.
+# GUIProxy, IdbAdapter, FrameProxy, CodeProxy, DictProxy,
+# GUIAdapter, IdbProxy, and 7 functions still need tests.
class IdbAdapterTest(unittest.TestCase):
diff --git a/Lib/idlelib/idle_test/test_editor.py b/Lib/idlelib/idle_test/test_editor.py
index 8665d680c0118f..fdb47abf43fb77 100644
--- a/Lib/idlelib/idle_test/test_editor.py
+++ b/Lib/idlelib/idle_test/test_editor.py
@@ -5,7 +5,6 @@
from collections import namedtuple
from test.support import requires
from tkinter import Tk
-from idlelib.idle_test.mock_idle import Func
Editor = editor.EditorWindow
diff --git a/Lib/idlelib/idle_test/test_iomenu.py b/Lib/idlelib/idle_test/test_iomenu.py
index 99f40487967124..2fb836dba21672 100644
--- a/Lib/idlelib/idle_test/test_iomenu.py
+++ b/Lib/idlelib/idle_test/test_iomenu.py
@@ -5,6 +5,8 @@
from test.support import requires
from tkinter import Tk
from idlelib.editor import EditorWindow
+from idlelib import util
+from idlelib.idle_test.mock_idle import Func
class IOBindingTest(unittest.TestCase):
@@ -36,14 +38,41 @@ def test_fixnewlines_end(self):
io = self.io
fix = io.fixnewlines
text = io.editwin.text
+
+ # Make the editor temporarily look like Shell.
self.editwin.interp = None
- eq(fix(), '')
- del self.editwin.interp
+ shelltext = '>>> if 1'
+ self.editwin.get_prompt_text = Func(result=shelltext)
+ eq(fix(), shelltext) # Get... call and '\n' not added.
+ del self.editwin.interp, self.editwin.get_prompt_text
+
text.insert(1.0, 'a')
eq(fix(), 'a'+io.eol_convention)
eq(text.get('1.0', 'end-1c'), 'a\n')
eq(fix(), 'a'+io.eol_convention)
+def _extension_in_filetypes(extension):
+ return any(
+ f'*{extension}' in filetype_tuple[1]
+ for filetype_tuple in iomenu.IOBinding.filetypes
+ )
+
+
+class FiletypesTest(unittest.TestCase):
+ def test_python_source_files(self):
+ for extension in util.py_extensions:
+ with self.subTest(extension=extension):
+ self.assertTrue(
+ _extension_in_filetypes(extension)
+ )
+
+ def test_text_files(self):
+ self.assertTrue(_extension_in_filetypes('.txt'))
+
+ def test_all_files(self):
+ self.assertTrue(_extension_in_filetypes(''))
+
+
if __name__ == '__main__':
unittest.main(verbosity=2)
diff --git a/Lib/idlelib/idle_test/test_macosx.py b/Lib/idlelib/idle_test/test_macosx.py
index b6bd922e4b99dd..86da8849e5ca00 100644
--- a/Lib/idlelib/idle_test/test_macosx.py
+++ b/Lib/idlelib/idle_test/test_macosx.py
@@ -12,6 +12,15 @@
alltypes = mactypes | nontypes
+def setUpModule():
+ global orig_tktype
+ orig_tktype = macosx._tk_type
+
+
+def tearDownModule():
+ macosx._tk_type = orig_tktype
+
+
class InitTktypeTest(unittest.TestCase):
"Test _init_tk_type."
@@ -34,7 +43,7 @@ def test_init_sets_tktype(self):
for platform, types in ('darwin', alltypes), ('other', nontypes):
with self.subTest(platform=platform):
macosx.platform = platform
- macosx._tk_type == None
+ macosx._tk_type = None
macosx._init_tk_type()
self.assertIn(macosx._tk_type, types)
diff --git a/Lib/idlelib/idle_test/test_multicall.py b/Lib/idlelib/idle_test/test_multicall.py
index ba582bb3ca51b4..b3a3bfb88f9c31 100644
--- a/Lib/idlelib/idle_test/test_multicall.py
+++ b/Lib/idlelib/idle_test/test_multicall.py
@@ -37,7 +37,7 @@ def test_init(self):
def test_yview(self):
# Added for tree.wheel_event
- # (it depends on yview to not be overriden)
+ # (it depends on yview to not be overridden)
mc = self.mc
self.assertIs(mc.yview, Text.yview)
mctext = self.mc(self.root)
diff --git a/Lib/idlelib/idle_test/test_parenmatch.py b/Lib/idlelib/idle_test/test_parenmatch.py
index 4a41d8433d5483..2e10d7cd36760f 100644
--- a/Lib/idlelib/idle_test/test_parenmatch.py
+++ b/Lib/idlelib/idle_test/test_parenmatch.py
@@ -83,12 +83,12 @@ def test_paren_corner(self):
"""
Test corner cases in flash_paren_event and paren_closed_event.
- These cases force conditional expression and alternate paths.
+ Force execution of conditional expressions and alternate paths.
"""
text = self.text
pm = self.get_parenmatch()
- text.insert('insert', '# this is a commen)')
+ text.insert('insert', '# Comment.)')
pm.paren_closed_event('event')
text.insert('insert', '\ndef')
diff --git a/Lib/idlelib/idle_test/test_pyparse.py b/Lib/idlelib/idle_test/test_pyparse.py
index fb5726db1d821e..384db566ac76cd 100644
--- a/Lib/idlelib/idle_test/test_pyparse.py
+++ b/Lib/idlelib/idle_test/test_pyparse.py
@@ -284,7 +284,7 @@ def test_get_num_lines_in_stmt(self):
tests = (
TestInfo('[x for x in a]\n', 1), # Closed on one line.
TestInfo('[x\nfor x in a\n', 2), # Not closed.
- TestInfo('[x\\\nfor x in a\\\n', 2), # "", uneeded backslashes.
+ TestInfo('[x\\\nfor x in a\\\n', 2), # "", unneeded backslashes.
TestInfo('[x\nfor x in a\n]\n', 3), # Closed on multi-line.
TestInfo('\n"""Docstring comment L1"""\nL2\nL3\nL4\n', 1),
TestInfo('\n"""Docstring comment L1\nL2"""\nL3\nL4\n', 1),
diff --git a/Lib/idlelib/idle_test/test_query.py b/Lib/idlelib/idle_test/test_query.py
index e968862688b95e..bb12b2b08652d5 100644
--- a/Lib/idlelib/idle_test/test_query.py
+++ b/Lib/idlelib/idle_test/test_query.py
@@ -1,4 +1,4 @@
-"""Test query, coverage 93%).
+"""Test query, coverage 93%.
Non-gui tests for Query, SectionName, ModuleName, and HelpSource use
dummy versions that extract the non-gui methods and add other needed
@@ -136,8 +136,8 @@ def test_good_module_name(self):
dialog = self.Dummy_ModuleName('idlelib')
self.assertTrue(dialog.entry_ok().endswith('__init__.py'))
self.assertEqual(dialog.entry_error['text'], '')
- dialog = self.Dummy_ModuleName('os.path')
- self.assertTrue(dialog.entry_ok().endswith('path.py'))
+ dialog = self.Dummy_ModuleName('idlelib.idle')
+ self.assertTrue(dialog.entry_ok().endswith('idle.py'))
self.assertEqual(dialog.entry_error['text'], '')
diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py
index a31671ee0485fa..ec4637c5ca617a 100644
--- a/Lib/idlelib/idle_test/test_run.py
+++ b/Lib/idlelib/idle_test/test_run.py
@@ -1,4 +1,4 @@
-"Test run, coverage 49%."
+"Test run, coverage 54%."
from idlelib import run
import io
@@ -12,7 +12,7 @@
idlelib.testing = True # Use {} for executing test user code.
-class PrintExceptionTest(unittest.TestCase):
+class ExceptionTest(unittest.TestCase):
def test_print_exception_unhashable(self):
class UnhashableException(Exception):
@@ -28,8 +28,7 @@ def __eq__(self, other):
raise ex1
except UnhashableException:
with captured_stderr() as output:
- with mock.patch.object(run,
- 'cleanup_traceback') as ct:
+ with mock.patch.object(run, 'cleanup_traceback') as ct:
ct.side_effect = lambda t, e: t
run.print_exception()
@@ -38,6 +37,46 @@ def __eq__(self, other):
self.assertIn('UnhashableException: ex2', tb[3])
self.assertIn('UnhashableException: ex1', tb[10])
+ data = (('1/0', ZeroDivisionError, "division by zero\n"),
+ ('abc', NameError, "name 'abc' is not defined. "
+ "Did you mean: 'abs'?\n"),
+ ('int.reel', AttributeError,
+ "type object 'int' has no attribute 'reel'. "
+ "Did you mean: 'real'?\n"),
+ )
+
+ def test_get_message(self):
+ for code, exc, msg in self.data:
+ with self.subTest(code=code):
+ try:
+ eval(compile(code, '', 'eval'))
+ except exc:
+ typ, val, tb = sys.exc_info()
+ actual = run.get_message_lines(typ, val, tb)[0]
+ expect = f'{exc.__name__}: {msg}'
+ self.assertEqual(actual, expect)
+
+ @mock.patch.object(run, 'cleanup_traceback',
+ new_callable=lambda: (lambda t, e: None))
+ def test_get_multiple_message(self, mock):
+ d = self.data
+ data2 = ((d[0], d[1]), (d[1], d[2]), (d[2], d[0]))
+ subtests = 0
+ for (code1, exc1, msg1), (code2, exc2, msg2) in data2:
+ with self.subTest(codes=(code1,code2)):
+ try:
+ eval(compile(code1, '', 'eval'))
+ except exc1:
+ try:
+ eval(compile(code2, '', 'eval'))
+ except exc2:
+ with captured_stderr() as output:
+ run.print_exception()
+ actual = output.getvalue()
+ self.assertIn(msg1, actual)
+ self.assertIn(msg2, actual)
+ subtests += 1
+ self.assertEqual(subtests, len(data2)) # All subtests ran?
# StdioFile tests.
diff --git a/Lib/idlelib/idle_test/test_sidebar.py b/Lib/idlelib/idle_test/test_sidebar.py
index 43e8137d7079c9..049531e66a414e 100644
--- a/Lib/idlelib/idle_test/test_sidebar.py
+++ b/Lib/idlelib/idle_test/test_sidebar.py
@@ -6,6 +6,7 @@
import unittest
import unittest.mock
from test.support import requires, swap_attr
+from test import support
import tkinter as tk
from idlelib.idle_test.tkinter_testing_utils import run_in_tk_mainloop
@@ -474,10 +475,7 @@ def get_shell_line_y_coords(self):
index = text.index("@0,0")
if index.split('.', 1)[1] != '0':
index = text.index(f"{index} +1line linestart")
- while True:
- lineinfo = text.dlineinfo(index)
- if lineinfo is None:
- break
+ while (lineinfo := text.dlineinfo(index)) is not None:
y_coords.append(lineinfo[1])
index = text.index(f"{index} +1line")
return y_coords
@@ -510,19 +508,19 @@ def test_initial_state(self):
)
self.assert_sidebar_lines_synced()
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_single_empty_input(self):
self.do_input('\n')
yield
self.assert_sidebar_lines_end_with(['>>>', '>>>'])
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_single_line_statement(self):
self.do_input('1\n')
yield
self.assert_sidebar_lines_end_with(['>>>', None, '>>>'])
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_multi_line_statement(self):
# Block statements are not indented because IDLE auto-indents.
self.do_input(dedent('''\
@@ -540,14 +538,14 @@ def test_multi_line_statement(self):
'>>>',
])
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_single_long_line_wraps(self):
self.do_input('1' * 200 + '\n')
yield
self.assert_sidebar_lines_end_with(['>>>', None, '>>>'])
self.assert_sidebar_lines_synced()
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_squeeze_multi_line_output(self):
shell = self.shell
text = shell.text
@@ -567,7 +565,7 @@ def test_squeeze_multi_line_output(self):
self.assert_sidebar_lines_end_with(['>>>', None, None, None, '>>>'])
self.assert_sidebar_lines_synced()
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_interrupt_recall_undo_redo(self):
text = self.shell.text
# Block statements are not indented because IDLE auto-indents.
@@ -613,9 +611,10 @@ def test_interrupt_recall_undo_redo(self):
['>>>', '...', '...', '...', None, '>>>']
)
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_very_long_wrapped_line(self):
- with swap_attr(self.shell, 'squeezer', None):
+ with support.adjust_int_max_str_digits(11_111), \
+ swap_attr(self.shell, 'squeezer', None):
self.do_input('x = ' + '1'*10_000 + '\n')
yield
self.assertEqual(self.get_sidebar_lines(), ['>>>'])
@@ -678,7 +677,7 @@ def get_sidebar_colors():
sidebar.update_colors()
self.assertEqual(get_sidebar_colors(), test_colors)
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_mousewheel(self):
sidebar = self.shell.shell_sidebar
text = self.shell.text
@@ -703,7 +702,7 @@ def test_mousewheel(self):
yield
self.assertIsNotNone(text.dlineinfo(text.index(f'{last_lineno}.0')))
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_copy(self):
sidebar = self.shell.shell_sidebar
text = self.shell.text
@@ -728,7 +727,7 @@ def test_copy(self):
copied_text = text.clipboard_get()
self.assertEqual(copied_text, selected_text)
- @run_in_tk_mainloop
+ @run_in_tk_mainloop()
def test_copy_with_prompts(self):
sidebar = self.shell.shell_sidebar
text = self.shell.text
@@ -736,7 +735,7 @@ def test_copy_with_prompts(self):
first_line = get_end_linenumber(text)
self.do_input(dedent('''\
if True:
- print(1)
+ print(1)
'''))
yield
@@ -747,9 +746,10 @@ def test_copy_with_prompts(self):
selected_lines_text = text.get('sel.first linestart', 'sel.last')
selected_lines = selected_lines_text.split('\n')
- # Expect a block of input, a single output line, and a new prompt
+ selected_lines.pop() # Final '' is a split artifact, not a line.
+ # Expect a block of input and a single output line.
expected_prompts = \
- ['>>>'] + ['...'] * (len(selected_lines) - 3) + [None, '>>>']
+ ['>>>'] + ['...'] * (len(selected_lines) - 2) + [None]
selected_text_with_prompts = '\n'.join(
line if prompt is None else prompt + ' ' + line
for prompt, line in zip(expected_prompts,
diff --git a/Lib/idlelib/idle_test/test_squeezer.py b/Lib/idlelib/idle_test/test_squeezer.py
index 86c5d41b629719..86c21f00bb8d00 100644
--- a/Lib/idlelib/idle_test/test_squeezer.py
+++ b/Lib/idlelib/idle_test/test_squeezer.py
@@ -170,6 +170,7 @@ def test_write_not_stdout(self):
def test_write_stdout(self):
"""Test Squeezer's overriding of the EditorWindow's write() method."""
+ requires('gui')
editwin = self.make_mock_editor_window()
for text in ['', 'TEXT']:
diff --git a/Lib/idlelib/idle_test/test_text.py b/Lib/idlelib/idle_test/test_text.py
index 0f31179e04b28f..43a9ba02c3d3c9 100644
--- a/Lib/idlelib/idle_test/test_text.py
+++ b/Lib/idlelib/idle_test/test_text.py
@@ -6,7 +6,7 @@
from test.support import requires
from _tkinter import TclError
-class TextTest(object):
+class TextTest:
"Define items common to both sets of tests."
hw = 'hello\nworld' # Several tests insert this after initialization.
diff --git a/Lib/idlelib/idle_test/test_util.py b/Lib/idlelib/idle_test/test_util.py
new file mode 100644
index 00000000000000..20721fe980c784
--- /dev/null
+++ b/Lib/idlelib/idle_test/test_util.py
@@ -0,0 +1,14 @@
+"""Test util, coverage 100%"""
+
+import unittest
+from idlelib import util
+
+
+class UtilTest(unittest.TestCase):
+ def test_extensions(self):
+ for extension in {'.pyi', '.py', '.pyw'}:
+ self.assertIn(extension, util.py_extensions)
+
+
+if __name__ == '__main__':
+ unittest.main(verbosity=2)
diff --git a/Lib/idlelib/idle_test/test_zzdummy.py b/Lib/idlelib/idle_test/test_zzdummy.py
index 1013cdc3c46f4f..209d8564da0664 100644
--- a/Lib/idlelib/idle_test/test_zzdummy.py
+++ b/Lib/idlelib/idle_test/test_zzdummy.py
@@ -19,7 +19,7 @@
}
code_sample = """\
-class C1():
+class C1:
# Class comment.
def __init__(self, a, b):
self.a = a
diff --git a/Lib/idlelib/idle_test/tkinter_testing_utils.py b/Lib/idlelib/idle_test/tkinter_testing_utils.py
index a9f8386e2cd9f6..a89839bbe38add 100644
--- a/Lib/idlelib/idle_test/tkinter_testing_utils.py
+++ b/Lib/idlelib/idle_test/tkinter_testing_utils.py
@@ -2,7 +2,7 @@
import functools
-def run_in_tk_mainloop(test_method):
+def run_in_tk_mainloop(delay=1):
"""Decorator for running a test method with a real Tk mainloop.
This starts a Tk mainloop before running the test, and stops it
@@ -13,44 +13,50 @@ def run_in_tk_mainloop(test_method):
using "yield" to allow the mainloop to process events and "after"
callbacks, and then continue the test from that point.
+ The delay argument is passed into root.after(...) calls as the number
+ of ms to wait before passing execution back to the generator function.
+
This also assumes that the test class has a .root attribute,
which is a tkinter.Tk object.
For example (from test_sidebar.py):
- @run_test_with_tk_mainloop
+ @run_test_with_tk_mainloop()
def test_single_empty_input(self):
self.do_input('\n')
yield
self.assert_sidebar_lines_end_with(['>>>', '>>>'])
"""
- @functools.wraps(test_method)
- def new_test_method(self):
- test_generator = test_method(self)
- root = self.root
- # Exceptions raised by self.assert...() need to be raised
- # outside of the after() callback in order for the test
- # harness to capture them.
- exception = None
- def after_callback():
- nonlocal exception
- try:
- next(test_generator)
- except StopIteration:
- root.quit()
- except Exception as exc:
- exception = exc
- root.quit()
- else:
- # Schedule the Tk mainloop to call this function again,
- # using a robust method of ensuring that it gets a
- # chance to process queued events before doing so.
- # See: https://stackoverflow.com/q/18499082#comment65004099_38817470
- root.after(1, root.after_idle, after_callback)
- root.after(0, root.after_idle, after_callback)
- root.mainloop()
-
- if exception:
- raise exception
-
- return new_test_method
+ def decorator(test_method):
+ @functools.wraps(test_method)
+ def new_test_method(self):
+ test_generator = test_method(self)
+ root = self.root
+ # Exceptions raised by self.assert...() need to be raised
+ # outside of the after() callback in order for the test
+ # harness to capture them.
+ exception = None
+ def after_callback():
+ nonlocal exception
+ try:
+ next(test_generator)
+ except StopIteration:
+ root.quit()
+ except Exception as exc:
+ exception = exc
+ root.quit()
+ else:
+ # Schedule the Tk mainloop to call this function again,
+ # using a robust method of ensuring that it gets a
+ # chance to process queued events before doing so.
+ # See: https://stackoverflow.com/q/18499082#comment65004099_38817470
+ root.after(delay, root.after_idle, after_callback)
+ root.after(0, root.after_idle, after_callback)
+ root.mainloop()
+
+ if exception:
+ raise exception
+
+ return new_test_method
+
+ return decorator
diff --git a/Lib/idlelib/iomenu.py b/Lib/idlelib/iomenu.py
index 5ebf7089fb9abe..af8159c2b33f51 100644
--- a/Lib/idlelib/iomenu.py
+++ b/Lib/idlelib/iomenu.py
@@ -9,15 +9,12 @@
from tkinter import messagebox
from tkinter.simpledialog import askstring
-import idlelib
from idlelib.config import idleConf
+from idlelib.util import py_extensions
+py_extensions = ' '.join("*"+ext for ext in py_extensions)
encoding = 'utf-8'
-if sys.platform == 'win32':
- errors = 'surrogatepass'
-else:
- errors = 'surrogateescape'
-
+errors = 'surrogatepass' if sys.platform == 'win32' else 'surrogateescape'
class IOBinding:
@@ -254,11 +251,17 @@ def writefile(self, filename):
return False
def fixnewlines(self):
- "Return text with final \n if needed and os eols."
- if (self.text.get("end-2c") != '\n'
- and not hasattr(self.editwin, "interp")): # Not shell.
- self.text.insert("end-1c", "\n")
- text = self.text.get("1.0", "end-1c")
+ """Return text with os eols.
+
+ Add prompts if shell else final \n if missing.
+ """
+
+ if hasattr(self.editwin, "interp"): # Saving shell.
+ text = self.editwin.get_prompt_text('1.0', self.text.index('end-1c'))
+ else:
+ if self.text.get("end-2c") != '\n':
+ self.text.insert("end-1c", "\n") # Changes 'end-1c' value.
+ text = self.text.get('1.0', "end-1c")
if self.eol_convention != "\n":
text = text.replace("\n", self.eol_convention)
return text
@@ -348,7 +351,7 @@ def print_window(self, event):
savedialog = None
filetypes = (
- ("Python files", "*.py *.pyw", "TEXT"),
+ ("Python files", py_extensions, "TEXT"),
("Text files", "*.txt", "TEXT"),
("All files", "*"),
)
diff --git a/Lib/idlelib/macosx.py b/Lib/idlelib/macosx.py
index eeaab59ae80295..f53bd58970e64d 100644
--- a/Lib/idlelib/macosx.py
+++ b/Lib/idlelib/macosx.py
@@ -14,12 +14,25 @@
_tk_type = None
def _init_tk_type():
- """
- Initializes OS X Tk variant values for
- isAquaTk(), isCarbonTk(), isCocoaTk(), and isXQuartz().
+ """ Initialize _tk_type for isXyzTk functions.
+
+ This function is only called once, when _tk_type is still None.
"""
global _tk_type
if platform == 'darwin':
+
+ # When running IDLE, GUI is present, test/* may not be.
+ # When running tests, test/* is present, GUI may not be.
+ # If not, guess most common. Does not matter for testing.
+ from idlelib.__init__ import testing
+ if testing:
+ from test.support import requires, ResourceDenied
+ try:
+ requires('gui')
+ except ResourceDenied:
+ _tk_type = "cocoa"
+ return
+
root = tkinter.Tk()
ws = root.tk.call('tk', 'windowingsystem')
if 'x11' in ws:
@@ -33,6 +46,7 @@ def _init_tk_type():
root.destroy()
else:
_tk_type = "other"
+ return
def isAquaTk():
"""
@@ -83,7 +97,7 @@ def tkVersionWarning(root):
return False
return ("WARNING: The version of Tcl/Tk ({0}) in use may"
" be unstable.\n"
- "Visit http://www.python.org/download/mac/tcltk/"
+ "Visit https://www.python.org/download/mac/tcltk/"
" for current information.".format(patchlevel))
else:
return False
diff --git a/Lib/idlelib/mainmenu.py b/Lib/idlelib/mainmenu.py
index 74edce23483829..91a32cebb513f9 100644
--- a/Lib/idlelib/mainmenu.py
+++ b/Lib/idlelib/mainmenu.py
@@ -34,24 +34,25 @@
None,
('Prin_t Window', '<>'),
None,
- ('_Close', '<>'),
- ('E_xit', '<>'),
+ ('_Close Window', '<>'),
+ ('E_xit IDLE', '<>'),
]),
('edit', [
('_Undo', '<>'),
('_Redo', '<>'),
None,
+ ('Select _All', '<>'),
('Cu_t', '<>'),
('_Copy', '<>'),
('_Paste', '<>'),
- ('Select _All', '<>'),
None,
('_Find...', '<>'),
('Find A_gain', '<>'),
('Find _Selection', '<>'),
('Find in Files...', '<>'),
('R_eplace...', '<>'),
+ None,
('Go to _Line', '<>'),
('S_how Completions', '<>'),
('E_xpand Word', '<>'),
@@ -110,7 +111,7 @@
('help', [
('_About IDLE', '<>'),
None,
- ('_IDLE Help', '<>'),
+ ('_IDLE Doc', '<>'),
('Python _Docs', '<>'),
]),
]
diff --git a/Lib/idlelib/pyparse.py b/Lib/idlelib/pyparse.py
index d34872b4396e1e..8545c63e1435d4 100644
--- a/Lib/idlelib/pyparse.py
+++ b/Lib/idlelib/pyparse.py
@@ -179,14 +179,10 @@ def find_good_parse_start(self, is_char_in_string):
# Peeking back worked; look forward until _synchre no longer
# matches.
i = pos + 1
- while 1:
- m = _synchre(code, i)
- if m:
- s, i = m.span()
- if not is_char_in_string(s):
- pos = s
- else:
- break
+ while m := _synchre(code, i):
+ s, i = m.span()
+ if not is_char_in_string(s):
+ pos = s
return pos
def set_lo(self, lo):
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index 4e7440038ac997..e7fef5f69daa30 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -66,6 +66,13 @@
HOST = '127.0.0.1' # python execution server on localhost loopback
PORT = 0 # someday pass in host, port for remote debug capability
+try: # In case IDLE started with -n.
+ eof = 'Ctrl-D (end-of-file)'
+ exit.eof = eof
+ quit.eof = eof
+except NameError: # In case python started with -S.
+ pass
+
# Override warnings module to write to warning_stream. Initialize to send IDLE
# internal warnings to the console. ScriptBinding.check_syntax() will
# temporarily redirect the stream to the shell window to display warnings when
@@ -988,6 +995,23 @@ def replace_event(self, event):
def get_standard_extension_names(self):
return idleConf.GetExtensions(shell_only=True)
+ def get_prompt_text(self, first, last):
+ """Return text between first and last with prompts added."""
+ text = self.text.get(first, last)
+ lineno_range = range(
+ int(float(first)),
+ int(float(last))
+ )
+ prompts = [
+ self.shell_sidebar.line_prompts.get(lineno)
+ for lineno in lineno_range
+ ]
+ return "\n".join(
+ line if prompt is None else f"{prompt} {line}"
+ for prompt, line in zip(prompts, text.splitlines())
+ ) + "\n"
+
+
def copy_with_prompts_callback(self, event=None):
"""Copy selected lines to the clipboard, with prompts.
@@ -998,31 +1022,15 @@ def copy_with_prompts_callback(self, event=None):
and/or last lines is selected.
"""
text = self.text
-
- selection_indexes = (
- self.text.index("sel.first linestart"),
- self.text.index("sel.last +1line linestart"),
- )
- if selection_indexes[0] is None:
- # There is no selection, so do nothing.
- return
-
- selected_text = self.text.get(*selection_indexes)
- selection_lineno_range = range(
- int(float(selection_indexes[0])),
- int(float(selection_indexes[1]))
- )
- prompts = [
- self.shell_sidebar.line_prompts.get(lineno)
- for lineno in selection_lineno_range
- ]
- selected_text_with_prompts = "\n".join(
- line if prompt is None else f"{prompt} {line}"
- for prompt, line in zip(prompts, selected_text.splitlines())
- ) + "\n"
-
+ selfirst = text.index('sel.first linestart')
+ if selfirst is None: # Should not be possible.
+ return # No selection, do nothing.
+ sellast = text.index('sel.last')
+ if sellast[-1] != '0':
+ sellast = text.index("sel.last+1line linestart")
text.clipboard_clear()
- text.clipboard_append(selected_text_with_prompts)
+ prompt_text = self.get_prompt_text(selfirst, sellast)
+ text.clipboard_append(prompt_text)
reading = False
executing = False
diff --git a/Lib/idlelib/query.py b/Lib/idlelib/query.py
index fefa5aac1b7f58..df02f2123ab02f 100644
--- a/Lib/idlelib/query.py
+++ b/Lib/idlelib/query.py
@@ -83,6 +83,7 @@ def __init__(self, parent, title, message, *, text0='', used_names={},
if not _utest:
self.deiconify() # Unhide now that geometry set.
+ self.entry.focus_set()
self.wait_window()
def create_widgets(self, ok_text='OK'): # Do not replace.
@@ -100,7 +101,6 @@ def create_widgets(self, ok_text='OK'): # Do not replace.
text=self.message)
self.entryvar = StringVar(self, self.text0)
self.entry = Entry(frame, width=30, textvariable=self.entryvar)
- self.entry.focus_set()
self.error_font = Font(name='TkCaptionFont',
exists=True, root=self.parent)
self.entry_error = Label(frame, text=' ', foreground='red',
diff --git a/Lib/idlelib/replace.py b/Lib/idlelib/replace.py
index 2f9ca231a05e49..ca83173877ad1d 100644
--- a/Lib/idlelib/replace.py
+++ b/Lib/idlelib/replace.py
@@ -158,11 +158,8 @@ def replace_all(self, event=None):
first = last = None
# XXX ought to replace circular instead of top-to-bottom when wrapping
text.undo_block_start()
- while True:
- res = self.engine.search_forward(text, prog, line, col,
- wrap=False, ok=ok)
- if not res:
- break
+ while res := self.engine.search_forward(
+ text, prog, line, col, wrap=False, ok=ok):
line, m = res
chars = text.get("%d.0" % line, "%d.0" % (line+1))
orig = m.group()
diff --git a/Lib/idlelib/rpc.py b/Lib/idlelib/rpc.py
index 8efcf048fa3aa2..62eec84c9c8d09 100644
--- a/Lib/idlelib/rpc.py
+++ b/Lib/idlelib/rpc.py
@@ -307,7 +307,7 @@ def _getresponse(self, myseq, wait):
self.debug("_getresponse:myseq:", myseq)
if threading.current_thread() is self.sockthread:
# this thread does all reading of requests or responses
- while 1:
+ while True:
response = self.pollresponse(myseq, wait)
if response is not None:
return response
@@ -417,7 +417,7 @@ def pollresponse(self, myseq, wait):
self.responses and notify the owning thread.
"""
- while 1:
+ while True:
# send queued response if there is one available
try:
qmsg = response_queue.get(0)
diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py
index 07e9a2bf9ceeae..577c49eb67b20d 100644
--- a/Lib/idlelib/run.py
+++ b/Lib/idlelib/run.py
@@ -4,6 +4,7 @@
f'''{sys.executable} -c "__import__('idlelib.run').run.main()"'''
'.run' is needed because __import__ returns idlelib, not idlelib.run.
"""
+import contextlib
import functools
import io
import linecache
@@ -39,6 +40,13 @@
LOCALHOST = '127.0.0.1'
+try:
+ eof = 'Ctrl-D (end-of-file)'
+ exit.eof = eof
+ quit.eof = eof
+except NameError: # In case subprocess started with -S (maybe in future).
+ pass
+
def idle_formatwarning(message, category, filename, lineno, line=None):
"""Format warnings the IDLE way."""
@@ -137,7 +145,7 @@ def main(del_exitfunc=False):
args=((LOCALHOST, port),))
sockthread.daemon = True
sockthread.start()
- while 1:
+ while True:
try:
if exit_now:
try:
@@ -211,6 +219,19 @@ def show_socket_error(err, address):
parent=root)
root.destroy()
+
+def get_message_lines(typ, exc, tb):
+ "Return line composing the exception message."
+ if typ in (AttributeError, NameError):
+ # 3.10+ hints are not directly accessible from python (#44026).
+ err = io.StringIO()
+ with contextlib.redirect_stderr(err):
+ sys.__excepthook__(typ, exc, tb)
+ return [err.getvalue().split("\n")[-2] + "\n"]
+ else:
+ return traceback.format_exception_only(typ, exc)
+
+
def print_exception():
import linecache
linecache.checkcache()
@@ -241,7 +262,7 @@ def print_exc(typ, exc, tb):
"debugger_r.py", "bdb.py")
cleanup_traceback(tbe, exclude)
traceback.print_list(tbe, file=efile)
- lines = traceback.format_exception_only(typ, exc)
+ lines = get_message_lines(typ, exc, tb)
for line in lines:
print(line, end='', file=efile)
@@ -461,9 +482,7 @@ def read(self, size=-1):
result = self._line_buffer
self._line_buffer = ''
if size < 0:
- while True:
- line = self.shell.readline()
- if not line: break
+ while line := self.shell.readline():
result += line
else:
while len(result) < size:
diff --git a/Lib/idlelib/searchengine.py b/Lib/idlelib/searchengine.py
index eddef581ab40a7..0684142f43644a 100644
--- a/Lib/idlelib/searchengine.py
+++ b/Lib/idlelib/searchengine.py
@@ -165,7 +165,7 @@ def search_backward(self, text, prog, line, col, wrap, ok=0):
wrapped = 0
startline = line
chars = text.get("%d.0" % line, "%d.0" % (line+1))
- while 1:
+ while True:
m = search_reverse(prog, chars[:-1], col)
if m:
if ok or m.start() < col:
diff --git a/Lib/idlelib/sidebar.py b/Lib/idlelib/sidebar.py
index 018c368f421c6c..fb1084dbf3f18b 100644
--- a/Lib/idlelib/sidebar.py
+++ b/Lib/idlelib/sidebar.py
@@ -471,10 +471,7 @@ def update_sidebar(self):
index = text.index("@0,0")
if index.split('.', 1)[1] != '0':
index = text.index(f'{index}+1line linestart')
- while True:
- lineinfo = text.dlineinfo(index)
- if lineinfo is None:
- break
+ while (lineinfo := text.dlineinfo(index)) is not None:
y = lineinfo[1]
prev_newline_tagnames = text_tagnames(f"{index} linestart -1c")
prompt = (
diff --git a/Lib/idlelib/util.py b/Lib/idlelib/util.py
new file mode 100644
index 00000000000000..ede670a4db5536
--- /dev/null
+++ b/Lib/idlelib/util.py
@@ -0,0 +1,21 @@
+"""
+Idlelib objects with no external idlelib dependencies
+which are needed in more than one idlelib module.
+
+They are included here because
+ a) they don't particularly belong elsewhere; or
+ b) because inclusion here simplifies the idlelib dependency graph.
+
+TODO:
+ * Python versions (editor and help_about),
+ * tk version and patchlevel (pyshell, help_about, maxos?, editor?),
+ * std streams (pyshell, run),
+ * warning stuff (pyshell, run).
+"""
+
+# .pyw is for Windows; .pyi is for stub files.
+py_extensions = ('.py', '.pyw', '.pyi') # Order needed for open/save dialogs.
+
+if __name__ == '__main__':
+ from unittest import main
+ main('idlelib.idle_test.test_util', verbosity=2)
diff --git a/Lib/imaplib.py b/Lib/imaplib.py
index 73184396d894a3..54122f985b9aca 100644
--- a/Lib/imaplib.py
+++ b/Lib/imaplib.py
@@ -52,6 +52,9 @@
# search command can be quite large, so we now use 1M.
_MAXLINE = 1000000
+# Data larger than this will be read in chunks, to prevent extreme
+# overallocation.
+_SAFE_BUF_SIZE = 1 << 20
# Commands
@@ -315,7 +318,13 @@ def open(self, host='', port=IMAP4_PORT, timeout=None):
def read(self, size):
"""Read 'size' bytes from remote."""
- return self.file.read(size)
+ cursize = min(size, _SAFE_BUF_SIZE)
+ data = self.file.read(cursize)
+ while cursize < size and len(data) == cursize:
+ delta = min(cursize, size - cursize)
+ data += self.file.read(delta)
+ cursize += delta
+ return data
def readline(self):
diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py
index a510f08db48a89..ce61883288aa35 100644
--- a/Lib/importlib/__init__.py
+++ b/Lib/importlib/__init__.py
@@ -79,7 +79,7 @@ def find_loader(name, path=None):
"""
warnings.warn('Deprecated since Python 3.4 and slated for removal in '
- 'Python 3.10; use importlib.util.find_spec() instead',
+ 'Python 3.12; use importlib.util.find_spec() instead',
DeprecationWarning, stacklevel=2)
try:
loader = sys.modules[name].__loader__
diff --git a/Lib/importlib/_adapters.py b/Lib/importlib/_adapters.py
index eedde49dd03ad9..e72edd10705c26 100644
--- a/Lib/importlib/_adapters.py
+++ b/Lib/importlib/_adapters.py
@@ -46,10 +46,11 @@ def is_dir(self):
def joinpath(self, other):
return DegenerateFiles.Path()
+ @property
def name(self):
return ''
- def open(self):
+ def open(self, mode='rb', *args, **kwargs):
raise ValueError()
def __init__(self, spec):
diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py
index 9a73c7b43a08cf..49bcaea78d7650 100644
--- a/Lib/importlib/_bootstrap_external.py
+++ b/Lib/importlib/_bootstrap_external.py
@@ -1212,10 +1212,15 @@ class _NamespacePath:
using path_finder. For top-level modules, the parent module's path
is sys.path."""
+ # When invalidate_caches() is called, this epoch is incremented
+ # https://bugs.python.org/issue45703
+ _epoch = 0
+
def __init__(self, name, path, path_finder):
self._name = name
self._path = path
self._last_parent_path = tuple(self._get_parent_path())
+ self._last_epoch = self._epoch
self._path_finder = path_finder
def _find_parent_path_names(self):
@@ -1235,7 +1240,7 @@ def _get_parent_path(self):
def _recalculate(self):
# If the parent's path has changed, recalculate _path
parent_path = tuple(self._get_parent_path()) # Make a copy
- if parent_path != self._last_parent_path:
+ if parent_path != self._last_parent_path or self._epoch != self._last_epoch:
spec = self._path_finder(self._name, parent_path)
# Note that no changes are made if a loader is returned, but we
# do remember the new parent path
@@ -1243,6 +1248,7 @@ def _recalculate(self):
if spec.submodule_search_locations:
self._path = spec.submodule_search_locations
self._last_parent_path = parent_path # Save the copy
+ self._last_epoch = self._epoch
return self._path
def __iter__(self):
@@ -1330,6 +1336,9 @@ def invalidate_caches():
del sys.path_importer_cache[name]
elif hasattr(finder, 'invalidate_caches'):
finder.invalidate_caches()
+ # Also invalidate the caches of _NamespacePaths
+ # https://bugs.python.org/issue45703
+ _NamespacePath._epoch += 1
@staticmethod
def _path_hooks(path):
diff --git a/Lib/importlib/_common.py b/Lib/importlib/_common.py
index 03383048ad7365..84144c038c3228 100644
--- a/Lib/importlib/_common.py
+++ b/Lib/importlib/_common.py
@@ -7,7 +7,7 @@
import importlib
from typing import Union, Any, Optional
-from .abc import ResourceReader
+from .abc import ResourceReader, Traversable
from ._adapters import wrap_spec
@@ -15,6 +15,7 @@
def files(package):
+ # type: (Package) -> Traversable
"""
Get a Traversable resource from a package
"""
@@ -30,7 +31,7 @@ def normalize_path(path):
str_path = str(path)
parent, file_name = os.path.split(str_path)
if parent:
- raise ValueError('{!r} must be only a file name'.format(path))
+ raise ValueError(f'{path!r} must be only a file name')
return file_name
@@ -64,7 +65,7 @@ def get_package(package):
"""
resolved = resolve(package)
if wrap_spec(resolved).submodule_search_locations is None:
- raise TypeError('{!r} is not a package'.format(package))
+ raise TypeError(f'{package!r} is not a package')
return resolved
@@ -79,7 +80,10 @@ def from_package(package):
@contextlib.contextmanager
-def _tempfile(reader, suffix=''):
+def _tempfile(reader, suffix='',
+ # gh-93353: Keep a reference to call os.remove() in late Python
+ # finalization.
+ *, _os_remove=os.remove):
# Not using tempfile.NamedTemporaryFile as it leads to deeper 'try'
# blocks due to the need to close the temporary file to work on Windows
# properly.
@@ -91,7 +95,7 @@ def _tempfile(reader, suffix=''):
yield pathlib.Path(raw_path)
finally:
try:
- os.remove(raw_path)
+ _os_remove(raw_path)
except FileNotFoundError:
pass
diff --git a/Lib/importlib/metadata/__init__.py b/Lib/importlib/metadata/__init__.py
index 142162196fa9b3..682067d3a3f04b 100644
--- a/Lib/importlib/metadata/__init__.py
+++ b/Lib/importlib/metadata/__init__.py
@@ -15,9 +15,11 @@
import collections
from . import _adapters, _meta
+from ._meta import PackageMetadata
from ._collections import FreezableDefaultDict, Pair
-from ._functools import method_cache
+from ._functools import method_cache, pass_none
from ._itertools import unique_everseen
+from ._meta import PackageMetadata, SimplePath
from contextlib import suppress
from importlib import import_module
@@ -29,6 +31,7 @@
__all__ = [
'Distribution',
'DistributionFinder',
+ 'PackageMetadata',
'PackageNotFoundError',
'distribution',
'distributions',
@@ -45,8 +48,7 @@ class PackageNotFoundError(ModuleNotFoundError):
"""The package was not found."""
def __str__(self):
- tmpl = "No package metadata was found for {self.name}"
- return tmpl.format(**locals())
+ return f"No package metadata was found for {self.name}"
@property
def name(self):
@@ -126,12 +128,21 @@ class EntryPoint(
See `the packaging docs on entry points
`_
for more information.
+
+ >>> ep = EntryPoint(
+ ... name=None, group=None, value='package.module:attr [extra1, extra2]')
+ >>> ep.module
+ 'package.module'
+ >>> ep.attr
+ 'attr'
+ >>> ep.extras
+ ['extra1', 'extra2']
"""
pattern = re.compile(
r'(?P[\w.]+)\s*'
- r'(:\s*(?P[\w.]+))?\s*'
- r'(?P\[.*\])?\s*$'
+ r'(:\s*(?P[\w.]+)\s*)?'
+ r'((?P\[.*\])\s*)?$'
)
"""
A regular expression describing the syntax for an entry point,
@@ -174,7 +185,7 @@ def attr(self):
@property
def extras(self):
match = self.pattern.match(self.value)
- return list(re.finditer(r'\w+', match.group('extras') or ''))
+ return re.findall(r'\w+', match.group('extras') or '')
def _for(self, dist):
self.dist = dist
@@ -198,11 +209,125 @@ def __reduce__(self):
)
def matches(self, **params):
+ """
+ EntryPoint matches the given parameters.
+
+ >>> ep = EntryPoint(group='foo', name='bar', value='bing:bong [extra1, extra2]')
+ >>> ep.matches(group='foo')
+ True
+ >>> ep.matches(name='bar', value='bing:bong [extra1, extra2]')
+ True
+ >>> ep.matches(group='foo', name='other')
+ False
+ >>> ep.matches()
+ True
+ >>> ep.matches(extras=['extra1', 'extra2'])
+ True
+ >>> ep.matches(module='bing')
+ True
+ >>> ep.matches(attr='bong')
+ True
+ """
attrs = (getattr(self, param) for param in params)
return all(map(operator.eq, params.values(), attrs))
-class EntryPoints(tuple):
+class DeprecatedList(list):
+ """
+ Allow an otherwise immutable object to implement mutability
+ for compatibility.
+
+ >>> recwarn = getfixture('recwarn')
+ >>> dl = DeprecatedList(range(3))
+ >>> dl[0] = 1
+ >>> dl.append(3)
+ >>> del dl[3]
+ >>> dl.reverse()
+ >>> dl.sort()
+ >>> dl.extend([4])
+ >>> dl.pop(-1)
+ 4
+ >>> dl.remove(1)
+ >>> dl += [5]
+ >>> dl + [6]
+ [1, 2, 5, 6]
+ >>> dl + (6,)
+ [1, 2, 5, 6]
+ >>> dl.insert(0, 0)
+ >>> dl
+ [0, 1, 2, 5]
+ >>> dl == [0, 1, 2, 5]
+ True
+ >>> dl == (0, 1, 2, 5)
+ True
+ >>> len(recwarn)
+ 1
+ """
+
+ __slots__ = ()
+
+ _warn = functools.partial(
+ warnings.warn,
+ "EntryPoints list interface is deprecated. Cast to list if needed.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+
+ def __setitem__(self, *args, **kwargs):
+ self._warn()
+ return super().__setitem__(*args, **kwargs)
+
+ def __delitem__(self, *args, **kwargs):
+ self._warn()
+ return super().__delitem__(*args, **kwargs)
+
+ def append(self, *args, **kwargs):
+ self._warn()
+ return super().append(*args, **kwargs)
+
+ def reverse(self, *args, **kwargs):
+ self._warn()
+ return super().reverse(*args, **kwargs)
+
+ def extend(self, *args, **kwargs):
+ self._warn()
+ return super().extend(*args, **kwargs)
+
+ def pop(self, *args, **kwargs):
+ self._warn()
+ return super().pop(*args, **kwargs)
+
+ def remove(self, *args, **kwargs):
+ self._warn()
+ return super().remove(*args, **kwargs)
+
+ def __iadd__(self, *args, **kwargs):
+ self._warn()
+ return super().__iadd__(*args, **kwargs)
+
+ def __add__(self, other):
+ if not isinstance(other, tuple):
+ self._warn()
+ other = tuple(other)
+ return self.__class__(tuple(self) + other)
+
+ def insert(self, *args, **kwargs):
+ self._warn()
+ return super().insert(*args, **kwargs)
+
+ def sort(self, *args, **kwargs):
+ self._warn()
+ return super().sort(*args, **kwargs)
+
+ def __eq__(self, other):
+ if not isinstance(other, tuple):
+ self._warn()
+ other = tuple(other)
+
+ return tuple(self).__eq__(other)
+
+
+class EntryPoints(DeprecatedList):
"""
An immutable collection of selectable EntryPoint objects.
"""
@@ -213,6 +338,14 @@ def __getitem__(self, name): # -> EntryPoint:
"""
Get the EntryPoint in self matching name.
"""
+ if isinstance(name, int):
+ warnings.warn(
+ "Accessing entry points by index is deprecated. "
+ "Cast to tuple if needed.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return super().__getitem__(name)
try:
return next(iter(self.select(name=name)))
except StopIteration:
@@ -259,14 +392,6 @@ def _parse_groups(text):
)
-def flake8_bypass(func):
- # defer inspect import as performance optimization.
- import inspect
-
- is_flake8 = any('flake8' in str(frame.filename) for frame in inspect.stack()[:5])
- return func if not is_flake8 else lambda: None
-
-
class Deprecated:
"""
Compatibility add-in for mapping to indicate that
@@ -302,7 +427,7 @@ def __getitem__(self, name):
return super().__getitem__(name)
def get(self, name, default=None):
- flake8_bypass(self._warn)()
+ self._warn()
return super().get(name, default)
def __iter__(self):
@@ -383,7 +508,7 @@ def __init__(self, spec):
self.mode, _, self.value = spec.partition('=')
def __repr__(self):
- return ''.format(self.mode, self.value)
+ return f''
class Distribution:
@@ -491,6 +616,11 @@ def name(self):
"""Return the 'Name' metadata for the distribution package."""
return self.metadata['Name']
+ @property
+ def _normalized_name(self):
+ """Return a normalized version of the name."""
+ return Prepared.normalize(self.name)
+
@property
def version(self):
"""Return the 'Version' metadata for the distribution package."""
@@ -548,7 +678,7 @@ def _read_dist_info_reqs(self):
def _read_egg_info_reqs(self):
source = self.read_text('requires.txt')
- return source and self._deps_from_requires_text(source)
+ return None if source is None else self._deps_from_requires_text(source)
@classmethod
def _deps_from_requires_text(cls, source):
@@ -567,18 +697,27 @@ def _convert_egg_info_reqs_to_simple_reqs(sections):
"""
def make_condition(name):
- return name and 'extra == "{name}"'.format(name=name)
+ return name and f'extra == "{name}"'
- def parse_condition(section):
+ def quoted_marker(section):
section = section or ''
extra, sep, markers = section.partition(':')
if extra and markers:
- markers = '({markers})'.format(markers=markers)
+ markers = f'({markers})'
conditions = list(filter(None, [markers, make_condition(extra)]))
return '; ' + ' and '.join(conditions) if conditions else ''
+ def url_req_space(req):
+ """
+ PEP 508 requires a space between the url_spec and the quoted_marker.
+ Ref python/importlib_metadata#357.
+ """
+ # '@' is uniquely indicative of a url_req.
+ return ' ' * ('@' in req)
+
for section in sections:
- yield section.value + parse_condition(section.name)
+ space = url_req_space(section.value)
+ yield section.value + space + quoted_marker(section.name)
class DistributionFinder(MetaPathFinder):
@@ -610,10 +749,11 @@ def __init__(self, **kwargs):
@property
def path(self):
"""
- The path that a distribution finder should search.
+ The sequence of directory path that a distribution finder
+ should search.
- Typically refers to Python package paths and defaults
- to ``sys.path``.
+ Typically refers to Python installed package paths such as
+ "site-packages" directories and defaults to ``sys.path``.
"""
return vars(self).get('path', sys.path)
@@ -640,14 +780,13 @@ def __new__(cls, root):
def __init__(self, root):
self.root = root
- self.base = os.path.basename(self.root).lower()
def joinpath(self, child):
return pathlib.Path(self.root, child)
def children(self):
with suppress(Exception):
- return os.listdir(self.root or '')
+ return os.listdir(self.root or '.')
with suppress(Exception):
return self.zip_children()
return []
@@ -770,11 +909,10 @@ def invalidate_caches(cls):
class PathDistribution(Distribution):
- def __init__(self, path):
- """Construct a distribution from a path to the metadata directory.
+ def __init__(self, path: SimplePath):
+ """Construct a distribution.
- :param path: A pathlib.Path or similar object supporting
- .joinpath(), __div__, .parent, and .read_text().
+ :param path: SimplePath indicating the metadata directory.
"""
self._path = path
@@ -793,6 +931,34 @@ def read_text(self, filename):
def locate_file(self, path):
return self._path.parent / path
+ @property
+ def _normalized_name(self):
+ """
+ Performance optimization: where possible, resolve the
+ normalized name from the file system path.
+ """
+ stem = os.path.basename(str(self._path))
+ return (
+ pass_none(Prepared.normalize)(self._name_from_stem(stem))
+ or super()._normalized_name
+ )
+
+ @staticmethod
+ def _name_from_stem(stem):
+ """
+ >>> PathDistribution._name_from_stem('foo-3.0.egg-info')
+ 'foo'
+ >>> PathDistribution._name_from_stem('CherryPy-3.0.dist-info')
+ 'CherryPy'
+ >>> PathDistribution._name_from_stem('face.egg-info')
+ 'face'
+ """
+ filename, ext = os.path.splitext(stem)
+ if ext not in ('.dist-info', '.egg-info'):
+ return
+ name, sep, rest = filename.partition('-')
+ return name
+
def distribution(distribution_name):
"""Get the ``Distribution`` instance for the named package.
@@ -847,7 +1013,8 @@ def entry_points(**params) -> Union[EntryPoints, SelectableGroups]:
:return: EntryPoints or SelectableGroups for all installed packages.
"""
- unique = functools.partial(unique_everseen, key=operator.attrgetter('name'))
+ norm_name = operator.attrgetter('_normalized_name')
+ unique = functools.partial(unique_everseen, key=norm_name)
eps = itertools.chain.from_iterable(
dist.entry_points for dist in unique(distributions())
)
@@ -868,7 +1035,7 @@ def requires(distribution_name):
Return a list of requirements for the named package.
:return: An iterator of requirements, suitable for
- packaging.requirement.Requirement.
+ packaging.requirement.Requirement.
"""
return distribution(distribution_name).requires
diff --git a/Lib/importlib/metadata/_adapters.py b/Lib/importlib/metadata/_adapters.py
index ab086180fc35fc..aa460d3eda50fb 100644
--- a/Lib/importlib/metadata/_adapters.py
+++ b/Lib/importlib/metadata/_adapters.py
@@ -19,6 +19,7 @@ class Message(email.message.Message):
'Requires-Dist',
'Requires-External',
'Supported-Platform',
+ 'Dynamic',
],
)
)
diff --git a/Lib/importlib/metadata/_functools.py b/Lib/importlib/metadata/_functools.py
index 73f50d00bc04c6..71f66bd03cb713 100644
--- a/Lib/importlib/metadata/_functools.py
+++ b/Lib/importlib/metadata/_functools.py
@@ -83,3 +83,22 @@ def wrapper(self, *args, **kwargs):
wrapper.cache_clear = lambda: None
return wrapper
+
+
+# From jaraco.functools 3.3
+def pass_none(func):
+ """
+ Wrap func so it's not called if its first param is None
+
+ >>> print_text = pass_none(print)
+ >>> print_text('text')
+ text
+ >>> print_text(None)
+ """
+
+ @functools.wraps(func)
+ def wrapper(param, *args, **kwargs):
+ if param is not None:
+ return func(param, *args, **kwargs)
+
+ return wrapper
diff --git a/Lib/importlib/metadata/_meta.py b/Lib/importlib/metadata/_meta.py
index 04d9a0235368ed..1a6edbf957d5a0 100644
--- a/Lib/importlib/metadata/_meta.py
+++ b/Lib/importlib/metadata/_meta.py
@@ -27,3 +27,21 @@ def json(self) -> Dict[str, Union[str, List[str]]]:
"""
A JSON-compatible form of the metadata.
"""
+
+
+class SimplePath(Protocol):
+ """
+ A minimal subset of pathlib.Path required by PathDistribution.
+ """
+
+ def joinpath(self) -> 'SimplePath':
+ ... # pragma: no cover
+
+ def __div__(self) -> 'SimplePath':
+ ... # pragma: no cover
+
+ def parent(self) -> 'SimplePath':
+ ... # pragma: no cover
+
+ def read_text(self) -> str:
+ ... # pragma: no cover
diff --git a/Lib/importlib/readers.py b/Lib/importlib/readers.py
index 535c8283e9081d..41089c071d8683 100644
--- a/Lib/importlib/readers.py
+++ b/Lib/importlib/readers.py
@@ -94,15 +94,15 @@ def joinpath(self, child):
__truediv__ = joinpath
def open(self, *args, **kwargs):
- raise FileNotFoundError('{} is not a file'.format(self))
+ raise FileNotFoundError(f'{self} is not a file')
+ @property
def name(self):
return self._paths[0].name
def __repr__(self):
- return 'MultiplexedPath({})'.format(
- ', '.join("'{}'".format(path) for path in self._paths)
- )
+ paths = ', '.join(f"'{path}'" for path in self._paths)
+ return f'MultiplexedPath({paths})'
class NamespaceReader(abc.TraversableResources):
diff --git a/Lib/importlib/resources.py b/Lib/importlib/resources.py
index db0e0c0eeff80b..8a98663ff8e6d5 100644
--- a/Lib/importlib/resources.py
+++ b/Lib/importlib/resources.py
@@ -68,9 +68,7 @@ def open_binary(package: Package, resource: Resource) -> BinaryIO:
if data is not None:
return BytesIO(data)
- raise FileNotFoundError(
- '{!r} resource not found in {!r}'.format(resource, spec.name)
- )
+ raise FileNotFoundError(f'{resource!r} resource not found in {spec.name!r}')
def open_text(
diff --git a/Lib/inspect.py b/Lib/inspect.py
index 9f8cc01310f10e..2999a6019e0f60 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -294,7 +294,7 @@ def _has_code_flag(f, flag):
while ismethod(f):
f = f.__func__
f = functools._unwrap_partial(f)
- if not isfunction(f):
+ if not (isfunction(f) or _signature_is_functionlike(f)):
return False
return bool(f.__code__.co_flags & flag)
@@ -395,7 +395,7 @@ def iscode(object):
co_kwonlyargcount number of keyword only arguments (not including ** arg)
co_lnotab encoded mapping of line numbers to bytecode indices
co_name name with which this code object was defined
- co_names tuple of names of local variables
+ co_names tuple of names other than arguments and function locals
co_nlocals number of local variables
co_stacksize virtual machine stack space required
co_varnames tuple of names of arguments and local variables"""
@@ -781,6 +781,8 @@ def getfile(object):
module = sys.modules.get(object.__module__)
if getattr(module, '__file__', None):
return module.__file__
+ if object.__module__ == '__main__':
+ raise OSError('source code not available')
raise TypeError('{!r} is a built-in class'.format(object))
if ismethod(object):
object = object.__func__
@@ -857,7 +859,7 @@ def getmodule(object, _filename=None):
# Try the cache again with the absolute file name
try:
file = getabsfile(object, _filename)
- except TypeError:
+ except (TypeError, FileNotFoundError):
return None
if file in modulesbyfile:
return sys.modules.get(modulesbyfile[file])
@@ -1050,7 +1052,6 @@ def __init__(self):
self.started = False
self.passline = False
self.indecorator = False
- self.decoratorhasargs = False
self.last = 1
self.body_col0 = None
@@ -1065,13 +1066,6 @@ def tokeneater(self, type, token, srowcol, erowcol, line):
self.islambda = True
self.started = True
self.passline = True # skip to the end of the line
- elif token == "(":
- if self.indecorator:
- self.decoratorhasargs = True
- elif token == ")":
- if self.indecorator:
- self.indecorator = False
- self.decoratorhasargs = False
elif type == tokenize.NEWLINE:
self.passline = False # stop skipping when a NEWLINE is seen
self.last = srowcol[0]
@@ -1079,7 +1073,7 @@ def tokeneater(self, type, token, srowcol, erowcol, line):
raise EndOfBlock
# hitting a NEWLINE when in a decorator without args
# ends the decorator
- if self.indecorator and not self.decoratorhasargs:
+ if self.indecorator:
self.indecorator = False
elif self.passline:
pass
@@ -1354,7 +1348,12 @@ def getargvalues(frame):
def formatannotation(annotation, base_module=None):
if getattr(annotation, '__module__', None) == 'typing':
- return repr(annotation).replace('typing.', '')
+ def repl(match):
+ text = match.group()
+ return text.removeprefix('typing.')
+ return re.sub(r'[\w\.]+', repl, repr(annotation))
+ if isinstance(annotation, types.GenericAlias):
+ return str(annotation)
if isinstance(annotation, type):
if annotation.__module__ in ('builtins', base_module):
return annotation.__qualname__
@@ -2062,7 +2061,7 @@ def _signature_strip_non_python_syntax(signature):
self_parameter = None
last_positional_only = None
- lines = [l.encode('ascii') for l in signature.split('\n')]
+ lines = [l.encode('ascii') for l in signature.split('\n') if l]
generator = iter(lines).__next__
token_stream = tokenize.tokenize(generator)
@@ -2142,7 +2141,6 @@ def _signature_fromstr(cls, obj, s, skip_bound_arg=True):
parameters = []
empty = Parameter.empty
- invalid = object()
module = None
module_dict = {}
@@ -2166,11 +2164,11 @@ def wrap_value(s):
try:
value = eval(s, sys_module_dict)
except NameError:
- raise RuntimeError()
+ raise ValueError
if isinstance(value, (str, int, float, bytes, bool, type(None))):
return ast.Constant(value)
- raise RuntimeError()
+ raise ValueError
class RewriteSymbolics(ast.NodeTransformer):
def visit_Attribute(self, node):
@@ -2180,7 +2178,7 @@ def visit_Attribute(self, node):
a.append(n.attr)
n = n.value
if not isinstance(n, ast.Name):
- raise RuntimeError()
+ raise ValueError
a.append(n.id)
value = ".".join(reversed(a))
return wrap_value(value)
@@ -2190,19 +2188,29 @@ def visit_Name(self, node):
raise ValueError()
return wrap_value(node.id)
+ def visit_BinOp(self, node):
+ # Support constant folding of a couple simple binary operations
+ # commonly used to define default values in text signatures
+ left = self.visit(node.left)
+ right = self.visit(node.right)
+ if not isinstance(left, ast.Constant) or not isinstance(right, ast.Constant):
+ raise ValueError
+ if isinstance(node.op, ast.Add):
+ return ast.Constant(left.value + right.value)
+ elif isinstance(node.op, ast.Sub):
+ return ast.Constant(left.value - right.value)
+ elif isinstance(node.op, ast.BitOr):
+ return ast.Constant(left.value | right.value)
+ raise ValueError
+
def p(name_node, default_node, default=empty):
name = parse_name(name_node)
- if name is invalid:
- return None
if default_node and default_node is not _empty:
try:
default_node = RewriteSymbolics().visit(default_node)
- o = ast.literal_eval(default_node)
+ default = ast.literal_eval(default_node)
except ValueError:
- o = invalid
- if o is invalid:
- return None
- default = o if o is not invalid else default
+ raise ValueError("{!r} builtin has invalid signature".format(obj)) from None
parameters.append(Parameter(name, kind, default=default, annotation=empty))
# non-keyword-only parameters
@@ -2399,7 +2407,10 @@ def _signature_from_callable(obj, *,
# Was this function wrapped by a decorator?
if follow_wrapper_chains:
- obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__")))
+ # Unwrap until we find an explicit signature or a MethodType (which will be
+ # handled explicitly below).
+ obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__")
+ or isinstance(f, types.MethodType)))
if isinstance(obj, types.MethodType):
# If the unwrapped object is a *method*, we might want to
# skip its first parameter (self).
@@ -2471,15 +2482,23 @@ def _signature_from_callable(obj, *,
if call is not None:
sig = _get_signature_of(call)
else:
- # Now we check if the 'obj' class has a '__new__' method
+ factory_method = None
new = _signature_get_user_defined_method(obj, '__new__')
- if new is not None:
- sig = _get_signature_of(new)
- else:
- # Finally, we should have at least __init__ implemented
- init = _signature_get_user_defined_method(obj, '__init__')
- if init is not None:
- sig = _get_signature_of(init)
+ init = _signature_get_user_defined_method(obj, '__init__')
+ # Now we check if the 'obj' class has an own '__new__' method
+ if '__new__' in obj.__dict__:
+ factory_method = new
+ # or an own '__init__' method
+ elif '__init__' in obj.__dict__:
+ factory_method = init
+ # If not, we take inherited '__new__' or '__init__', if present
+ elif new is not None:
+ factory_method = new
+ elif init is not None:
+ factory_method = init
+
+ if factory_method is not None:
+ sig = _get_signature_of(factory_method)
if sig is None:
# At this point we know, that `obj` is a class, with no user-
@@ -2499,9 +2518,9 @@ def _signature_from_callable(obj, *,
pass
else:
if text_sig:
- # If 'obj' class has a __text_signature__ attribute:
+ # If 'base' class has a __text_signature__ attribute:
# return a signature based on it
- return _signature_fromstr(sigcls, obj, text_sig)
+ return _signature_fromstr(sigcls, base, text_sig)
# No '__text_signature__' was found for the 'obj' class.
# Last option is to check if its '__init__' is
@@ -2561,6 +2580,9 @@ class _ParameterKind(enum.IntEnum):
KEYWORD_ONLY = 3
VAR_KEYWORD = 4
+ def __str__(self):
+ return self._name_
+
@property
def description(self):
return _PARAM_NAME_MAPPING[self]
diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py
index af7aedfa6e51a1..ac284ca125bead 100644
--- a/Lib/ipaddress.py
+++ b/Lib/ipaddress.py
@@ -16,6 +16,7 @@
IPV4LENGTH = 32
IPV6LENGTH = 128
+
class AddressValueError(ValueError):
"""A Value Error related to the address."""
@@ -50,8 +51,7 @@ def ip_address(address):
except (AddressValueError, NetmaskValueError):
pass
- raise ValueError('%r does not appear to be an IPv4 or IPv6 address' %
- address)
+ raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
def ip_network(address, strict=True):
@@ -80,8 +80,7 @@ def ip_network(address, strict=True):
except (AddressValueError, NetmaskValueError):
pass
- raise ValueError('%r does not appear to be an IPv4 or IPv6 network' %
- address)
+ raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 network')
def ip_interface(address):
@@ -115,8 +114,7 @@ def ip_interface(address):
except (AddressValueError, NetmaskValueError):
pass
- raise ValueError('%r does not appear to be an IPv4 or IPv6 interface' %
- address)
+ raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 interface')
def v4_int_to_packed(address):
@@ -159,7 +157,7 @@ def _split_optional_netmask(address):
"""Helper to split the netmask and raise AddressValueError if needed"""
addr = str(address).split('/')
if len(addr) > 2:
- raise AddressValueError("Only one '/' permitted in %r" % address)
+ raise AddressValueError(f"Only one '/' permitted in {address!r}")
return addr
@@ -312,7 +310,7 @@ def collapse_addresses(addresses):
[IPv4Network('192.0.2.0/24')]
Args:
- addresses: An iterator of IPv4Network or IPv6Network objects.
+ addresses: An iterable of IPv4Network or IPv6Network objects.
Returns:
An iterator of the collapsed IPv(4|6)Network objects.
@@ -1303,7 +1301,7 @@ def __init__(self, address):
# which converts into a formatted IP string.
addr_str = str(address)
if '/' in addr_str:
- raise AddressValueError("Unexpected '/' in %r" % address)
+ raise AddressValueError(f"Unexpected '/' in {address!r}")
self._ip = self._ip_int_from_string(addr_str)
@property
@@ -1325,18 +1323,41 @@ def is_reserved(self):
@property
@functools.lru_cache()
def is_private(self):
- """Test if this address is allocated for private networks.
+ """``True`` if the address is defined as not globally reachable by
+ iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_
+ (for IPv6) with the following exceptions:
- Returns:
- A boolean, True if the address is reserved per
- iana-ipv4-special-registry.
+ * ``is_private`` is ``False`` for ``100.64.0.0/10``
+ * For IPv4-mapped IPv6-addresses the ``is_private`` value is determined by the
+ semantics of the underlying IPv4 addresses and the following condition holds
+ (see :attr:`IPv6Address.ipv4_mapped`)::
+
+ address.is_private == address.ipv4_mapped.is_private
+ ``is_private`` has value opposite to :attr:`is_global`, except for the ``100.64.0.0/10``
+ IPv4 range where they are both ``False``.
"""
- return any(self in net for net in self._constants._private_networks)
+ return (
+ any(self in net for net in self._constants._private_networks)
+ and all(self not in net for net in self._constants._private_networks_exceptions)
+ )
@property
@functools.lru_cache()
def is_global(self):
+ """``True`` if the address is defined as globally reachable by
+ iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_
+ (for IPv6) with the following exception:
+
+ For IPv4-mapped IPv6-addresses the ``is_private`` value is determined by the
+ semantics of the underlying IPv4 addresses and the following condition holds
+ (see :attr:`IPv6Address.ipv4_mapped`)::
+
+ address.is_global == address.ipv4_mapped.is_global
+
+ ``is_global`` has value opposite to :attr:`is_private`, except for the ``100.64.0.0/10``
+ IPv4 range where they are both ``False``.
+ """
return self not in self._constants._public_network and not self.is_private
@property
@@ -1540,13 +1561,15 @@ class _IPv4Constants:
_public_network = IPv4Network('100.64.0.0/10')
+ # Not globally reachable address blocks listed on
+ # https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
_private_networks = [
IPv4Network('0.0.0.0/8'),
IPv4Network('10.0.0.0/8'),
IPv4Network('127.0.0.0/8'),
IPv4Network('169.254.0.0/16'),
IPv4Network('172.16.0.0/12'),
- IPv4Network('192.0.0.0/29'),
+ IPv4Network('192.0.0.0/24'),
IPv4Network('192.0.0.170/31'),
IPv4Network('192.0.2.0/24'),
IPv4Network('192.168.0.0/16'),
@@ -1557,6 +1580,11 @@ class _IPv4Constants:
IPv4Network('255.255.255.255/32'),
]
+ _private_networks_exceptions = [
+ IPv4Network('192.0.0.9/32'),
+ IPv4Network('192.0.0.10/32'),
+ ]
+
_reserved_network = IPv4Network('240.0.0.0/4')
_unspecified_address = IPv4Address('0.0.0.0')
@@ -1812,9 +1840,6 @@ def _string_from_ip_int(cls, ip_int=None):
def _explode_shorthand_ip_string(self):
"""Expand a shortened IPv6 address.
- Args:
- ip_str: A string, the IPv6 address.
-
Returns:
A string, the expanded IPv6 address.
@@ -1912,7 +1937,7 @@ def __init__(self, address):
# which converts into a formatted IP string.
addr_str = str(address)
if '/' in addr_str:
- raise AddressValueError("Unexpected '/' in %r" % address)
+ raise AddressValueError(f"Unexpected '/' in {address!r}")
addr_str, self._scope_id = self._split_scope_id(addr_str)
self._ip = self._ip_int_from_string(addr_str)
@@ -1958,6 +1983,9 @@ def is_multicast(self):
See RFC 2373 2.7 for details.
"""
+ ipv4_mapped = self.ipv4_mapped
+ if ipv4_mapped is not None:
+ return ipv4_mapped.is_multicast
return self in self._constants._multicast_network
@property
@@ -1969,6 +1997,9 @@ def is_reserved(self):
reserved IPv6 Network ranges.
"""
+ ipv4_mapped = self.ipv4_mapped
+ if ipv4_mapped is not None:
+ return ipv4_mapped.is_reserved
return any(self in x for x in self._constants._reserved_networks)
@property
@@ -1979,6 +2010,9 @@ def is_link_local(self):
A boolean, True if the address is reserved per RFC 4291.
"""
+ ipv4_mapped = self.ipv4_mapped
+ if ipv4_mapped is not None:
+ return ipv4_mapped.is_link_local
return self in self._constants._linklocal_network
@property
@@ -1998,24 +2032,46 @@ def is_site_local(self):
@property
@functools.lru_cache()
def is_private(self):
- """Test if this address is allocated for private networks.
+ """``True`` if the address is defined as not globally reachable by
+ iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_
+ (for IPv6) with the following exceptions:
- Returns:
- A boolean, True if the address is reserved per
- iana-ipv6-special-registry.
+ * ``is_private`` is ``False`` for ``100.64.0.0/10``
+ * For IPv4-mapped IPv6-addresses the ``is_private`` value is determined by the
+ semantics of the underlying IPv4 addresses and the following condition holds
+ (see :attr:`IPv6Address.ipv4_mapped`)::
+ address.is_private == address.ipv4_mapped.is_private
+
+ ``is_private`` has value opposite to :attr:`is_global`, except for the ``100.64.0.0/10``
+ IPv4 range where they are both ``False``.
"""
- return any(self in net for net in self._constants._private_networks)
+ ipv4_mapped = self.ipv4_mapped
+ if ipv4_mapped is not None:
+ return ipv4_mapped.is_private
+ return (
+ any(self in net for net in self._constants._private_networks)
+ and all(self not in net for net in self._constants._private_networks_exceptions)
+ )
@property
def is_global(self):
- """Test if this address is allocated for public networks.
+ """``True`` if the address is defined as globally reachable by
+ iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_
+ (for IPv6) with the following exception:
- Returns:
- A boolean, true if the address is not reserved per
- iana-ipv6-special-registry.
+ For IPv4-mapped IPv6-addresses the ``is_private`` value is determined by the
+ semantics of the underlying IPv4 addresses and the following condition holds
+ (see :attr:`IPv6Address.ipv4_mapped`)::
+ address.is_global == address.ipv4_mapped.is_global
+
+ ``is_global`` has value opposite to :attr:`is_private`, except for the ``100.64.0.0/10``
+ IPv4 range where they are both ``False``.
"""
+ ipv4_mapped = self.ipv4_mapped
+ if ipv4_mapped is not None:
+ return ipv4_mapped.is_global
return not self.is_private
@property
@@ -2027,6 +2083,9 @@ def is_unspecified(self):
RFC 2373 2.5.2.
"""
+ ipv4_mapped = self.ipv4_mapped
+ if ipv4_mapped is not None:
+ return ipv4_mapped.is_unspecified
return self._ip == 0
@property
@@ -2038,6 +2097,9 @@ def is_loopback(self):
RFC 2373 2.5.3.
"""
+ ipv4_mapped = self.ipv4_mapped
+ if ipv4_mapped is not None:
+ return ipv4_mapped.is_loopback
return self._ip == 1
@property
@@ -2154,7 +2216,7 @@ def is_unspecified(self):
@property
def is_loopback(self):
- return self._ip == 1 and self.network.is_loopback
+ return super().is_loopback and self.network.is_loopback
class IPv6Network(_BaseV6, _BaseNetwork):
@@ -2255,19 +2317,33 @@ class _IPv6Constants:
_multicast_network = IPv6Network('ff00::/8')
+ # Not globally reachable address blocks listed on
+ # https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml
_private_networks = [
IPv6Network('::1/128'),
IPv6Network('::/128'),
IPv6Network('::ffff:0:0/96'),
+ IPv6Network('64:ff9b:1::/48'),
IPv6Network('100::/64'),
IPv6Network('2001::/23'),
- IPv6Network('2001:2::/48'),
IPv6Network('2001:db8::/32'),
- IPv6Network('2001:10::/28'),
+ # IANA says N/A, let's consider it not globally reachable to be safe
+ IPv6Network('2002::/16'),
+ # RFC 9637: https://www.rfc-editor.org/rfc/rfc9637.html#section-6-2.2
+ IPv6Network('3fff::/20'),
IPv6Network('fc00::/7'),
IPv6Network('fe80::/10'),
]
+ _private_networks_exceptions = [
+ IPv6Network('2001:1::1/128'),
+ IPv6Network('2001:1::2/128'),
+ IPv6Network('2001:3::/32'),
+ IPv6Network('2001:4:112::/48'),
+ IPv6Network('2001:20::/28'),
+ IPv6Network('2001:30::/28'),
+ ]
+
_reserved_networks = [
IPv6Network('::/8'), IPv6Network('100::/8'),
IPv6Network('200::/7'), IPv6Network('400::/6'),
diff --git a/Lib/json/__init__.py b/Lib/json/__init__.py
index 2c52bdeba67546..ed2c74771ea87d 100644
--- a/Lib/json/__init__.py
+++ b/Lib/json/__init__.py
@@ -1,4 +1,4 @@
-r"""JSON (JavaScript Object Notation) is a subset of
+r"""JSON (JavaScript Object Notation) is a subset of
JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data
interchange format.
@@ -133,7 +133,7 @@ def dump(obj, fp, *, skipkeys=False, ensure_ascii=True, check_circular=True,
If ``check_circular`` is false, then the circular reference check
for container types will be skipped and a circular reference will
- result in an ``OverflowError`` (or worse).
+ result in an ``RecursionError`` (or worse).
If ``allow_nan`` is false, then it will be a ``ValueError`` to
serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``)
@@ -195,7 +195,7 @@ def dumps(obj, *, skipkeys=False, ensure_ascii=True, check_circular=True,
If ``check_circular`` is false, then the circular reference check
for container types will be skipped and a circular reference will
- result in an ``OverflowError`` (or worse).
+ result in an ``RecursionError`` (or worse).
If ``allow_nan`` is false, then it will be a ``ValueError`` to
serialize out of range ``float`` values (``nan``, ``inf``, ``-inf``) in
diff --git a/Lib/json/decoder.py b/Lib/json/decoder.py
index d7d824454e1ba3..c5d9ae2d0d5d04 100644
--- a/Lib/json/decoder.py
+++ b/Lib/json/decoder.py
@@ -252,7 +252,7 @@ def JSONArray(s_and_end, scan_once, _w=WHITESPACE.match, _ws=WHITESPACE_STR):
class JSONDecoder(object):
- """Simple JSON decoder
+ """Simple JSON decoder
Performs the following translations in decoding by default:
diff --git a/Lib/json/encoder.py b/Lib/json/encoder.py
index c8c78b9c237652..687c50683a09c0 100644
--- a/Lib/json/encoder.py
+++ b/Lib/json/encoder.py
@@ -71,7 +71,7 @@ def replace(match):
c_encode_basestring_ascii or py_encode_basestring_ascii)
class JSONEncoder(object):
- """Extensible JSON encoder for Python data structures.
+ """Extensible JSON encoder for Python data structures.
Supports the following objects and types by default:
@@ -116,7 +116,7 @@ def __init__(self, *, skipkeys=False, ensure_ascii=True,
If check_circular is true, then lists, dicts, and custom encoded
objects will be checked for circular references during encoding to
- prevent an infinite recursion (which would cause an OverflowError).
+ prevent an infinite recursion (which would cause an RecursionError).
Otherwise, no such check takes place.
If allow_nan is true, then NaN, Infinity, and -Infinity will be
diff --git a/Lib/json/tool.py b/Lib/json/tool.py
index 5dee0a744b2a99..0490b8c0be11df 100644
--- a/Lib/json/tool.py
+++ b/Lib/json/tool.py
@@ -13,6 +13,7 @@
import argparse
import json
import sys
+from pathlib import Path
def main():
@@ -25,9 +26,9 @@ def main():
help='a JSON file to be validated or pretty-printed',
default=sys.stdin)
parser.add_argument('outfile', nargs='?',
- type=argparse.FileType('w', encoding="utf-8"),
+ type=Path,
help='write the output of infile to outfile',
- default=sys.stdout)
+ default=None)
parser.add_argument('--sort-keys', action='store_true', default=False,
help='sort the output of dictionaries alphabetically by key')
parser.add_argument('--no-ensure-ascii', dest='ensure_ascii', action='store_false',
@@ -58,15 +59,21 @@ def main():
dump_args['indent'] = None
dump_args['separators'] = ',', ':'
- with options.infile as infile, options.outfile as outfile:
+ with options.infile as infile:
try:
if options.json_lines:
objs = (json.loads(line) for line in infile)
else:
- objs = (json.load(infile), )
- for obj in objs:
- json.dump(obj, outfile, **dump_args)
- outfile.write('\n')
+ objs = (json.load(infile),)
+
+ if options.outfile is None:
+ out = sys.stdout
+ else:
+ out = options.outfile.open('w', encoding='utf-8')
+ with out as outfile:
+ for obj in objs:
+ json.dump(obj, outfile, **dump_args)
+ outfile.write('\n')
except ValueError as e:
raise SystemExit(e)
diff --git a/Lib/lib2to3/fixes/fix_metaclass.py b/Lib/lib2to3/fixes/fix_metaclass.py
index d1cd10d327587c..fe547b2228072a 100644
--- a/Lib/lib2to3/fixes/fix_metaclass.py
+++ b/Lib/lib2to3/fixes/fix_metaclass.py
@@ -51,7 +51,7 @@ def fixup_parse_tree(cls_node):
# already in the preferred format, do nothing
return
- # !%@#! oneliners have no suite node, we have to fake one up
+ # !%@#! one-liners have no suite node, we have to fake one up
for i, node in enumerate(cls_node.children):
if node.type == token.COLON:
break
diff --git a/Lib/lib2to3/fixes/fix_paren.py b/Lib/lib2to3/fixes/fix_paren.py
index b205aa7e1e93fb..df3da5f5232c9c 100644
--- a/Lib/lib2to3/fixes/fix_paren.py
+++ b/Lib/lib2to3/fixes/fix_paren.py
@@ -1,4 +1,4 @@
-"""Fixer that addes parentheses where they are required
+"""Fixer that adds parentheses where they are required
This converts ``[x for x in 1, 2]`` to ``[x for x in (1, 2)]``."""
diff --git a/Lib/lib2to3/pgen2/tokenize.py b/Lib/lib2to3/pgen2/tokenize.py
index 0e2685d40433d7..099dfa7798afd4 100644
--- a/Lib/lib2to3/pgen2/tokenize.py
+++ b/Lib/lib2to3/pgen2/tokenize.py
@@ -512,13 +512,14 @@ def generate_tokens(readline):
stashed = tok
continue
- if token == 'def':
+ if token in ('def', 'for'):
if (stashed
and stashed[0] == NAME
and stashed[1] == 'async'):
- async_def = True
- async_def_indent = indents[-1]
+ if token == 'def':
+ async_def = True
+ async_def_indent = indents[-1]
yield (ASYNC, stashed[1],
stashed[2], stashed[3],
diff --git a/Lib/lib2to3/pytree.py b/Lib/lib2to3/pytree.py
index 2a6ef2ef5240a2..729023df0284e1 100644
--- a/Lib/lib2to3/pytree.py
+++ b/Lib/lib2to3/pytree.py
@@ -720,8 +720,8 @@ def generate_matches(self, nodes):
r[self.name] = nodes[:count]
yield count, r
except RuntimeError:
- # We fall back to the iterative pattern matching scheme if the recursive
- # scheme hits the recursion limit.
+ # Fall back to the iterative pattern matching scheme if the
+ # recursive scheme hits the recursion limit (RecursionError).
for count, r in self._iterative_matches(nodes):
if self.name:
r[self.name] = nodes[:count]
diff --git a/Lib/lib2to3/tests/data/infinite_recursion.py b/Lib/lib2to3/tests/data/infinite_recursion.py
index 71715ef29dc488..acc62ed4f1a44c 100644
--- a/Lib/lib2to3/tests/data/infinite_recursion.py
+++ b/Lib/lib2to3/tests/data/infinite_recursion.py
@@ -1,5 +1,5 @@
-# This file is used to verify that 2to3 falls back to a slower, iterative pattern matching
-# scheme in the event that the faster recursive system fails due to infinite recursion.
+# Verify that 2to3 falls back from the recursive pattern matching scheme to a
+# slower, iterative scheme in the event of a RecursionError.
from ctypes import *
STRING = c_char_p
diff --git a/Lib/lib2to3/tests/data/py2_test_grammar.py b/Lib/lib2to3/tests/data/py2_test_grammar.py
index b5a4137d1615ef..f9e4ea1374f907 100644
--- a/Lib/lib2to3/tests/data/py2_test_grammar.py
+++ b/Lib/lib2to3/tests/data/py2_test_grammar.py
@@ -8,7 +8,7 @@
# regression test, the filterwarnings() call has been added to
# regrtest.py.
-from test.test_support import run_unittest, check_syntax_error
+from test.test_support import check_syntax_error
import unittest
import sys
# testing import *
@@ -735,7 +735,7 @@ def testSelectors(self):
s = a[-5:]
s = a[:-1]
s = a[-4:-3]
- # A rough test of SF bug 1333982. http://python.org/sf/1333982
+ # A rough test of SF bug 1333982. https://python.org/sf/1333982
# The testing here is fairly incomplete.
# Test cases should include: commas with 1 and 2 colons
d = {}
@@ -967,8 +967,5 @@ def _checkeval(msg, ret):
self.assertEqual((6 < 4 if 0 else 2), 2)
-def test_main():
- run_unittest(TokenTests, GrammarTests)
-
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/lib2to3/tests/data/py3_test_grammar.py b/Lib/lib2to3/tests/data/py3_test_grammar.py
index d06223207e1ec0..a4a3f7eac0dded 100644
--- a/Lib/lib2to3/tests/data/py3_test_grammar.py
+++ b/Lib/lib2to3/tests/data/py3_test_grammar.py
@@ -8,7 +8,7 @@
# regression test, the filterwarnings() call has been added to
# regrtest.py.
-from test.support import run_unittest, check_syntax_error
+from test.support import check_syntax_error
import unittest
import sys
# testing import *
@@ -714,7 +714,7 @@ def testSelectors(self):
s = a[-5:]
s = a[:-1]
s = a[-4:-3]
- # A rough test of SF bug 1333982. http://python.org/sf/1333982
+ # A rough test of SF bug 1333982. https://python.org/sf/1333982
# The testing here is fairly incomplete.
# Test cases should include: commas with 1 and 2 colons
d = {}
@@ -952,8 +952,5 @@ def _checkeval(msg, ret):
self.assertEqual((6 < 4 if 0 else 2), 2)
-def test_main():
- run_unittest(TokenTests, GrammarTests)
-
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/lib2to3/tests/test_all_fixers.py b/Lib/lib2to3/tests/test_all_fixers.py
index c0507cf3fbcbfe..a265941490d5dc 100644
--- a/Lib/lib2to3/tests/test_all_fixers.py
+++ b/Lib/lib2to3/tests/test_all_fixers.py
@@ -6,8 +6,10 @@
# Author: Collin Winter
# Python imports
-import unittest
+import os.path
+import sys
import test.support
+import unittest
# Local imports
from . import support
@@ -19,9 +21,22 @@ class Test_all(support.TestCase):
def setUp(self):
self.refactor = support.get_refactorer()
+ def refactor_file(self, filepath):
+ if test.support.verbose:
+ print(f"Refactor file: {filepath}")
+ if os.path.basename(filepath) == 'infinite_recursion.py':
+ # bpo-46542: Processing infinite_recursion.py can crash Python
+ # if Python is built in debug mode: lower the recursion limit
+ # to prevent a crash.
+ with test.support.infinite_recursion(150):
+ self.refactor.refactor_file(filepath)
+ else:
+ self.refactor.refactor_file(filepath)
+
def test_all_project_files(self):
for filepath in support.all_project_files():
- self.refactor.refactor_file(filepath)
+ with self.subTest(filepath=filepath):
+ self.refactor_file(filepath)
if __name__ == '__main__':
unittest.main()
diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index 1aba0e88945a40..90a1b34b5fff85 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -20,6 +20,7 @@
import subprocess
import sys
import tempfile
+import test.support
import unittest
# Local imports
@@ -200,20 +201,27 @@ def test_async_var(self):
self.validate("""await = 1""")
self.validate("""def async(): pass""")
- def test_async_with(self):
+ def test_async_for(self):
self.validate("""async def foo():
async for a in b: pass""")
- self.invalid_syntax("""def foo():
- async for a in b: pass""")
-
- def test_async_for(self):
+ def test_async_with(self):
self.validate("""async def foo():
async with a: pass""")
self.invalid_syntax("""def foo():
async with a: pass""")
+ def test_async_generator(self):
+ self.validate(
+ """async def foo():
+ return (i * 2 async for i in arange(42))"""
+ )
+ self.validate(
+ """def foo():
+ return (i * 2 async for i in arange(42))"""
+ )
+
class TestRaiseChanges(GrammarTest):
def test_2x_style_1(self):
@@ -579,25 +587,31 @@ class TestParserIdempotency(support.TestCase):
"""A cut-down version of pytree_idempotency.py."""
+ def parse_file(self, filepath):
+ if test.support.verbose:
+ print(f"Parse file: {filepath}")
+ with open(filepath, "rb") as fp:
+ encoding = tokenize.detect_encoding(fp.readline)[0]
+ self.assertIsNotNone(encoding,
+ "can't detect encoding for %s" % filepath)
+ with open(filepath, "r", encoding=encoding) as fp:
+ source = fp.read()
+ try:
+ tree = driver.parse_string(source)
+ except ParseError:
+ try:
+ tree = driver_no_print_statement.parse_string(source)
+ except ParseError as err:
+ self.fail('ParseError on file %s (%s)' % (filepath, err))
+ new = str(tree)
+ if new != source:
+ print(diff_texts(source, new, filepath))
+ self.fail("Idempotency failed: %s" % filepath)
+
def test_all_project_files(self):
for filepath in support.all_project_files():
- with open(filepath, "rb") as fp:
- encoding = tokenize.detect_encoding(fp.readline)[0]
- self.assertIsNotNone(encoding,
- "can't detect encoding for %s" % filepath)
- with open(filepath, "r", encoding=encoding) as fp:
- source = fp.read()
- try:
- tree = driver.parse_string(source)
- except ParseError:
- try:
- tree = driver_no_print_statement.parse_string(source)
- except ParseError as err:
- self.fail('ParseError on file %s (%s)' % (filepath, err))
- new = str(tree)
- if new != source:
- print(diff_texts(source, new, filepath))
- self.fail("Idempotency failed: %s" % filepath)
+ with self.subTest(filepath=filepath):
+ self.parse_file(filepath)
def test_extended_unpacking(self):
driver.parse_string("a, *b, c = x\n")
diff --git a/Lib/linecache.py b/Lib/linecache.py
index 513b17e999880b..97644a8e3794e1 100644
--- a/Lib/linecache.py
+++ b/Lib/linecache.py
@@ -135,7 +135,7 @@ def updatecache(filename, module_globals=None):
try:
with tokenize.open(fullname) as fp:
lines = fp.readlines()
- except OSError:
+ except (OSError, UnicodeDecodeError, SyntaxError):
return []
if lines and not lines[-1].endswith('\n'):
lines[-1] += '\n'
@@ -154,7 +154,7 @@ def lazycache(filename, module_globals):
:return: True if a lazy load is registered in the cache,
otherwise False. To register such a load a module loader with a
- get_source method must be found, the filename must be a cachable
+ get_source method must be found, the filename must be a cacheable
filename, and the filename must not be already cached.
"""
if filename in cache:
diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
index 555f598de7a925..d1d43338585b64 100644
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -325,7 +325,7 @@ def __init__(self, name, level, pathname, lineno,
self.lineno = lineno
self.funcName = func
self.created = ct
- self.msecs = (ct - int(ct)) * 1000
+ self.msecs = int((ct - int(ct)) * 1000) + 0.0 # see gh-89047
self.relativeCreated = (self.created - _startTime) * 1000
if logThreads:
self.thread = threading.get_ident()
@@ -487,7 +487,7 @@ def __init__(self, *args, **kwargs):
def usesTime(self):
fmt = self._fmt
- return fmt.find('$asctime') >= 0 or fmt.find(self.asctime_format) >= 0
+ return fmt.find('$asctime') >= 0 or fmt.find(self.asctime_search) >= 0
def validate(self):
pattern = Template.pattern
@@ -878,6 +878,7 @@ def __init__(self, level=NOTSET):
self._name = None
self.level = _checkLevel(level)
self.formatter = None
+ self._closed = False
# Add the handler to the global _handlerList (for cleanup on shutdown)
_addHandlerRef(self)
self.createLock()
@@ -996,6 +997,7 @@ def close(self):
#get the module data lock, as we're updating a shared structure.
_acquireLock()
try: #unlikely to raise an exception, but you never know...
+ self._closed = True
if self._name and self._name in _handlers:
del _handlers[self._name]
finally:
@@ -1184,6 +1186,8 @@ def close(self):
finally:
# Issue #19523: call unconditionally to
# prevent a handler leak when delay is set
+ # Also see Issue #42378: we also rely on
+ # self._closed being set to True there
StreamHandler.close(self)
finally:
self.release()
@@ -1203,10 +1207,15 @@ def emit(self, record):
If the stream was not opened because 'delay' was specified in the
constructor, open it before calling the superclass's emit.
+
+ If stream is not open, current mode is 'w' and `_closed=True`, record
+ will not be emitted (see Issue #42378).
"""
if self.stream is None:
- self.stream = self._open()
- StreamHandler.emit(self, record)
+ if self.mode != 'w' or not self._closed:
+ self.stream = self._open()
+ if self.stream:
+ StreamHandler.emit(self, record)
def __repr__(self):
level = getLevelName(self.level)
diff --git a/Lib/logging/config.py b/Lib/logging/config.py
index 3bc63b78621aba..5cab008f8d1268 100644
--- a/Lib/logging/config.py
+++ b/Lib/logging/config.py
@@ -794,6 +794,7 @@ def configure_logger(self, name, config, incremental=False):
"""Configure a non-root logger from a dictionary."""
logger = logging.getLogger(name)
self.common_logger_config(logger, config, incremental)
+ logger.disabled = False
propagate = config.get('propagate', None)
if propagate is not None:
logger.propagate = propagate
diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py
index b0d5885989ea42..f0fdedae5ebb5c 100644
--- a/Lib/logging/handlers.py
+++ b/Lib/logging/handlers.py
@@ -1,4 +1,4 @@
-# Copyright 2001-2016 by Vinay Sajip. All Rights Reserved.
+# Copyright 2001-2021 by Vinay Sajip. All Rights Reserved.
#
# Permission to use, copy, modify, and distribute this software and its
# documentation for any purpose and without fee is hereby granted,
@@ -18,7 +18,7 @@
Additional handlers for the logging package for Python. The core package is
based on PEP 282 and comments thereto in comp.lang.python.
-Copyright (C) 2001-2016 Vinay Sajip. All Rights Reserved.
+Copyright (C) 2001-2021 Vinay Sajip. All Rights Reserved.
To use, simply 'import logging.handlers' and log away!
"""
@@ -187,14 +187,17 @@ def shouldRollover(self, record):
Basically, see if the supplied record would cause the file to exceed
the size limit we have.
"""
+ # See bpo-45401: Never rollover anything other than regular files
+ if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename):
+ return False
if self.stream is None: # delay was set...
self.stream = self._open()
if self.maxBytes > 0: # are we rolling over?
msg = "%s\n" % self.format(record)
self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
if self.stream.tell() + len(msg) >= self.maxBytes:
- return 1
- return 0
+ return True
+ return False
class TimedRotatingFileHandler(BaseRotatingHandler):
"""
@@ -347,8 +350,15 @@ def shouldRollover(self, record):
"""
t = int(time.time())
if t >= self.rolloverAt:
- return 1
- return 0
+ # See #89564: Never rollover anything other than regular files
+ if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename):
+ # The file is not a regular file, so do not rollover, but do
+ # set the next rollover time to avoid repeated checks.
+ self.rolloverAt = self.computeRollover(t)
+ return False
+
+ return True
+ return False
def getFilesToDelete(self):
"""
@@ -359,13 +369,32 @@ def getFilesToDelete(self):
dirName, baseName = os.path.split(self.baseFilename)
fileNames = os.listdir(dirName)
result = []
- prefix = baseName + "."
+ # See bpo-44753: Don't use the extension when computing the prefix.
+ n, e = os.path.splitext(baseName)
+ prefix = n + '.'
plen = len(prefix)
for fileName in fileNames:
+ if self.namer is None:
+ # Our files will always start with baseName
+ if not fileName.startswith(baseName):
+ continue
+ else:
+ # Our files could be just about anything after custom naming, but
+ # likely candidates are of the form
+ # foo.log.DATETIME_SUFFIX or foo.DATETIME_SUFFIX.log
+ if (not fileName.startswith(baseName) and fileName.endswith(e) and
+ len(fileName) > (plen + 1) and not fileName[plen+1].isdigit()):
+ continue
+
if fileName[:plen] == prefix:
suffix = fileName[plen:]
- if self.extMatch.match(suffix):
- result.append(os.path.join(dirName, fileName))
+ # See bpo-45628: The date/time suffix could be anywhere in the
+ # filename
+ parts = suffix.split('.')
+ for part in parts:
+ if self.extMatch.match(part):
+ result.append(os.path.join(dirName, fileName))
+ break
if len(result) < self.backupCount:
result = []
else:
@@ -1069,7 +1098,16 @@ def __init__(self, appname, dllname=None, logtype="Application"):
dllname = os.path.join(dllname[0], r'win32service.pyd')
self.dllname = dllname
self.logtype = logtype
- self._welu.AddSourceToRegistry(appname, dllname, logtype)
+ # Administrative privileges are required to add a source to the registry.
+ # This may not be available for a user that just wants to add to an
+ # existing source - handle this specific case.
+ try:
+ self._welu.AddSourceToRegistry(appname, dllname, logtype)
+ except Exception as e:
+ # This will probably be a pywintypes.error. Only raise if it's not
+ # an "access denied" error, else let it pass
+ if getattr(e, 'winerror', None) != 5: # not access denied
+ raise
self.deftype = win32evtlog.EVENTLOG_ERROR_TYPE
self.typemap = {
logging.DEBUG : win32evtlog.EVENTLOG_INFORMATION_TYPE,
@@ -1147,7 +1185,7 @@ def close(self):
class HTTPHandler(logging.Handler):
"""
- A class which sends records to a Web server, using either GET or
+ A class which sends records to a web server, using either GET or
POST semantics.
"""
def __init__(self, host, url, method="GET", secure=False, credentials=None,
@@ -1196,7 +1234,7 @@ def emit(self, record):
"""
Emit a record.
- Send the record to the Web server as a percent-encoded dictionary
+ Send the record to the web server as a percent-encoded dictionary
"""
try:
import urllib.parse
@@ -1414,7 +1452,7 @@ def prepare(self, record):
# (if there's exception data), and also returns the formatted
# message. We can then use this to replace the original
# msg + args, as these might be unpickleable. We also zap the
- # exc_info and exc_text attributes, as they are no longer
+ # exc_info, exc_text and stack_info attributes, as they are no longer
# needed and, if not None, will typically not be pickleable.
msg = self.format(record)
# bpo-35726: make copy of record to avoid affecting other handlers in the chain.
@@ -1424,6 +1462,7 @@ def prepare(self, record):
record.args = None
record.exc_info = None
record.exc_text = None
+ record.stack_info = None
return record
def emit(self, record):
diff --git a/Lib/lzma.py b/Lib/lzma.py
index 2ada7d81d3c813..800f52198fbb79 100644
--- a/Lib/lzma.py
+++ b/Lib/lzma.py
@@ -221,22 +221,26 @@ def readline(self, size=-1):
self._check_can_read()
return self._buffer.readline(size)
- def __iter__(self):
- self._check_can_read()
- return self._buffer.__iter__()
-
def write(self, data):
"""Write a bytes object to the file.
Returns the number of uncompressed bytes written, which is
- always len(data). Note that due to buffering, the file on disk
- may not reflect the data written until close() is called.
+ always the length of data in bytes. Note that due to buffering,
+ the file on disk may not reflect the data written until close()
+ is called.
"""
self._check_can_write()
+ if isinstance(data, (bytes, bytearray)):
+ length = len(data)
+ else:
+ # accept any data that supports the buffer protocol
+ data = memoryview(data)
+ length = data.nbytes
+
compressed = self._compressor.compress(data)
self._fp.write(compressed)
- self._pos += len(data)
- return len(data)
+ self._pos += length
+ return length
def seek(self, offset, whence=io.SEEK_SET):
"""Change the file position.
diff --git a/Lib/mailcap.py b/Lib/mailcap.py
index ae416a8e9fb273..444c6408b54c6b 100644
--- a/Lib/mailcap.py
+++ b/Lib/mailcap.py
@@ -2,6 +2,7 @@
import os
import warnings
+import re
__all__ = ["getcaps","findmatch"]
@@ -13,6 +14,11 @@ def lineno_sort_key(entry):
else:
return 1, 0
+_find_unsafe = re.compile(r'[^\xa1-\U0010FFFF\w@+=:,./-]').search
+
+class UnsafeMailcapInput(Warning):
+ """Warning raised when refusing unsafe input"""
+
# Part 1: top-level interface.
@@ -165,15 +171,22 @@ def findmatch(caps, MIMEtype, key='view', filename="/dev/null", plist=[]):
entry to use.
"""
+ if _find_unsafe(filename):
+ msg = "Refusing to use mailcap with filename %r. Use a safe temporary filename." % (filename,)
+ warnings.warn(msg, UnsafeMailcapInput)
+ return None, None
entries = lookup(caps, MIMEtype, key)
# XXX This code should somehow check for the needsterminal flag.
for e in entries:
if 'test' in e:
test = subst(e['test'], filename, plist)
+ if test is None:
+ continue
if test and os.system(test) != 0:
continue
command = subst(e[key], MIMEtype, filename, plist)
- return command, e
+ if command is not None:
+ return command, e
return None, None
def lookup(caps, MIMEtype, key=None):
@@ -206,6 +219,10 @@ def subst(field, MIMEtype, filename, plist=[]):
elif c == 's':
res = res + filename
elif c == 't':
+ if _find_unsafe(MIMEtype):
+ msg = "Refusing to substitute MIME type %r into a shell command." % (MIMEtype,)
+ warnings.warn(msg, UnsafeMailcapInput)
+ return None
res = res + MIMEtype
elif c == '{':
start = i
@@ -213,7 +230,12 @@ def subst(field, MIMEtype, filename, plist=[]):
i = i+1
name = field[start:i]
i = i+1
- res = res + findparam(name, plist)
+ param = findparam(name, plist)
+ if _find_unsafe(param):
+ msg = "Refusing to substitute parameter %r (%s) into a shell command" % (param, name)
+ warnings.warn(msg, UnsafeMailcapInput)
+ return None
+ res = res + param
# XXX To do:
# %n == number of parts if type is multipart/*
# %F == list of alternating type and filename for parts
diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
index 018793c4f01e46..b72ce083a69d06 100644
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -27,6 +27,12 @@
import sys
import posixpath
import urllib.parse
+
+try:
+ from _winapi import _mimetypes_read_windows_registry
+except ImportError:
+ _mimetypes_read_windows_registry = None
+
try:
import winreg as _winreg
except ImportError:
@@ -135,25 +141,23 @@ def guess_type(self, url, strict=True):
type = 'text/plain'
return type, None # never compressed, so encoding is None
base, ext = posixpath.splitext(url)
- while ext in self.suffix_map:
- base, ext = posixpath.splitext(base + self.suffix_map[ext])
+ while (ext_lower := ext.lower()) in self.suffix_map:
+ base, ext = posixpath.splitext(base + self.suffix_map[ext_lower])
+ # encodings_map is case sensitive
if ext in self.encodings_map:
encoding = self.encodings_map[ext]
base, ext = posixpath.splitext(base)
else:
encoding = None
+ ext = ext.lower()
types_map = self.types_map[True]
if ext in types_map:
return types_map[ext], encoding
- elif ext.lower() in types_map:
- return types_map[ext.lower()], encoding
elif strict:
return None, encoding
types_map = self.types_map[False]
if ext in types_map:
return types_map[ext], encoding
- elif ext.lower() in types_map:
- return types_map[ext.lower()], encoding
else:
return None, encoding
@@ -169,7 +173,7 @@ def guess_all_extensions(self, type, strict=True):
but non-standard types.
"""
type = type.lower()
- extensions = self.types_map_inv[True].get(type, [])
+ extensions = list(self.types_map_inv[True].get(type, []))
if not strict:
for ext in self.types_map_inv[False].get(type, []):
if ext not in extensions:
@@ -237,10 +241,21 @@ def read_windows_registry(self, strict=True):
types.
"""
- # Windows only
- if not _winreg:
+ if not _mimetypes_read_windows_registry and not _winreg:
return
+ add_type = self.add_type
+ if strict:
+ add_type = lambda type, ext: self.add_type(type, ext, True)
+
+ # Accelerated function if it is available
+ if _mimetypes_read_windows_registry:
+ _mimetypes_read_windows_registry(add_type)
+ elif _winreg:
+ self._read_windows_registry(add_type)
+
+ @classmethod
+ def _read_windows_registry(cls, add_type):
def enum_types(mimedb):
i = 0
while True:
@@ -265,7 +280,7 @@ def enum_types(mimedb):
subkey, 'Content Type')
if datatype != _winreg.REG_SZ:
continue
- self.add_type(mimetype, subkeyname, strict)
+ add_type(mimetype, subkeyname)
except OSError:
continue
@@ -349,8 +364,8 @@ def init(files=None):
if files is None or _db is None:
db = MimeTypes()
- if _winreg:
- db.read_windows_registry()
+ # Quick return if not supported
+ db.read_windows_registry()
if files is None:
files = knownfiles
diff --git a/Lib/msilib/schema.py b/Lib/msilib/schema.py
index eeb3ecd27ab658..9f5745cfb14de2 100644
--- a/Lib/msilib/schema.py
+++ b/Lib/msilib/schema.py
@@ -664,7 +664,7 @@
('Class','Component_','N',None, None, 'Component',1,'Identifier',None, 'Required foreign key into the Component Table, specifying the component for which to return a path when called through LocateComponent.',),
('Class','Context','N',None, None, None, None, 'Identifier',None, 'The numeric server context for this server. CLSCTX_xxxx',),
('Class','DefInprocHandler','Y',None, None, None, None, 'Filename','1;2;3','Optional default inproc handler. Only optionally provided if Context=CLSCTX_LOCAL_SERVER. Typically "ole32.dll" or "mapi32.dll"',),
-('Class','FileTypeMask','Y',None, None, None, None, 'Text',None, 'Optional string containing information for the HKCRthis CLSID) key. If multiple patterns exist, they must be delimited by a semicolon, and numeric subkeys will be generated: 0,1,2...',),
+('Class','FileTypeMask','Y',None, None, None, None, 'Text',None, 'Optional string containing information for the HKCRthis CLSID key. If multiple patterns exist, they must be delimited by a semicolon, and numeric subkeys will be generated: 0,1,2...',),
('Class','Icon_','Y',None, None, 'Icon',1,'Identifier',None, 'Optional foreign key into the Icon Table, specifying the icon file associated with this CLSID. Will be written under the DefaultIcon key.',),
('Class','IconIndex','Y',-32767,32767,None, None, None, None, 'Optional icon index.',),
('Class','ProgId_Default','Y',None, None, 'ProgId',1,'Text',None, 'Optional ProgId associated with this CLSID.',),
diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py
index 510e4b5aba44a6..8e2facf92a94aa 100644
--- a/Lib/multiprocessing/connection.py
+++ b/Lib/multiprocessing/connection.py
@@ -73,11 +73,6 @@ def arbitrary_address(family):
if family == 'AF_INET':
return ('localhost', 0)
elif family == 'AF_UNIX':
- # Prefer abstract sockets if possible to avoid problems with the address
- # size. When coding portable applications, some implementations have
- # sun_path as short as 92 bytes in the sockaddr_un struct.
- if util.abstract_sockets_supported:
- return f"\0listener-{os.getpid()}-{next(_mmap_counter)}"
return tempfile.mktemp(prefix='listener-', dir=util.get_temp_dir())
elif family == 'AF_PIPE':
return tempfile.mktemp(prefix=r'\\.\pipe\pyc-%d-%d-' %
diff --git a/Lib/multiprocessing/context.py b/Lib/multiprocessing/context.py
index 8d0525d5d62179..b1960ea296fe20 100644
--- a/Lib/multiprocessing/context.py
+++ b/Lib/multiprocessing/context.py
@@ -223,6 +223,10 @@ class Process(process.BaseProcess):
def _Popen(process_obj):
return _default_context.get_context().Process._Popen(process_obj)
+ @staticmethod
+ def _after_fork():
+ return _default_context.get_context().Process._after_fork()
+
class DefaultContext(BaseContext):
Process = Process
@@ -283,6 +287,11 @@ def _Popen(process_obj):
from .popen_spawn_posix import Popen
return Popen(process_obj)
+ @staticmethod
+ def _after_fork():
+ # process is spawned, nothing to do
+ pass
+
class ForkServerProcess(process.BaseProcess):
_start_method = 'forkserver'
@staticmethod
@@ -326,6 +335,11 @@ def _Popen(process_obj):
from .popen_spawn_win32 import Popen
return Popen(process_obj)
+ @staticmethod
+ def _after_fork():
+ # process is spawned, nothing to do
+ pass
+
class SpawnContext(BaseContext):
_name = 'spawn'
Process = SpawnProcess
diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py
index 6ee9521c76b48f..22292c78b7b5b3 100644
--- a/Lib/multiprocessing/managers.py
+++ b/Lib/multiprocessing/managers.py
@@ -8,8 +8,7 @@
# Licensed to PSF under a Contributor Agreement.
#
-__all__ = [ 'BaseManager', 'SyncManager', 'BaseProxy', 'Token',
- 'SharedMemoryManager' ]
+__all__ = [ 'BaseManager', 'SyncManager', 'BaseProxy', 'Token' ]
#
# Imports
@@ -35,9 +34,11 @@
from . import get_context
try:
from . import shared_memory
- HAS_SHMEM = True
except ImportError:
HAS_SHMEM = False
+else:
+ HAS_SHMEM = True
+ __all__.append('SharedMemoryManager')
#
# Register some things for pickling
@@ -676,7 +677,7 @@ def _finalize_manager(process, address, authkey, state, _Client):
if hasattr(process, 'terminate'):
util.info('trying to `terminate()` manager process')
process.terminate()
- process.join(timeout=0.1)
+ process.join(timeout=1.0)
if process.is_alive():
util.info('manager still alive after terminate')
@@ -967,7 +968,7 @@ def MakeProxyType(name, exposed, _cache={}):
def AutoProxy(token, serializer, manager=None, authkey=None,
- exposed=None, incref=True):
+ exposed=None, incref=True, manager_owned=False):
'''
Return an auto-proxy for `token`
'''
@@ -987,7 +988,7 @@ def AutoProxy(token, serializer, manager=None, authkey=None,
ProxyType = MakeProxyType('AutoProxy[%s]' % token.typeid, exposed)
proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,
- incref=incref)
+ incref=incref, manager_owned=manager_owned)
proxy._isauto = True
return proxy
diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py
index bbe05a550c349c..961d7e5991847a 100644
--- a/Lib/multiprocessing/pool.py
+++ b/Lib/multiprocessing/pool.py
@@ -203,6 +203,9 @@ def __init__(self, processes=None, initializer=None, initargs=(),
processes = os.cpu_count() or 1
if processes < 1:
raise ValueError("Number of processes must be at least 1")
+ if maxtasksperchild is not None:
+ if not isinstance(maxtasksperchild, int) or maxtasksperchild <= 0:
+ raise ValueError("maxtasksperchild must be a positive int or None")
if initializer is not None and not callable(initializer):
raise TypeError('initializer must be a callable')
diff --git a/Lib/multiprocessing/process.py b/Lib/multiprocessing/process.py
index 0b2e0b45b2397b..2d5372e6d947c8 100644
--- a/Lib/multiprocessing/process.py
+++ b/Lib/multiprocessing/process.py
@@ -304,8 +304,7 @@ def _bootstrap(self, parent_sentinel=None):
if threading._HAVE_THREAD_NATIVE_ID:
threading.main_thread()._set_native_id()
try:
- util._finalizer_registry.clear()
- util._run_after_forkers()
+ self._after_fork()
finally:
# delay finalization of the old process object until after
# _run_after_forkers() is executed
@@ -336,6 +335,13 @@ def _bootstrap(self, parent_sentinel=None):
return exitcode
+ @staticmethod
+ def _after_fork():
+ from . import util
+ util._finalizer_registry.clear()
+ util._run_after_forkers()
+
+
#
# We subclass bytes to avoid accidental transmission of auth keys over network
#
diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
index a2901814876d6c..f37f114a968871 100644
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -139,13 +139,10 @@ def put_nowait(self, obj):
def close(self):
self._closed = True
- try:
- self._reader.close()
- finally:
- close = self._close
- if close:
- self._close = None
- close()
+ close = self._close
+ if close:
+ self._close = None
+ close()
def join_thread(self):
debug('Queue.join_thread()')
@@ -169,8 +166,9 @@ def _start_thread(self):
self._thread = threading.Thread(
target=Queue._feed,
args=(self._buffer, self._notempty, self._send_bytes,
- self._wlock, self._writer.close, self._ignore_epipe,
- self._on_queue_feeder_error, self._sem),
+ self._wlock, self._reader.close, self._writer.close,
+ self._ignore_epipe, self._on_queue_feeder_error,
+ self._sem),
name='QueueFeederThread'
)
self._thread.daemon = True
@@ -211,8 +209,8 @@ def _finalize_close(buffer, notempty):
notempty.notify()
@staticmethod
- def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
- onerror, queue_sem):
+ def _feed(buffer, notempty, send_bytes, writelock, reader_close,
+ writer_close, ignore_epipe, onerror, queue_sem):
debug('starting thread to feed data to pipe')
nacquire = notempty.acquire
nrelease = notempty.release
@@ -238,7 +236,8 @@ def _feed(buffer, notempty, send_bytes, writelock, close, ignore_epipe,
obj = bpopleft()
if obj is sentinel:
debug('feeder thread got sentinel -- exiting')
- close()
+ reader_close()
+ writer_close()
return
# serialize the data before acquiring the lock
diff --git a/Lib/multiprocessing/resource_tracker.py b/Lib/multiprocessing/resource_tracker.py
index cc42dbdda05b91..ea369507297f86 100644
--- a/Lib/multiprocessing/resource_tracker.py
+++ b/Lib/multiprocessing/resource_tracker.py
@@ -161,10 +161,10 @@ def unregister(self, name, rtype):
def _send(self, cmd, name, rtype):
self.ensure_running()
msg = '{0}:{1}:{2}\n'.format(cmd, name, rtype).encode('ascii')
- if len(name) > 512:
+ if len(msg) > 512:
# posix guarantees that writes to a pipe of less than PIPE_BUF
# bytes are atomic, and that PIPE_BUF >= 512
- raise ValueError('name too long')
+ raise ValueError('msg too long')
nbytes = os.write(self._fd, msg)
assert nbytes == len(msg), "nbytes {0:n} but len(msg) {1:n}".format(
nbytes, len(msg))
diff --git a/Lib/multiprocessing/shared_memory.py b/Lib/multiprocessing/shared_memory.py
index 122b3fcebf3fed..9a1e5aa17b87a2 100644
--- a/Lib/multiprocessing/shared_memory.py
+++ b/Lib/multiprocessing/shared_memory.py
@@ -23,6 +23,7 @@
import _posixshmem
_USE_POSIX = True
+from . import resource_tracker
_O_CREX = os.O_CREAT | os.O_EXCL
@@ -116,8 +117,7 @@ def __init__(self, name=None, create=False, size=0):
self.unlink()
raise
- from .resource_tracker import register
- register(self._name, "shared_memory")
+ resource_tracker.register(self._name, "shared_memory")
else:
@@ -173,7 +173,10 @@ def __init__(self, name=None, create=False, size=0):
)
finally:
_winapi.CloseHandle(h_map)
- size = _winapi.VirtualQuerySize(p_buf)
+ try:
+ size = _winapi.VirtualQuerySize(p_buf)
+ finally:
+ _winapi.UnmapViewOfFile(p_buf)
self._mmap = mmap.mmap(-1, size, tagname=name)
self._size = size
@@ -237,9 +240,8 @@ def unlink(self):
called once (and only once) across all processes which have access
to the shared memory block."""
if _USE_POSIX and self._name:
- from .resource_tracker import unregister
_posixshmem.shm_unlink(self._name)
- unregister(self._name, "shared_memory")
+ resource_tracker.unregister(self._name, "shared_memory")
_encoding = "utf8"
diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py
index a4683339820f5f..9e07a4e93e597e 100644
--- a/Lib/multiprocessing/util.py
+++ b/Lib/multiprocessing/util.py
@@ -120,7 +120,7 @@ def is_abstract_socket_namespace(address):
return address[0] == 0
elif isinstance(address, str):
return address[0] == "\0"
- raise TypeError('address type of {address!r} unrecognized')
+ raise TypeError(f'address type of {address!r} unrecognized')
abstract_sockets_supported = _platform_supports_abstract_sockets()
diff --git a/Lib/ntpath.py b/Lib/ntpath.py
index 527c7ae1938fbb..c14e5c7ceca5de 100644
--- a/Lib/ntpath.py
+++ b/Lib/ntpath.py
@@ -23,6 +23,7 @@
import genericpath
from genericpath import *
+
__all__ = ["normcase","isabs","join","splitdrive","split","splitext",
"basename","dirname","commonprefix","getsize","getmtime",
"getatime","getctime", "islink","exists","lexists","isdir","isfile",
@@ -41,14 +42,39 @@ def _get_bothseps(path):
# Other normalizations (such as optimizing '../' away) are not done
# (this is done by normpath).
-def normcase(s):
- """Normalize case of pathname.
-
- Makes all characters lowercase and all slashes into backslashes."""
- s = os.fspath(s)
- if isinstance(s, bytes):
- return s.replace(b'/', b'\\').lower()
- else:
+try:
+ from _winapi import (
+ LCMapStringEx as _LCMapStringEx,
+ LOCALE_NAME_INVARIANT as _LOCALE_NAME_INVARIANT,
+ LCMAP_LOWERCASE as _LCMAP_LOWERCASE)
+
+ def normcase(s):
+ """Normalize case of pathname.
+
+ Makes all characters lowercase and all slashes into backslashes.
+ """
+ s = os.fspath(s)
+ if not s:
+ return s
+ if isinstance(s, bytes):
+ encoding = sys.getfilesystemencoding()
+ s = s.decode(encoding, 'surrogateescape').replace('/', '\\')
+ s = _LCMapStringEx(_LOCALE_NAME_INVARIANT,
+ _LCMAP_LOWERCASE, s)
+ return s.encode(encoding, 'surrogateescape')
+ else:
+ return _LCMapStringEx(_LOCALE_NAME_INVARIANT,
+ _LCMAP_LOWERCASE,
+ s.replace('/', '\\'))
+except ImportError:
+ def normcase(s):
+ """Normalize case of pathname.
+
+ Makes all characters lowercase and all slashes into backslashes.
+ """
+ s = os.fspath(s)
+ if isinstance(s, bytes):
+ return os.fsencode(os.fsdecode(s).replace('/', '\\').lower())
return s.replace('/', '\\').lower()
@@ -599,12 +625,15 @@ def _getfinalpathname_nonstrict(path):
# 21: ERROR_NOT_READY (implies drive with no media)
# 32: ERROR_SHARING_VIOLATION (probably an NTFS paging file)
# 50: ERROR_NOT_SUPPORTED
+ # 53: ERROR_BAD_NETPATH
+ # 65: ERROR_NETWORK_ACCESS_DENIED
# 67: ERROR_BAD_NET_NAME (implies remote server unavailable)
# 87: ERROR_INVALID_PARAMETER
# 123: ERROR_INVALID_NAME
+ # 161: ERROR_BAD_PATHNAME
# 1920: ERROR_CANT_ACCESS_FILE
# 1921: ERROR_CANT_RESOLVE_FILENAME (implies unfollowable symlink)
- allowed_winerror = 1, 2, 3, 5, 21, 32, 50, 67, 87, 123, 1920, 1921
+ allowed_winerror = 1, 2, 3, 5, 21, 32, 50, 53, 65, 67, 87, 123, 161, 1920, 1921
# Non-strict algorithm is to find as much of the target directory
# as we can and join the rest.
diff --git a/Lib/numbers.py b/Lib/numbers.py
index ed815ef41ebe12..0985dd85f60a78 100644
--- a/Lib/numbers.py
+++ b/Lib/numbers.py
@@ -33,7 +33,7 @@ class Complex(Number):
"""Complex defines the operations that work on the builtin complex type.
In short, those are: a conversion to complex, .real, .imag, +, -,
- *, /, abs(), .conjugate, ==, and !=.
+ *, /, **, abs(), .conjugate, ==, and !=.
If it is given heterogeneous arguments, and doesn't have special
knowledge about them, it should fall back to the builtin complex
@@ -288,11 +288,15 @@ def __float__(self):
so that ratios of huge integers convert without overflowing.
"""
- return self.numerator / self.denominator
+ return int(self.numerator) / int(self.denominator)
class Integral(Rational):
- """Integral adds a conversion to int and the bit-string operations."""
+ """Integral adds methods that work on integral numbers.
+
+ In short, these are conversion to int, pow with modulus, and the
+ bit-string operations.
+ """
__slots__ = ()
diff --git a/Lib/operator.py b/Lib/operator.py
index fb58851fa6ef67..241fdbb679e7c1 100644
--- a/Lib/operator.py
+++ b/Lib/operator.py
@@ -155,10 +155,10 @@ def contains(a, b):
return b in a
def countOf(a, b):
- "Return the number of times b occurs in a."
+ "Return the number of items in a which are, or which equal, b."
count = 0
for i in a:
- if i == b:
+ if i is b or i == b:
count += 1
return count
@@ -173,7 +173,7 @@ def getitem(a, b):
def indexOf(a, b):
"Return the first index of b in a."
for i, j in enumerate(a):
- if j == b:
+ if j is b or j == b:
return i
else:
raise ValueError('sequence.index(x): x not in sequence')
diff --git a/Lib/os.py b/Lib/os.py
index d26cfc99939f39..4f2ffceaaf3835 100644
--- a/Lib/os.py
+++ b/Lib/os.py
@@ -288,7 +288,8 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
dirpath, dirnames, filenames
dirpath is a string, the path to the directory. dirnames is a list of
- the names of the subdirectories in dirpath (excluding '.' and '..').
+ the names of the subdirectories in dirpath (including symlinks to directories,
+ and excluding '.' and '..').
filenames is a list of the names of the non-directory files in dirpath.
Note that the names in the lists are just names, with no path components.
To get a full path (which begins with top) to a file or directory in
diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index cf40370c049a31..97b23ca45a3a19 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -124,16 +124,25 @@ class _WindowsFlavour(_Flavour):
ext_namespace_prefix = '\\\\?\\'
reserved_names = (
- {'CON', 'PRN', 'AUX', 'NUL'} |
- {'COM%d' % i for i in range(1, 10)} |
- {'LPT%d' % i for i in range(1, 10)}
+ {'CON', 'PRN', 'AUX', 'NUL', 'CONIN$', 'CONOUT$'} |
+ {'COM%s' % c for c in '123456789\xb9\xb2\xb3'} |
+ {'LPT%s' % c for c in '123456789\xb9\xb2\xb3'}
)
# Interesting findings about extended paths:
- # - '\\?\c:\a', '//?/c:\a' and '//?/c:/a' are all supported
- # but '\\?\c:/a' is not
- # - extended paths are always absolute; "relative" extended paths will
- # fail.
+ # * '\\?\c:\a' is an extended path, which bypasses normal Windows API
+ # path processing. Thus relative paths are not resolved and slash is not
+ # translated to backslash. It has the native NT path limit of 32767
+ # characters, but a bit less after resolving device symbolic links,
+ # such as '\??\C:' => '\Device\HarddiskVolume2'.
+ # * '\\?\c:/a' looks for a device named 'C:/a' because slash is a
+ # regular name character in the object namespace.
+ # * '\\?\c:\foo/bar' is invalid because '/' is illegal in NT filesystems.
+ # The only path separator at the filesystem level is backslash.
+ # * '//?/c:\a' and '//?/c:/a' are effectively equivalent to '\\.\c:\a' and
+ # thus limited to MAX_PATH.
+ # * Prior to Windows 8, ANSI API bytes paths are limited to MAX_PATH,
+ # even with the '\\?\' prefix.
def splitroot(self, part, sep=sep):
first = part[0:1]
@@ -195,15 +204,16 @@ def _split_extended_path(self, s, ext_prefix=ext_namespace_prefix):
def is_reserved(self, parts):
# NOTE: the rules for reserved names seem somewhat complicated
- # (e.g. r"..\NUL" is reserved but not r"foo\NUL").
- # We err on the side of caution and return True for paths which are
- # not considered reserved by Windows.
+ # (e.g. r"..\NUL" is reserved but not r"foo\NUL" if "foo" does not
+ # exist). We err on the side of caution and return True for paths
+ # which are not considered reserved by Windows.
if not parts:
return False
if parts[0].startswith('\\\\'):
# UNC paths are never reserved
return False
- return parts[-1].partition('.')[0].upper() in self.reserved_names
+ name = parts[-1].partition('.')[0].partition(':')[0].rstrip(' ')
+ return name.upper() in self.reserved_names
def make_uri(self, path):
# Under Windows, file URIs use the UTF-8 encoding.
@@ -518,6 +528,8 @@ def __getitem__(self, idx):
if idx >= len(self) or idx < -len(self):
raise IndexError(idx)
+ if idx < 0:
+ idx += len(self)
return self._pathcls._from_parsed_parts(self._drv, self._root,
self._parts[:-idx - 1])
@@ -1265,7 +1277,7 @@ def link_to(self, target):
warnings.warn("pathlib.Path.link_to() is deprecated and is scheduled "
"for removal in Python 3.12. "
"Use pathlib.Path.hardlink_to() instead.",
- DeprecationWarning)
+ DeprecationWarning, stacklevel=2)
self._accessor.link(self, target)
# Convenience functions for querying the stat results
@@ -1295,7 +1307,7 @@ def is_dir(self):
if not _ignore_error(e):
raise
# Path doesn't exist or is a broken symlink
- # (see https://bitbucket.org/pitrou/pathlib/issue/12/)
+ # (see http://web.archive.org/web/20200623061726/https://bitbucket.org/pitrou/pathlib/issues/12/ )
return False
except ValueError:
# Non-encodable path
@@ -1312,7 +1324,7 @@ def is_file(self):
if not _ignore_error(e):
raise
# Path doesn't exist or is a broken symlink
- # (see https://bitbucket.org/pitrou/pathlib/issue/12/)
+ # (see http://web.archive.org/web/20200623061726/https://bitbucket.org/pitrou/pathlib/issues/12/ )
return False
except ValueError:
# Non-encodable path
@@ -1363,7 +1375,7 @@ def is_block_device(self):
if not _ignore_error(e):
raise
# Path doesn't exist or is a broken symlink
- # (see https://bitbucket.org/pitrou/pathlib/issue/12/)
+ # (see http://web.archive.org/web/20200623061726/https://bitbucket.org/pitrou/pathlib/issues/12/ )
return False
except ValueError:
# Non-encodable path
@@ -1379,7 +1391,7 @@ def is_char_device(self):
if not _ignore_error(e):
raise
# Path doesn't exist or is a broken symlink
- # (see https://bitbucket.org/pitrou/pathlib/issue/12/)
+ # (see http://web.archive.org/web/20200623061726/https://bitbucket.org/pitrou/pathlib/issues/12/ )
return False
except ValueError:
# Non-encodable path
@@ -1395,7 +1407,7 @@ def is_fifo(self):
if not _ignore_error(e):
raise
# Path doesn't exist or is a broken symlink
- # (see https://bitbucket.org/pitrou/pathlib/issue/12/)
+ # (see http://web.archive.org/web/20200623061726/https://bitbucket.org/pitrou/pathlib/issues/12/ )
return False
except ValueError:
# Non-encodable path
@@ -1411,7 +1423,7 @@ def is_socket(self):
if not _ignore_error(e):
raise
# Path doesn't exist or is a broken symlink
- # (see https://bitbucket.org/pitrou/pathlib/issue/12/)
+ # (see http://web.archive.org/web/20200623061726/https://bitbucket.org/pitrou/pathlib/issues/12/ )
return False
except ValueError:
# Non-encodable path
diff --git a/Lib/pdb.py b/Lib/pdb.py
index 98dc975eafafdc..3bdb2ac36f7192 100755
--- a/Lib/pdb.py
+++ b/Lib/pdb.py
@@ -104,15 +104,6 @@ def find_function(funcname, filename):
return funcname, filename, lineno
return None
-def getsourcelines(obj):
- lines, lineno = inspect.findsource(obj)
- if inspect.isframe(obj) and obj.f_globals is obj.f_locals:
- # must be a module frame: do not try to cut a block out of it
- return lines, 1
- elif inspect.ismodule(obj):
- return lines, 1
- return inspect.getblock(lines[lineno:]), lineno+1
-
def lasti2lineno(code, lasti):
linestarts = list(dis.findlinestarts(code))
linestarts.reverse()
@@ -384,8 +375,7 @@ def default(self, line):
sys.stdin = save_stdin
sys.displayhook = save_displayhook
except:
- exc_info = sys.exc_info()[:2]
- self.error(traceback.format_exception_only(*exc_info)[-1].strip())
+ self._error_exc()
def precmd(self, line):
"""Handle alias expansion and ';;' separator."""
@@ -752,7 +742,8 @@ def checkline(self, filename, lineno):
"""
# this method should be callable before starting debugging, so default
# to "no globals" if there is no current frame
- globs = self.curframe.f_globals if hasattr(self, 'curframe') else None
+ frame = getattr(self, 'curframe', None)
+ globs = frame.f_globals if frame else None
line = linecache.getline(filename, lineno, globs)
if not line:
self.message('End of file')
@@ -893,7 +884,7 @@ def do_clear(self, arg):
except ValueError:
err = "Invalid line number (%s)" % arg
else:
- bplist = self.get_breaks(filename, lineno)
+ bplist = self.get_breaks(filename, lineno)[:]
err = self.clear_break(filename, lineno)
if err:
self.error(err)
@@ -1026,7 +1017,11 @@ def do_run(self, arg):
if arg:
import shlex
argv0 = sys.argv[0:1]
- sys.argv = shlex.split(arg)
+ try:
+ sys.argv = shlex.split(arg)
+ except ValueError as e:
+ self.error('Cannot run %s: %s' % (arg, e))
+ return
sys.argv[:0] = argv0
# this is caught in the main debugger loop
raise Restart
@@ -1103,8 +1098,7 @@ def do_debug(self, arg):
try:
sys.call_tracing(p.run, (arg, globals, locals))
except Exception:
- exc_info = sys.exc_info()[:2]
- self.error(traceback.format_exception_only(*exc_info)[-1].strip())
+ self._error_exc()
self.message("LEAVING RECURSIVE DEBUGGER")
sys.settrace(self.trace_dispatch)
self.lastcmd = p.lastcmd
@@ -1162,8 +1156,7 @@ def _getval(self, arg):
try:
return eval(arg, self.curframe.f_globals, self.curframe_locals)
except:
- exc_info = sys.exc_info()[:2]
- self.error(traceback.format_exception_only(*exc_info)[-1].strip())
+ self._error_exc()
raise
def _getval_except(self, arg, frame=None):
@@ -1177,23 +1170,31 @@ def _getval_except(self, arg, frame=None):
err = traceback.format_exception_only(*exc_info)[-1].strip()
return _rstr('** raised %s **' % err)
+ def _error_exc(self):
+ exc_info = sys.exc_info()[:2]
+ self.error(traceback.format_exception_only(*exc_info)[-1].strip())
+
+ def _msg_val_func(self, arg, func):
+ try:
+ val = self._getval(arg)
+ except:
+ return # _getval() has displayed the error
+ try:
+ self.message(func(val))
+ except:
+ self._error_exc()
+
def do_p(self, arg):
"""p expression
Print the value of the expression.
"""
- try:
- self.message(repr(self._getval(arg)))
- except:
- pass
+ self._msg_val_func(arg, repr)
def do_pp(self, arg):
"""pp expression
Pretty-print the value of the expression.
"""
- try:
- self.message(pprint.pformat(self._getval(arg)))
- except:
- pass
+ self._msg_val_func(arg, pprint.pformat)
complete_print = _complete_expression
complete_p = _complete_expression
@@ -1238,6 +1239,12 @@ def do_list(self, arg):
if last is None:
last = first + 10
filename = self.curframe.f_code.co_filename
+ # gh-93696: stdlib frozen modules provide a useful __file__
+ # this workaround can be removed with the closure of gh-89815
+ if filename.startswith("",
someMoreData = b"" * 10,
- aDate = datetime.datetime.fromtimestamp(time.mktime(time.gmtime())),
+ aDate = datetime.datetime.now()
)
- with open(fileName, 'wb') as fp:
- dump(pl, fp)
+ print(plistlib.dumps(pl).decode())
Parse Plist example:
- with open(fileName, 'rb') as fp:
- pl = load(fp)
- print(pl["aKey"])
+ import plistlib
+
+ plist = b'''
+
+ foo
+ bar
+
+ '''
+ pl = plistlib.loads(plist)
+ print(pl["foo"])
"""
__all__ = [
"InvalidFileException", "FMT_XML", "FMT_BINARY", "load", "dump", "loads", "dumps", "UID"
@@ -61,7 +70,8 @@
from xml.parsers.expat import ParserCreate
-PlistFormat = enum.global_enum(enum.Enum('PlistFormat', 'FMT_XML FMT_BINARY', module=__name__))
+PlistFormat = enum.Enum('PlistFormat', 'FMT_XML FMT_BINARY', module=__name__)
+globals().update(PlistFormat.__members__)
class UID:
@@ -151,7 +161,7 @@ def _date_to_string(d):
def _escape(text):
m = _controlCharPat.search(text)
if m is not None:
- raise ValueError("strings can't contains control characters; "
+ raise ValueError("strings can't contain control characters; "
"use bytes instead")
text = text.replace("\r\n", "\n") # convert DOS line endings
text = text.replace("\r", "\n") # convert Mac line endings
diff --git a/Lib/posixpath.py b/Lib/posixpath.py
index 259baa64b193b8..e550b470da5beb 100644
--- a/Lib/posixpath.py
+++ b/Lib/posixpath.py
@@ -195,6 +195,7 @@ def ismount(path):
if stat.S_ISLNK(s1.st_mode):
return False
+ path = os.fspath(path)
if isinstance(path, bytes):
parent = join(path, b'..')
else:
@@ -352,6 +353,7 @@ def normpath(path):
initial_slashes = path.startswith(sep)
# POSIX allows one or two initial slashes, but treats three or more
# as single slash.
+ # (see https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13)
if (initial_slashes and
path.startswith(sep*2) and not path.startswith(sep*3)):
initial_slashes = 2
diff --git a/Lib/pprint.py b/Lib/pprint.py
index 13819f3fef212a..d91421f0a6bf60 100644
--- a/Lib/pprint.py
+++ b/Lib/pprint.py
@@ -50,7 +50,8 @@ def pprint(object, stream=None, indent=1, width=80, depth=None, *,
"""Pretty-print a Python object to a stream [default is sys.stdout]."""
printer = PrettyPrinter(
stream=stream, indent=indent, width=width, depth=depth,
- compact=compact, sort_dicts=sort_dicts, underscore_numbers=False)
+ compact=compact, sort_dicts=sort_dicts,
+ underscore_numbers=underscore_numbers)
printer.pprint(object)
def pformat(object, indent=1, width=80, depth=None, *,
diff --git a/Lib/profile.py b/Lib/profile.py
index d8599fb4eebd66..90c4e4c9ff583e 100755
--- a/Lib/profile.py
+++ b/Lib/profile.py
@@ -24,6 +24,7 @@
# governing permissions and limitations under the License.
+import io
import sys
import time
import marshal
@@ -587,7 +588,7 @@ def main():
else:
progname = args[0]
sys.path.insert(0, os.path.dirname(progname))
- with open(progname, 'rb') as fp:
+ with io.open_code(progname) as fp:
code = compile(fp.read(), progname, 'exec')
globs = {
'__file__': progname,
diff --git a/Lib/pstats.py b/Lib/pstats.py
index e77459d38b33ec..ac88c04dee25af 100644
--- a/Lib/pstats.py
+++ b/Lib/pstats.py
@@ -26,15 +26,14 @@
import marshal
import re
-from enum import StrEnum, _simple_enum
+from enum import Enum
from functools import cmp_to_key
from dataclasses import dataclass
from typing import Dict
__all__ = ["Stats", "SortKey", "FunctionProfile", "StatsProfile"]
-@_simple_enum(StrEnum)
-class SortKey:
+class SortKey(str, Enum):
CALLS = 'calls', 'ncalls'
CUMULATIVE = 'cumulative', 'cumtime'
FILENAME = 'filename', 'module'
@@ -57,7 +56,7 @@ def __new__(cls, *values):
@dataclass(unsafe_hash=True)
class FunctionProfile:
- ncalls: int
+ ncalls: str
tottime: float
percall_tottime: float
cumtime: float
diff --git a/Lib/pty.py b/Lib/pty.py
index a32432041fa1db..8d8ce40df541c1 100644
--- a/Lib/pty.py
+++ b/Lib/pty.py
@@ -1,7 +1,7 @@
"""Pseudo terminal utilities."""
# Bugs: No signal handling. Doesn't set slave termios and window size.
-# Only tested on Linux.
+# Only tested on Linux, FreeBSD, and macOS.
# See: W. Richard Stevens. 1992. Advanced Programming in the
# UNIX Environment. Chapter 19.
# Author: Steen Lumholt -- with additions by Guido.
@@ -11,7 +11,11 @@
import sys
import tty
-__all__ = ["openpty","fork","spawn"]
+# names imported directly for test mocking purposes
+from os import close, waitpid
+from tty import setraw, tcgetattr, tcsetattr
+
+__all__ = ["openpty", "fork", "spawn"]
STDIN_FILENO = 0
STDOUT_FILENO = 1
@@ -105,8 +109,8 @@ def fork():
os.dup2(slave_fd, STDIN_FILENO)
os.dup2(slave_fd, STDOUT_FILENO)
os.dup2(slave_fd, STDERR_FILENO)
- if (slave_fd > STDERR_FILENO):
- os.close (slave_fd)
+ if slave_fd > STDERR_FILENO:
+ os.close(slave_fd)
# Explicitly open the tty to make it become a controlling tty.
tmp_fd = os.open(os.ttyname(STDOUT_FILENO), os.O_RDWR)
@@ -133,14 +137,22 @@ def _copy(master_fd, master_read=_read, stdin_read=_read):
pty master -> standard output (master_read)
standard input -> pty master (stdin_read)"""
fds = [master_fd, STDIN_FILENO]
- while True:
- rfds, wfds, xfds = select(fds, [], [])
+ while fds:
+ rfds, _wfds, _xfds = select(fds, [], [])
+
if master_fd in rfds:
- data = master_read(master_fd)
+ # Some OSes signal EOF by returning an empty byte string,
+ # some throw OSErrors.
+ try:
+ data = master_read(master_fd)
+ except OSError:
+ data = b""
if not data: # Reached EOF.
- fds.remove(master_fd)
+ return # Assume the child process has exited and is
+ # unreachable, so we clean up.
else:
os.write(STDOUT_FILENO, data)
+
if STDIN_FILENO in rfds:
data = stdin_read(STDIN_FILENO)
if not data:
@@ -153,20 +165,23 @@ def spawn(argv, master_read=_read, stdin_read=_read):
if type(argv) == type(''):
argv = (argv,)
sys.audit('pty.spawn', argv)
+
pid, master_fd = fork()
if pid == CHILD:
os.execlp(argv[0], *argv)
+
try:
- mode = tty.tcgetattr(STDIN_FILENO)
- tty.setraw(STDIN_FILENO)
- restore = 1
+ mode = tcgetattr(STDIN_FILENO)
+ setraw(STDIN_FILENO)
+ restore = True
except tty.error: # This is the same as termios.error
- restore = 0
+ restore = False
+
try:
_copy(master_fd, master_read, stdin_read)
- except OSError:
+ finally:
if restore:
- tty.tcsetattr(STDIN_FILENO, tty.TCSAFLUSH, mode)
+ tcsetattr(STDIN_FILENO, tty.TCSAFLUSH, mode)
- os.close(master_fd)
- return os.waitpid(pid, 0)[1]
+ close(master_fd)
+ return waitpid(pid, 0)[1]
diff --git a/Lib/py_compile.py b/Lib/py_compile.py
index 0f9b59025cee38..388614e51b1847 100644
--- a/Lib/py_compile.py
+++ b/Lib/py_compile.py
@@ -190,7 +190,7 @@ def main():
)
args = parser.parse_args()
if args.filenames == ['-']:
- filenames = sys.stdin.readlines()
+ filenames = [filename.rstrip('\n') for filename in sys.stdin.readlines()]
else:
filenames = args.filenames
for filename in filenames:
diff --git a/Lib/pydoc.py b/Lib/pydoc.py
index 8eecd66a2c2fbf..e00ba4191c4100 100755
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -23,7 +23,7 @@ class or function within a module or module in a package. If the
local machine. Port number 0 can be used to get an arbitrary unused port.
Run "pydoc -b" to start an HTTP server on an arbitrary unused port and
-open a Web browser to interactively browse documentation. Combine with
+open a web browser to interactively browse documentation. Combine with
the -n and -p options to control the hostname and port used.
Run "pydoc -w " to write out the HTML documentation for a module
@@ -69,6 +69,7 @@ class or function within a module or module in a package. If the
import sysconfig
import time
import tokenize
+import types
import urllib.parse
import warnings
from collections import deque
@@ -90,13 +91,16 @@ def pathdirs():
normdirs.append(normdir)
return dirs
+def _isclass(object):
+ return inspect.isclass(object) and not isinstance(object, types.GenericAlias)
+
def _findclass(func):
cls = sys.modules.get(func.__module__)
if cls is None:
return None
for name in func.__qualname__.split('.')[:-1]:
cls = getattr(cls, name)
- if not inspect.isclass(cls):
+ if not _isclass(cls):
return None
return cls
@@ -104,7 +108,7 @@ def _finddoc(obj):
if inspect.ismethod(obj):
name = obj.__func__.__name__
self = obj.__self__
- if (inspect.isclass(self) and
+ if (_isclass(self) and
getattr(getattr(self, name, None), '__func__') is obj.__func__):
# classmethod
cls = self
@@ -118,7 +122,7 @@ def _finddoc(obj):
elif inspect.isbuiltin(obj):
name = obj.__name__
self = obj.__self__
- if (inspect.isclass(self) and
+ if (_isclass(self) and
self.__qualname__ + '.' + name == obj.__qualname__):
# classmethod
cls = self
@@ -205,7 +209,7 @@ def classname(object, modname):
def isdata(object):
"""Check if an object is of a type that probably means it's data."""
- return not (inspect.ismodule(object) or inspect.isclass(object) or
+ return not (inspect.ismodule(object) or _isclass(object) or
inspect.isroutine(object) or inspect.isframe(object) or
inspect.istraceback(object) or inspect.iscode(object))
@@ -470,7 +474,7 @@ def document(self, object, name=None, *args):
# by lacking a __name__ attribute) and an instance.
try:
if inspect.ismodule(object): return self.docmodule(*args)
- if inspect.isclass(object): return self.docclass(*args)
+ if _isclass(object): return self.docclass(*args)
if inspect.isroutine(object): return self.docroutine(*args)
except AttributeError:
pass
@@ -694,7 +698,7 @@ def markup(self, text, escape=None, funcs={}, classes={}, methods={}):
url = 'http://www.rfc-editor.org/rfc/rfc%d.txt' % int(rfc)
results.append('%s' % (url, escape(all)))
elif pep:
- url = 'http://www.python.org/dev/peps/pep-%04d/' % int(pep)
+ url = 'https://www.python.org/dev/peps/pep-%04d/' % int(pep)
results.append('%s' % (url, escape(all)))
elif selfdot:
# Create a link for methods like 'self.method(...)'
@@ -775,7 +779,7 @@ def docmodule(self, object, name=None, mod=None, *ignored):
modules = inspect.getmembers(object, inspect.ismodule)
classes, cdict = [], {}
- for key, value in inspect.getmembers(object, inspect.isclass):
+ for key, value in inspect.getmembers(object, _isclass):
# if __all__ exists, believe it. Otherwise use old heuristic.
if (all is not None or
(inspect.getmodule(value) or object) is object):
@@ -1217,7 +1221,7 @@ def docmodule(self, object, name=None, mod=None):
result = result + self.section('DESCRIPTION', desc)
classes = []
- for key, value in inspect.getmembers(object, inspect.isclass):
+ for key, value in inspect.getmembers(object, _isclass):
# if __all__ exists, believe it. Otherwise use old heuristic.
if (all is not None
or (inspect.getmodule(value) or object) is object):
@@ -1618,13 +1622,14 @@ def pipepager(text, cmd):
def tempfilepager(text, cmd):
"""Page through text by invoking a program on a temporary file."""
import tempfile
- filename = tempfile.mktemp()
- with open(filename, 'w', errors='backslashreplace') as file:
- file.write(text)
- try:
+ with tempfile.TemporaryDirectory() as tempdir:
+ filename = os.path.join(tempdir, 'pydoc.out')
+ with open(filename, 'w', errors='backslashreplace',
+ encoding=os.device_encoding(0) if
+ sys.platform == 'win32' else None
+ ) as file:
+ file.write(text)
os.system(cmd + ' "' + filename + '"')
- finally:
- os.unlink(filename)
def _escape_stdout(text):
# Escape non-encodable characters to avoid encoding errors later
@@ -1698,7 +1703,7 @@ def describe(thing):
return 'member descriptor %s.%s.%s' % (
thing.__objclass__.__module__, thing.__objclass__.__name__,
thing.__name__)
- if inspect.isclass(thing):
+ if _isclass(thing):
return 'class ' + thing.__name__
if inspect.isfunction(thing):
return 'function ' + thing.__name__
@@ -1759,7 +1764,7 @@ def render_doc(thing, title='Python Library Documentation: %s', forceload=0,
desc += ' in module ' + module.__name__
if not (inspect.ismodule(object) or
- inspect.isclass(object) or
+ _isclass(object) or
inspect.isroutine(object) or
inspect.isdatadescriptor(object) or
_getdoc(object)):
@@ -2065,7 +2070,7 @@ def intro(self):
Welcome to Python {0}'s help utility!
If this is your first time using Python, you should definitely check out
-the tutorial on the Internet at https://docs.python.org/{0}/tutorial/.
+the tutorial on the internet at https://docs.python.org/{0}/tutorial/.
Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules. To quit this help utility and
@@ -2279,13 +2284,13 @@ def onerror(modname):
warnings.filterwarnings('ignore') # ignore problems during import
ModuleScanner().run(callback, key, onerror=onerror)
-# --------------------------------------- enhanced Web browser interface
+# --------------------------------------- enhanced web browser interface
def _start_server(urlhandler, hostname, port):
"""Start an HTTP server thread on a specific port.
Start an HTML/text server thread, so HTML or text documents can be
- browsed dynamically and interactively with a Web browser. Example use:
+ browsed dynamically and interactively with a web browser. Example use:
>>> import time
>>> import pydoc
@@ -2671,7 +2676,7 @@ def get_html_page(url):
def browse(port=0, *, open_browser=True, hostname='localhost'):
- """Start the enhanced pydoc Web server and open a Web browser.
+ """Start the enhanced pydoc web server and open a web browser.
Use port '0' to start the server on an arbitrary port.
Set open_browser to False to suppress opening a browser.
@@ -2823,7 +2828,7 @@ class BadUsage(Exception): pass
number 0 can be used to get an arbitrary unused port.
{cmd} -b
- Start an HTTP server on an arbitrary unused port and open a Web browser
+ Start an HTTP server on an arbitrary unused port and open a web browser
to interactively browse documentation. This option can be used in
combination with -n and/or -p.
diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
index 83e1a975ec78b2..491dcc80f5c855 100644
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,5 +1,5 @@
# -*- coding: utf-8 -*-
-# Autogenerated by Sphinx on Mon Apr 5 17:39:41 2021
+# Autogenerated by Sphinx on Tue Apr 8 13:10:45 2025
topics = {'assert': 'The "assert" statement\n'
'**********************\n'
'\n'
@@ -93,11 +93,7 @@
' optionally in parentheses, the object is assigned to that '
'target.\n'
'\n'
- '* Else: The object must be an iterable with the same number of '
- 'items\n'
- ' as there are targets in the target list, and the items are '
- 'assigned,\n'
- ' from left to right, to the corresponding targets.\n'
+ '* Else:\n'
'\n'
' * If the target list contains one target prefixed with an '
'asterisk,\n'
@@ -362,7 +358,7 @@
'yield_expression)]\n'
'\n'
'The difference from normal Assignment statements is that only '
- 'single\n'
+ 'a single\n'
'target is allowed.\n'
'\n'
'For simple names as assignment targets, if in class or module '
@@ -412,12 +408,13 @@
'analysis\n'
' tools and IDEs.\n'
'\n'
- 'Changed in version 3.8: Now annotated assignments allow same\n'
- 'expressions in the right hand side as the regular '
- 'assignments.\n'
- 'Previously, some expressions (like un-parenthesized tuple '
- 'expressions)\n'
- 'caused a syntax error.\n',
+ 'Changed in version 3.8: Now annotated assignments allow the '
+ 'same\n'
+ 'expressions in the right hand side as regular assignments. '
+ 'Previously,\n'
+ 'some expressions (like un-parenthesized tuple expressions) '
+ 'caused a\n'
+ 'syntax error.\n',
'async': 'Coroutines\n'
'**********\n'
'\n'
@@ -551,13 +548,65 @@
'exception.\n'
' That new exception causes the old one to be lost.\n'
'\n'
- '[2] A string literal appearing as the first statement in the '
+ '[2] In pattern matching, a sequence is defined as one of the\n'
+ ' following:\n'
+ '\n'
+ ' * a class that inherits from "collections.abc.Sequence"\n'
+ '\n'
+ ' * a Python class that has been registered as\n'
+ ' "collections.abc.Sequence"\n'
+ '\n'
+ ' * a builtin class that has its (CPython) '
+ '"Py_TPFLAGS_SEQUENCE"\n'
+ ' bit set\n'
+ '\n'
+ ' * a class that inherits from any of the above\n'
+ '\n'
+ ' The following standard library classes are sequences:\n'
+ '\n'
+ ' * "array.array"\n'
+ '\n'
+ ' * "collections.deque"\n'
+ '\n'
+ ' * "list"\n'
+ '\n'
+ ' * "memoryview"\n'
+ '\n'
+ ' * "range"\n'
+ '\n'
+ ' * "tuple"\n'
+ '\n'
+ ' Note:\n'
+ '\n'
+ ' Subject values of type "str", "bytes", and "bytearray" do '
+ 'not\n'
+ ' match sequence patterns.\n'
+ '\n'
+ '[3] In pattern matching, a mapping is defined as one of the '
+ 'following:\n'
+ '\n'
+ ' * a class that inherits from "collections.abc.Mapping"\n'
+ '\n'
+ ' * a Python class that has been registered as\n'
+ ' "collections.abc.Mapping"\n'
+ '\n'
+ ' * a builtin class that has its (CPython) '
+ '"Py_TPFLAGS_MAPPING"\n'
+ ' bit set\n'
+ '\n'
+ ' * a class that inherits from any of the above\n'
+ '\n'
+ ' The standard library classes "dict" and '
+ '"types.MappingProxyType"\n'
+ ' are mappings.\n'
+ '\n'
+ '[4] A string literal appearing as the first statement in the '
'function\n'
' body is transformed into the function’s "__doc__" attribute '
'and\n'
' therefore the function’s *docstring*.\n'
'\n'
- '[3] A string literal appearing as the first statement in the class\n'
+ '[5] A string literal appearing as the first statement in the class\n'
' body is transformed into the namespace’s "__doc__" item and\n'
' therefore the class’s *docstring*.\n',
'atom-identifiers': 'Identifiers (Names)\n'
@@ -884,32 +933,6 @@
'*instance* of the\n'
' owner class.\n'
'\n'
- 'object.__set_name__(self, owner, name)\n'
- '\n'
- ' Called at the time the owning class *owner* is '
- 'created. The\n'
- ' descriptor has been assigned to *name*.\n'
- '\n'
- ' Note:\n'
- '\n'
- ' "__set_name__()" is only called implicitly as part '
- 'of the "type"\n'
- ' constructor, so it will need to be called '
- 'explicitly with the\n'
- ' appropriate parameters when a descriptor is added '
- 'to a class\n'
- ' after initial creation:\n'
- '\n'
- ' class A:\n'
- ' pass\n'
- ' descr = custom_descriptor()\n'
- ' A.attr = descr\n'
- " descr.__set_name__(A, 'attr')\n"
- '\n'
- ' See Creating the class object for more details.\n'
- '\n'
- ' New in version 3.6.\n'
- '\n'
'The attribute "__objclass__" is interpreted by the '
'"inspect" module as\n'
'specifying the class where this object was defined '
@@ -981,16 +1004,16 @@
'"super(B,\n'
' obj).m()" searches "obj.__class__.__mro__" for the '
'base class "A"\n'
- ' immediately preceding "B" and then invokes the '
+ ' immediately following "B" and then invokes the '
'descriptor with the\n'
' call: "A.__dict__[\'m\'].__get__(obj, '
'obj.__class__)".\n'
'\n'
'For instance bindings, the precedence of descriptor '
'invocation depends\n'
- 'on the which descriptor methods are defined. A '
- 'descriptor can define\n'
- 'any combination of "__get__()", "__set__()" and '
+ 'on which descriptor methods are defined. A descriptor '
+ 'can define any\n'
+ 'combination of "__get__()", "__set__()" and '
'"__delete__()". If it\n'
'does not define "__get__()", then accessing the '
'attribute will return\n'
@@ -1012,14 +1035,15 @@
'can be\n'
'overridden by instances.\n'
'\n'
- 'Python methods (including "staticmethod()" and '
- '"classmethod()") are\n'
- 'implemented as non-data descriptors. Accordingly, '
- 'instances can\n'
- 'redefine and override methods. This allows individual '
- 'instances to\n'
- 'acquire behaviors that differ from other instances of '
- 'the same class.\n'
+ 'Python methods (including those decorated with '
+ '"@staticmethod" and\n'
+ '"@classmethod") are implemented as non-data '
+ 'descriptors. Accordingly,\n'
+ 'instances can redefine and override methods. This '
+ 'allows individual\n'
+ 'instances to acquire behaviors that differ from other '
+ 'instances of the\n'
+ 'same class.\n'
'\n'
'The "property()" function is implemented as a data '
'descriptor.\n'
@@ -1032,12 +1056,12 @@
'\n'
'*__slots__* allow us to explicitly declare data members '
'(like\n'
- 'properties) and deny the creation of *__dict__* and '
+ 'properties) and deny the creation of "__dict__" and '
'*__weakref__*\n'
'(unless explicitly declared in *__slots__* or available '
'in a parent.)\n'
'\n'
- 'The space saved over using *__dict__* can be '
+ 'The space saved over using "__dict__" can be '
'significant. Attribute\n'
'lookup speed can be significantly improved as well.\n'
'\n'
@@ -1049,7 +1073,7 @@
'*__slots__*\n'
' reserves space for the declared variables and '
'prevents the\n'
- ' automatic creation of *__dict__* and *__weakref__* '
+ ' automatic creation of "__dict__" and *__weakref__* '
'for each\n'
' instance.\n'
'\n'
@@ -1058,11 +1082,11 @@
'--------------------------\n'
'\n'
'* When inheriting from a class without *__slots__*, the '
- '*__dict__* and\n'
+ '"__dict__" and\n'
' *__weakref__* attribute of the instances will always '
'be accessible.\n'
'\n'
- '* Without a *__dict__* variable, instances cannot be '
+ '* Without a "__dict__" variable, instances cannot be '
'assigned new\n'
' variables not listed in the *__slots__* definition. '
'Attempts to\n'
@@ -1076,28 +1100,28 @@
'\n'
'* Without a *__weakref__* variable for each instance, '
'classes defining\n'
- ' *__slots__* do not support weak references to its '
- 'instances. If weak\n'
- ' reference support is needed, then add '
+ ' *__slots__* do not support "weak references" to its '
+ 'instances. If\n'
+ ' weak reference support is needed, then add '
'"\'__weakref__\'" to the\n'
' sequence of strings in the *__slots__* declaration.\n'
'\n'
'* *__slots__* are implemented at the class level by '
'creating\n'
- ' descriptors (Implementing Descriptors) for each '
- 'variable name. As a\n'
- ' result, class attributes cannot be used to set default '
- 'values for\n'
- ' instance variables defined by *__slots__*; otherwise, '
- 'the class\n'
- ' attribute would overwrite the descriptor assignment.\n'
+ ' descriptors for each variable name. As a result, '
+ 'class attributes\n'
+ ' cannot be used to set default values for instance '
+ 'variables defined\n'
+ ' by *__slots__*; otherwise, the class attribute would '
+ 'overwrite the\n'
+ ' descriptor assignment.\n'
'\n'
'* The action of a *__slots__* declaration is not limited '
'to the class\n'
' where it is defined. *__slots__* declared in parents '
'are available\n'
' in child classes. However, child subclasses will get a '
- '*__dict__*\n'
+ '"__dict__"\n'
' and *__weakref__* unless they also define *__slots__* '
'(which should\n'
' only contain names of any *additional* slots).\n'
@@ -1112,18 +1136,25 @@
'future, a\n'
' check may be added to prevent this.\n'
'\n'
- '* Nonempty *__slots__* does not work for classes derived '
- 'from\n'
- ' “variable-length” built-in types such as "int", '
- '"bytes" and "tuple".\n'
+ '* "TypeError" will be raised if nonempty *__slots__* are '
+ 'defined for a\n'
+ ' class derived from a ""variable-length" built-in type" '
+ 'such as\n'
+ ' "int", "bytes", and "tuple".\n'
'\n'
- '* Any non-string iterable may be assigned to '
- '*__slots__*. Mappings may\n'
- ' also be used; however, in the future, special meaning '
- 'may be\n'
- ' assigned to the values corresponding to each key.\n'
+ '* Any non-string *iterable* may be assigned to '
+ '*__slots__*.\n'
'\n'
- '* *__class__* assignment works only if both classes have '
+ '* If a "dictionary" is used to assign *__slots__*, the '
+ 'dictionary keys\n'
+ ' will be used as the slot names. The values of the '
+ 'dictionary can be\n'
+ ' used to provide per-attribute docstrings that will be '
+ 'recognised by\n'
+ ' "inspect.getdoc()" and displayed in the output of '
+ '"help()".\n'
+ '\n'
+ '* "__class__" assignment works only if both classes have '
'the same\n'
' *__slots__*.\n'
'\n'
@@ -1135,10 +1166,10 @@
'violations\n'
' raise "TypeError".\n'
'\n'
- '* If an iterator is used for *__slots__* then a '
- 'descriptor is created\n'
- ' for each of the iterator’s values. However, the '
- '*__slots__*\n'
+ '* If an *iterator* is used for *__slots__* then a '
+ '*descriptor* is\n'
+ ' created for each of the iterator’s values. However, '
+ 'the *__slots__*\n'
' attribute will be an empty iterator.\n',
'attribute-references': 'Attribute references\n'
'********************\n'
@@ -1261,6 +1292,10 @@
'In the latter case, sequence repetition is performed; a negative\n'
'repetition factor yields an empty sequence.\n'
'\n'
+ 'This operation can be customized using the special "__mul__()" '
+ 'and\n'
+ '"__rmul__()" methods.\n'
+ '\n'
'The "@" (at) operator is intended to be used for matrix\n'
'multiplication. No builtin Python types implement this operator.\n'
'\n'
@@ -1276,6 +1311,10 @@
'result. Division by zero raises the "ZeroDivisionError" '
'exception.\n'
'\n'
+ 'This operation can be customized using the special "__truediv__()" '
+ 'and\n'
+ '"__floordiv__()" methods.\n'
+ '\n'
'The "%" (modulo) operator yields the remainder from the division '
'of\n'
'the first argument by the second. The numeric arguments are '
@@ -1307,6 +1346,10 @@
'string formatting is described in the Python Library Reference,\n'
'section printf-style String Formatting.\n'
'\n'
+ 'The *modulo* operation can be customized using the special '
+ '"__mod__()"\n'
+ 'method.\n'
+ '\n'
'The floor division operator, the modulo operator, and the '
'"divmod()"\n'
'function are not defined for complex numbers. Instead, convert to '
@@ -1321,9 +1364,16 @@
'and then added together. In the latter case, the sequences are\n'
'concatenated.\n'
'\n'
+ 'This operation can be customized using the special "__add__()" '
+ 'and\n'
+ '"__radd__()" methods.\n'
+ '\n'
'The "-" (subtraction) operator yields the difference of its '
'arguments.\n'
- 'The numeric arguments are first converted to a common type.\n',
+ 'The numeric arguments are first converted to a common type.\n'
+ '\n'
+ 'This operation can be customized using the special "__sub__()" '
+ 'method.\n',
'bitwise': 'Binary bitwise operations\n'
'*************************\n'
'\n'
@@ -1336,14 +1386,18 @@
'\n'
'The "&" operator yields the bitwise AND of its arguments, which '
'must\n'
- 'be integers.\n'
+ 'be integers or one of them must be a custom object overriding\n'
+ '"__and__()" or "__rand__()" special methods.\n'
'\n'
'The "^" operator yields the bitwise XOR (exclusive OR) of its\n'
- 'arguments, which must be integers.\n'
+ 'arguments, which must be integers or one of them must be a '
+ 'custom\n'
+ 'object overriding "__xor__()" or "__rxor__()" special methods.\n'
'\n'
'The "|" operator yields the bitwise (inclusive) OR of its '
'arguments,\n'
- 'which must be integers.\n',
+ 'which must be integers or one of them must be a custom object\n'
+ 'overriding "__or__()" or "__ror__()" special methods.\n',
'bltin-code-objects': 'Code Objects\n'
'************\n'
'\n'
@@ -1360,6 +1414,10 @@
'through their "__code__" attribute. See also the '
'"code" module.\n'
'\n'
+ 'Accessing "__code__" raises an auditing event '
+ '"object.__getattr__"\n'
+ 'with arguments "obj" and ""__code__"".\n'
+ '\n'
'A code object can be executed or evaluated by passing '
'it (instead of a\n'
'source string) to the "exec()" or "eval()" built-in '
@@ -1618,10 +1676,26 @@
'If the syntax "**expression" appears in the function call,\n'
'"expression" must evaluate to a *mapping*, the contents of which '
'are\n'
- 'treated as additional keyword arguments. If a keyword is already\n'
- 'present (as an explicit keyword argument, or from another '
- 'unpacking),\n'
- 'a "TypeError" exception is raised.\n'
+ 'treated as additional keyword arguments. If a parameter matching a '
+ 'key\n'
+ 'has already been given a value (by an explicit keyword argument, '
+ 'or\n'
+ 'from another unpacking), a "TypeError" exception is raised.\n'
+ '\n'
+ 'When "**expression" is used, each key in this mapping must be a\n'
+ 'string. Each value from the mapping is assigned to the first '
+ 'formal\n'
+ 'parameter eligible for keyword assignment whose name is equal to '
+ 'the\n'
+ 'key. A key need not be a Python identifier (e.g. ""max-temp °F"" '
+ 'is\n'
+ 'acceptable, although it will not match any formal parameter that '
+ 'could\n'
+ 'be declared). If there is no match to a formal parameter the '
+ 'key-value\n'
+ 'pair is collected by the "**" parameter, if there is one, or if '
+ 'there\n'
+ 'is not, a "TypeError" exception is raised.\n'
'\n'
'Formal parameters using the syntax "*identifier" or "**identifier"\n'
'cannot be used as positional argument slots or as keyword argument\n'
@@ -1704,7 +1778,7 @@
'original global namespace. (Usually, the suite contains mostly\n'
'function definitions.) When the class’s suite finishes execution, '
'its\n'
- 'execution frame is discarded but its local namespace is saved. [3] '
+ 'execution frame is discarded but its local namespace is saved. [5] '
'A\n'
'class object is then created using the inheritance list for the '
'base\n'
@@ -1785,7 +1859,11 @@
' comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="\n'
' | "is" ["not"] | ["not"] "in"\n'
'\n'
- 'Comparisons yield boolean values: "True" or "False".\n'
+ 'Comparisons yield boolean values: "True" or "False". Custom '
+ '*rich\n'
+ 'comparison methods* may return non-boolean values. In this '
+ 'case Python\n'
+ 'will call "bool()" on such value in boolean contexts.\n'
'\n'
'Comparisons can be chained arbitrarily, e.g., "x < y <= z" '
'is\n'
@@ -1965,7 +2043,7 @@
'\n'
'* Mappings (instances of "dict") compare equal if and only if '
'they\n'
- ' have equal *(key, value)* pairs. Equality comparison of the '
+ ' have equal "(key, value)" pairs. Equality comparison of the '
'keys and\n'
' values enforces reflexivity.\n'
'\n'
@@ -2314,39 +2392,10 @@
'finished,\n'
'but if the sequence is empty, they will not have been assigned '
'to at\n'
- 'all by the loop. Hint: the built-in function "range()" returns '
- 'an\n'
- 'iterator of integers suitable to emulate the effect of Pascal’s '
- '"for i\n'
- ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, '
- '2]".\n'
- '\n'
- 'Note:\n'
- '\n'
- ' There is a subtlety when the sequence is being modified by the '
- 'loop\n'
- ' (this can only occur for mutable sequences, e.g. lists). An\n'
- ' internal counter is used to keep track of which item is used '
- 'next,\n'
- ' and this is incremented on each iteration. When this counter '
- 'has\n'
- ' reached the length of the sequence the loop terminates. This '
- 'means\n'
- ' that if the suite deletes the current (or a previous) item '
- 'from the\n'
- ' sequence, the next item will be skipped (since it gets the '
- 'index of\n'
- ' the current item which has already been treated). Likewise, '
- 'if the\n'
- ' suite inserts an item in the sequence before the current item, '
- 'the\n'
- ' current item will be treated again the next time through the '
- 'loop.\n'
- ' This can lead to nasty bugs that can be avoided by making a\n'
- ' temporary copy using a slice of the whole sequence, e.g.,\n'
- '\n'
- ' for x in a[:]:\n'
- ' if x < 0: a.remove(x)\n'
+ 'all by the loop. Hint: the built-in type "range()" represents\n'
+ 'immutable arithmetic sequences of integers. For instance, '
+ 'iterating\n'
+ '"range(3)" successively yields 0, 1, and then 2.\n'
'\n'
'\n'
'The "try" statement\n'
@@ -2381,11 +2430,11 @@
'resulting\n'
'object is “compatible” with the exception. An object is '
'compatible\n'
- 'with an exception if it is the class or a base class of the '
- 'exception\n'
- 'object, or a tuple containing an item that is the class or a '
+ 'with an exception if the object is the class or a *non-virtual '
'base\n'
- 'class of the exception object.\n'
+ 'class* of the exception object, or a tuple containing an item '
+ 'that is\n'
+ 'the class or a non-virtual base class of the exception object.\n'
'\n'
'If no except clause matches the exception, the search for an '
'exception\n'
@@ -2594,7 +2643,7 @@
'the\n'
' target list, it will be treated the same as an error '
'occurring\n'
- ' within the suite would be. See step 6 below.\n'
+ ' within the suite would be. See step 7 below.\n'
'\n'
'6. The suite is executed.\n'
'\n'
@@ -2694,7 +2743,7 @@
' subject_expr ::= star_named_expression "," '
'star_named_expressions?\n'
' | named_expression\n'
- " case_block ::= 'case' patterns [guard] ':' block\n"
+ ' case_block ::= \'case\' patterns [guard] ":" block\n'
'\n'
'Note:\n'
'\n'
@@ -2764,7 +2813,7 @@
'have\n'
' happened.\n'
'\n'
- ' * If the guard evaluates as truthy or missing, the "block" '
+ ' * If the guard evaluates as true or is missing, the "block" '
'inside\n'
' "case_block" is executed.\n'
'\n'
@@ -2825,12 +2874,12 @@
'\n'
'2. If the pattern succeeded, evaluate the "guard".\n'
'\n'
- ' * If the "guard" condition evaluates to “truthy”, the case '
- 'block is\n'
+ ' * If the "guard" condition evaluates as true, the case block '
+ 'is\n'
' selected.\n'
'\n'
- ' * If the "guard" condition evaluates to “falsy”, the case '
- 'block is\n'
+ ' * If the "guard" condition evaluates as false, the case block '
+ 'is\n'
' not selected.\n'
'\n'
' * If the "guard" raises an exception during evaluation, the\n'
@@ -2951,7 +3000,7 @@
'AS\n'
'pattern binds the subject to the name on the right of the as '
'keyword\n'
- 'and succeeds. "capture_pattern" cannot be a a "_".\n'
+ 'and succeeds. "capture_pattern" cannot be a "_".\n'
'\n'
'In simple terms "P as NAME" will match with "P", and on success '
'it\n'
@@ -3001,7 +3050,7 @@
'\n'
'A single underscore "_" is not a capture pattern (this is what '
'"!\'_\'"\n'
- 'expresses). And is instead treated as a "wildcard_pattern".\n'
+ 'expresses). It is instead treated as a "wildcard_pattern".\n'
'\n'
'In a given pattern, a given name can only be bound once. E.g. '
'"case\n'
@@ -3029,7 +3078,10 @@
'\n'
" wildcard_pattern ::= '_'\n"
'\n'
- '"_" is a soft keyword.\n'
+ '"_" is a soft keyword within any pattern, but only within '
+ 'patterns.\n'
+ 'It is an identifier, as usual, even within "match" subject\n'
+ 'expressions, "guard"s, and "case" blocks.\n'
'\n'
'In simple terms, "_" will always succeed.\n'
'\n'
@@ -3073,7 +3125,7 @@
'additional\n'
'syntax. Syntax:\n'
'\n'
- " group_pattern ::= '(' pattern ')'\n"
+ ' group_pattern ::= "(" pattern ")"\n'
'\n'
'In simple terms "(P)" has the same effect as "P".\n'
'\n'
@@ -3120,8 +3172,9 @@
'pattern\n'
'against a subject value:\n'
'\n'
- '1. If the subject value is not an instance of a\n'
- ' "collections.abc.Sequence" the sequence pattern fails.\n'
+ '1. If the subject value is not a sequence [2], the sequence '
+ 'pattern\n'
+ ' fails.\n'
'\n'
'2. If the subject value is an instance of "str", "bytes" or\n'
' "bytearray" the sequence pattern fails.\n'
@@ -3176,7 +3229,7 @@
'the\n'
'following happens:\n'
'\n'
- '* "isinstance(, collections.abc.Sequence)"\n'
+ '* check "" is a sequence\n'
'\n'
'* "len(subject) == "\n'
'\n'
@@ -3210,18 +3263,19 @@
'double star pattern must be the last subpattern in the mapping\n'
'pattern.\n'
'\n'
- 'Duplicate key values in mapping patterns are disallowed. (If all '
- 'key\n'
- 'patterns are literal patterns this is considered a syntax '
- 'error;\n'
- 'otherwise this is a runtime error and will raise "ValueError".)\n'
+ 'Duplicate keys in mapping patterns are disallowed. Duplicate '
+ 'literal\n'
+ 'keys will raise a "SyntaxError". Two keys that otherwise have '
+ 'the same\n'
+ 'value will raise a "ValueError" at runtime.\n'
'\n'
'The following is the logical flow for matching a mapping '
'pattern\n'
'against a subject value:\n'
'\n'
- '1. If the subject value is not an instance of\n'
- ' "collections.abc.Mapping", the mapping pattern fails.\n'
+ '1. If the subject value is not a mapping [3],the mapping '
+ 'pattern\n'
+ ' fails.\n'
'\n'
'2. If every key given in the mapping pattern is present in the '
'subject\n'
@@ -3231,7 +3285,10 @@
'\n'
'3. If duplicate keys are detected in the mapping pattern, the '
'pattern\n'
- ' is considered invalid and "ValueError" is raised.\n'
+ ' is considered invalid. A "SyntaxError" is raised for '
+ 'duplicate\n'
+ ' literal values; or a "ValueError" for named keys of the same '
+ 'value.\n'
'\n'
'Note:\n'
'\n'
@@ -3247,7 +3304,7 @@
'the\n'
'following happens:\n'
'\n'
- '* "isinstance(, collections.abc.Mapping)"\n'
+ '* check "" is a mapping\n'
'\n'
'* "KEY1 in "\n'
'\n'
@@ -3274,9 +3331,9 @@
'\n'
'The same keyword should not be repeated in class patterns.\n'
'\n'
- 'The following is the logical flow for matching a mapping '
- 'pattern\n'
- 'against a subject value:\n'
+ 'The following is the logical flow for matching a class pattern '
+ 'against\n'
+ 'a subject value:\n'
'\n'
'1. If "name_or_attr" is not an instance of the builtin "type" , '
'raise\n'
@@ -3293,7 +3350,9 @@
' For a number of built-in types (specified below), a single\n'
' positional subpattern is accepted which will match the '
'entire\n'
- ' subject; for these types no keyword patterns are accepted.\n'
+ ' subject; for these types keyword patterns also work as for '
+ 'other\n'
+ ' types.\n'
'\n'
' If only keyword patterns are present, they are processed as\n'
' follows, one by one:\n'
@@ -3324,15 +3383,14 @@
'class\n'
' "name_or_attr" before matching:\n'
'\n'
- ' I. The equivalent of "getattr(cls, "__match_args__", ()))" '
- 'is\n'
+ ' I. The equivalent of "getattr(cls, "__match_args__", ())" is\n'
' called.\n'
'\n'
' * If this raises an exception, the exception bubbles up.\n'
'\n'
- ' * If the returned value is not a list or tuple, the '
- 'conversion\n'
- ' fails and "TypeError" is raised.\n'
+ ' * If the returned value is not a tuple, the conversion '
+ 'fails and\n'
+ ' "TypeError" is raised.\n'
'\n'
' * If there are more positional patterns than\n'
' "len(cls.__match_args__)", "TypeError" is raised.\n'
@@ -3387,8 +3445,8 @@
' there is matched against the whole object rather than an '
'attribute.\n'
' For example "int(0|1)" matches the value "0", but not the '
- 'values\n'
- ' "0.0" or "False".\n'
+ 'value\n'
+ ' "0.0".\n'
'\n'
'In simple terms "CLS(P1, attr=P2)" matches only if the '
'following\n'
@@ -3426,7 +3484,6 @@
' decorators ::= decorator+\n'
' decorator ::= "@" assignment_expression '
'NEWLINE\n'
- ' dotted_name ::= identifier ("." identifier)*\n'
' parameter_list ::= defparameter ("," '
'defparameter)* "," "/" ["," [parameter_list_no_posonly]]\n'
' | parameter_list_no_posonly\n'
@@ -3451,7 +3508,7 @@
'\n'
'The function definition does not execute the function body; this '
'gets\n'
- 'executed only when the function is called. [2]\n'
+ 'executed only when the function is called. [4]\n'
'\n'
'A function definition may be wrapped by one or more *decorator*\n'
'expressions. Decorator expressions are evaluated when the '
@@ -3526,7 +3583,7 @@
'Calls.\n'
'A function call always assigns values to all parameters '
'mentioned in\n'
- 'the parameter list, either from position arguments, from '
+ 'the parameter list, either from positional arguments, from '
'keyword\n'
'arguments, or from default values. If the form “"*identifier"” '
'is\n'
@@ -3538,8 +3595,14 @@
'new\n'
'empty mapping of the same type. Parameters after “"*"” or\n'
'“"*identifier"” are keyword-only parameters and may only be '
- 'passed\n'
- 'used keyword arguments.\n'
+ 'passed by\n'
+ 'keyword arguments. Parameters before “"/"” are positional-only\n'
+ 'parameters and may only be passed by positional arguments.\n'
+ '\n'
+ 'Changed in version 3.8: The "/" function parameter syntax may be '
+ 'used\n'
+ 'to indicate positional-only parameters. See **PEP 570** for '
+ 'details.\n'
'\n'
'Parameters may have an *annotation* of the form “": '
'expression"”\n'
@@ -3552,11 +3615,20 @@
'parameter list. These annotations can be any valid Python '
'expression.\n'
'The presence of annotations does not change the semantics of a\n'
- 'function. The annotation values are available as string values '
- 'in a\n'
+ 'function. The annotation values are available as values of a\n'
'dictionary keyed by the parameters’ names in the '
'"__annotations__"\n'
- 'attribute of the function object.\n'
+ 'attribute of the function object. If the "annotations" import '
+ 'from\n'
+ '"__future__" is used, annotations are preserved as strings at '
+ 'runtime\n'
+ 'which enables postponed evaluation. Otherwise, they are '
+ 'evaluated\n'
+ 'when the function definition is executed. In this case '
+ 'annotations\n'
+ 'may be evaluated in a different order than they appear in the '
+ 'source\n'
+ 'code.\n'
'\n'
'It is also possible to create anonymous functions (functions not '
'bound\n'
@@ -3641,7 +3713,7 @@
'function definitions.) When the class’s suite finishes '
'execution, its\n'
'execution frame is discarded but its local namespace is saved. '
- '[3] A\n'
+ '[5] A\n'
'class object is then created using the inheritance list for the '
'base\n'
'classes and the saved local namespace for the attribute '
@@ -3845,13 +3917,65 @@
'exception.\n'
' That new exception causes the old one to be lost.\n'
'\n'
- '[2] A string literal appearing as the first statement in the '
+ '[2] In pattern matching, a sequence is defined as one of the\n'
+ ' following:\n'
+ '\n'
+ ' * a class that inherits from "collections.abc.Sequence"\n'
+ '\n'
+ ' * a Python class that has been registered as\n'
+ ' "collections.abc.Sequence"\n'
+ '\n'
+ ' * a builtin class that has its (CPython) '
+ '"Py_TPFLAGS_SEQUENCE"\n'
+ ' bit set\n'
+ '\n'
+ ' * a class that inherits from any of the above\n'
+ '\n'
+ ' The following standard library classes are sequences:\n'
+ '\n'
+ ' * "array.array"\n'
+ '\n'
+ ' * "collections.deque"\n'
+ '\n'
+ ' * "list"\n'
+ '\n'
+ ' * "memoryview"\n'
+ '\n'
+ ' * "range"\n'
+ '\n'
+ ' * "tuple"\n'
+ '\n'
+ ' Note:\n'
+ '\n'
+ ' Subject values of type "str", "bytes", and "bytearray" do '
+ 'not\n'
+ ' match sequence patterns.\n'
+ '\n'
+ '[3] In pattern matching, a mapping is defined as one of the '
+ 'following:\n'
+ '\n'
+ ' * a class that inherits from "collections.abc.Mapping"\n'
+ '\n'
+ ' * a Python class that has been registered as\n'
+ ' "collections.abc.Mapping"\n'
+ '\n'
+ ' * a builtin class that has its (CPython) '
+ '"Py_TPFLAGS_MAPPING"\n'
+ ' bit set\n'
+ '\n'
+ ' * a class that inherits from any of the above\n'
+ '\n'
+ ' The standard library classes "dict" and '
+ '"types.MappingProxyType"\n'
+ ' are mappings.\n'
+ '\n'
+ '[4] A string literal appearing as the first statement in the '
'function\n'
' body is transformed into the function’s "__doc__" attribute '
'and\n'
' therefore the function’s *docstring*.\n'
'\n'
- '[3] A string literal appearing as the first statement in the '
+ '[5] A string literal appearing as the first statement in the '
'class\n'
' body is transformed into the namespace’s "__doc__" item and\n'
' therefore the class’s *docstring*.\n',
@@ -3983,19 +4107,19 @@
' invoking the superclass’s "__new__()" method using\n'
' "super().__new__(cls[, ...])" with appropriate arguments '
'and then\n'
- ' modifying the newly-created instance as necessary before '
+ ' modifying the newly created instance as necessary before '
'returning\n'
' it.\n'
'\n'
' If "__new__()" is invoked during object construction and '
'it returns\n'
- ' an instance or subclass of *cls*, then the new '
- 'instance’s\n'
- ' "__init__()" method will be invoked like '
- '"__init__(self[, ...])",\n'
- ' where *self* is the new instance and the remaining '
- 'arguments are\n'
- ' the same as were passed to the object constructor.\n'
+ ' an instance of *cls*, then the new instance’s '
+ '"__init__()" method\n'
+ ' will be invoked like "__init__(self[, ...])", where '
+ '*self* is the\n'
+ ' new instance and the remaining arguments are the same as '
+ 'were\n'
+ ' passed to the object constructor.\n'
'\n'
' If "__new__()" does not return an instance of *cls*, '
'then the new\n'
@@ -4287,15 +4411,17 @@
'on members\n'
' of hashed collections including "set", "frozenset", and '
'"dict".\n'
- ' "__hash__()" should return an integer. The only required '
- 'property\n'
- ' is that objects which compare equal have the same hash '
- 'value; it is\n'
- ' advised to mix together the hash values of the '
- 'components of the\n'
- ' object that also play a part in comparison of objects by '
- 'packing\n'
- ' them into a tuple and hashing the tuple. Example:\n'
+ ' The "__hash__()" method should return an integer. The '
+ 'only required\n'
+ ' property is that objects which compare equal have the '
+ 'same hash\n'
+ ' value; it is advised to mix together the hash values of '
+ 'the\n'
+ ' components of the object that also play a part in '
+ 'comparison of\n'
+ ' objects by packing them into a tuple and hashing the '
+ 'tuple.\n'
+ ' Example:\n'
'\n'
' def __hash__(self):\n'
' return hash((self.name, self.nick, self.color))\n'
@@ -4326,7 +4452,7 @@
'objects and\n'
' implements an "__eq__()" method, it should not '
'implement\n'
- ' "__hash__()", since the implementation of hashable '
+ ' "__hash__()", since the implementation of *hashable* '
'collections\n'
' requires that a key’s hash value is immutable (if the '
'object’s hash\n'
@@ -4384,7 +4510,7 @@
'Python.This is\n'
' intended to provide protection against a '
'denial-of-service caused\n'
- ' by carefully-chosen inputs that exploit the worst '
+ ' by carefully chosen inputs that exploit the worst '
'case\n'
' performance of a dict insertion, O(n^2) complexity. '
'See\n'
@@ -4440,6 +4566,18 @@
'the source. The extension interface uses the modules "bdb" and '
'"cmd".\n'
'\n'
+ 'See also:\n'
+ '\n'
+ ' Module "faulthandler"\n'
+ ' Used to dump Python tracebacks explicitly, on a fault, '
+ 'after a\n'
+ ' timeout, or on a user signal.\n'
+ '\n'
+ ' Module "traceback"\n'
+ ' Standard interface to extract, format and print stack '
+ 'traces of\n'
+ ' Python programs.\n'
+ '\n'
'The debugger’s prompt is "(Pdb)". Typical usage to run a program '
'under\n'
'control of the debugger is:\n'
@@ -4465,7 +4603,7 @@
'scripts. For\n'
'example:\n'
'\n'
- ' python3 -m pdb myscript.py\n'
+ ' python -m pdb myscript.py\n'
'\n'
'When invoked as a script, pdb will automatically enter '
'post-mortem\n'
@@ -4485,22 +4623,21 @@
'\n'
'New in version 3.7: "pdb.py" now accepts a "-m" option that '
'execute\n'
- 'modules similar to the way "python3 -m" does. As with a script, '
+ 'modules similar to the way "python -m" does. As with a script, '
'the\n'
'debugger will pause execution just before the first line of the\n'
'module.\n'
'\n'
- 'The typical usage to break into the debugger from a running '
- 'program is\n'
- 'to insert\n'
+ 'The typical usage to break into the debugger is to insert:\n'
'\n'
' import pdb; pdb.set_trace()\n'
'\n'
- 'at the location you want to break into the debugger. You can '
- 'then\n'
- 'step through the code following this statement, and continue '
- 'running\n'
- 'without the debugger using the "continue" command.\n'
+ 'at the location you want to break into the debugger, and then '
+ 'run the\n'
+ 'program. You can then step through the code following this '
+ 'statement,\n'
+ 'and continue running without the debugger using the "continue"\n'
+ 'command.\n'
'\n'
'New in version 3.7: The built-in "breakpoint()", when called '
'with\n'
@@ -4550,8 +4687,8 @@
'object)\n'
' under debugger control. When "runeval()" returns, it returns '
'the\n'
- ' value of the expression. Otherwise this function is similar '
- 'to\n'
+ ' value of the *expression*. Otherwise this function is '
+ 'similar to\n'
' "run()".\n'
'\n'
'pdb.runcall(function, *args, **kwds)\n'
@@ -4699,7 +4836,10 @@
'is\n'
'applied to separating the commands; the input is split at the '
'first\n'
- '";;" pair, even if it is in the middle of a quoted string.\n'
+ '";;" pair, even if it is in the middle of a quoted string. A\n'
+ 'workaround for strings with double semicolons is to use '
+ 'implicit\n'
+ 'string concatenation "\';\'\';\'" or "";"";"".\n'
'\n'
'If a file ".pdbrc" exists in the user’s home directory or in '
'the\n'
@@ -4805,14 +4945,15 @@
'ignore bpnumber [count]\n'
'\n'
' Set the ignore count for the given breakpoint number. If '
- 'count is\n'
- ' omitted, the ignore count is set to 0. A breakpoint becomes '
- 'active\n'
- ' when the ignore count is zero. When non-zero, the count is\n'
- ' decremented each time the breakpoint is reached and the '
- 'breakpoint\n'
- ' is not disabled and any associated condition evaluates to '
- 'true.\n'
+ '*count*\n'
+ ' is omitted, the ignore count is set to 0. A breakpoint '
+ 'becomes\n'
+ ' active when the ignore count is zero. When non-zero, the '
+ '*count*\n'
+ ' is decremented each time the breakpoint is reached and the\n'
+ ' breakpoint is not disabled and any associated condition '
+ 'evaluates\n'
+ ' to true.\n'
'\n'
'condition bpnumber [condition]\n'
'\n'
@@ -4862,7 +5003,7 @@
' breakpoint—which could have its own command list, leading to\n'
' ambiguities about which list to execute.\n'
'\n'
- ' If you use the ‘silent’ command in the command list, the '
+ ' If you use the "silent" command in the command list, the '
'usual\n'
' message about stopping at a breakpoint is not printed. This '
'may be\n'
@@ -4897,11 +5038,10 @@
'number\n'
' greater than the current one is reached.\n'
'\n'
- ' With a line number, continue execution until a line with a '
- 'number\n'
- ' greater or equal to that is reached. In both cases, also '
- 'stop when\n'
- ' the current frame returns.\n'
+ ' With *lineno*, continue execution until a line with a number\n'
+ ' greater or equal to *lineno* is reached. In both cases, also '
+ 'stop\n'
+ ' when the current frame returns.\n'
'\n'
' Changed in version 3.2: Allow giving an explicit line '
'number.\n'
@@ -4965,9 +5105,8 @@
'\n'
'p expression\n'
'\n'
- ' Evaluate the *expression* in the current context and print '
- 'its\n'
- ' value.\n'
+ ' Evaluate *expression* in the current context and print its '
+ 'value.\n'
'\n'
' Note:\n'
'\n'
@@ -4977,26 +5116,26 @@
'\n'
'pp expression\n'
'\n'
- ' Like the "p" command, except the value of the expression is '
+ ' Like the "p" command, except the value of *expression* is '
'pretty-\n'
' printed using the "pprint" module.\n'
'\n'
'whatis expression\n'
'\n'
- ' Print the type of the *expression*.\n'
+ ' Print the type of *expression*.\n'
'\n'
'source expression\n'
'\n'
- ' Try to get source code for the given object and display it.\n'
+ ' Try to get source code of *expression* and display it.\n'
'\n'
' New in version 3.2.\n'
'\n'
'display [expression]\n'
'\n'
- ' Display the value of the expression if it changed, each time\n'
+ ' Display the value of *expression* if it changed, each time\n'
' execution stops in the current frame.\n'
'\n'
- ' Without expression, list all display expressions for the '
+ ' Without *expression*, list all display expressions for the '
'current\n'
' frame.\n'
'\n'
@@ -5004,10 +5143,10 @@
'\n'
'undisplay [expression]\n'
'\n'
- ' Do not display the expression any more in the current frame.\n'
- ' Without expression, clear all display expressions for the '
- 'current\n'
- ' frame.\n'
+ ' Do not display *expression* anymore in the current frame. '
+ 'Without\n'
+ ' *expression*, clear all display expressions for the current '
+ 'frame.\n'
'\n'
' New in version 3.2.\n'
'\n'
@@ -5023,16 +5162,16 @@
'\n'
'alias [name [command]]\n'
'\n'
- ' Create an alias called *name* that executes *command*. The '
- 'command\n'
- ' must *not* be enclosed in quotes. Replaceable parameters can '
- 'be\n'
- ' indicated by "%1", "%2", and so on, while "%*" is replaced by '
- 'all\n'
- ' the parameters. If no command is given, the current alias '
- 'for\n'
- ' *name* is shown. If no arguments are given, all aliases are '
- 'listed.\n'
+ ' Create an alias called *name* that executes *command*. The\n'
+ ' *command* must *not* be enclosed in quotes. Replaceable '
+ 'parameters\n'
+ ' can be indicated by "%1", "%2", and so on, while "%*" is '
+ 'replaced\n'
+ ' by all the parameters. If *command* is omitted, the current '
+ 'alias\n'
+ ' for *name* is shown. If no arguments are given, all aliases '
+ 'are\n'
+ ' listed.\n'
'\n'
' Aliases may be nested and can contain anything that can be '
'legally\n'
@@ -5051,14 +5190,14 @@
' in the ".pdbrc" file):\n'
'\n'
' # Print instance variables (usage "pi classInst")\n'
- ' alias pi for k in %1.__dict__.keys(): '
- 'print("%1.",k,"=",%1.__dict__[k])\n'
+ ' alias pi for k in %1.__dict__.keys(): print(f"%1.{k} = '
+ '{%1.__dict__[k]}")\n'
' # Print instance variables in self\n'
' alias ps pi self\n'
'\n'
'unalias name\n'
'\n'
- ' Delete the specified alias.\n'
+ ' Delete the specified alias *name*.\n'
'\n'
'! statement\n'
'\n'
@@ -5078,12 +5217,13 @@
'run [args ...]\n'
'restart [args ...]\n'
'\n'
- ' Restart the debugged Python program. If an argument is '
- 'supplied,\n'
- ' it is split with "shlex" and the result is used as the new\n'
- ' "sys.argv". History, breakpoints, actions and debugger '
- 'options are\n'
- ' preserved. "restart" is an alias for "run".\n'
+ ' Restart the debugged Python program. If *args* is supplied, '
+ 'it is\n'
+ ' split with "shlex" and the result is used as the new '
+ '"sys.argv".\n'
+ ' History, breakpoints, actions and debugger options are '
+ 'preserved.\n'
+ ' "restart" is an alias for "run".\n'
'\n'
'q(uit)\n'
'\n'
@@ -5092,11 +5232,11 @@
'\n'
'debug code\n'
'\n'
- ' Enter a recursive debugger that steps through the code '
- 'argument\n'
- ' (which is an arbitrary expression or statement to be executed '
- 'in\n'
- ' the current environment).\n'
+ ' Enter a recursive debugger that steps through *code* (which '
+ 'is an\n'
+ ' arbitrary expression or statement to be executed in the '
+ 'current\n'
+ ' environment).\n'
'\n'
'retval\n'
'\n'
@@ -5280,11 +5420,11 @@
'clause is\n'
'selected depending on the class of the instance: it must '
'reference the\n'
- 'class of the instance or a base class thereof. The instance '
- 'can be\n'
- 'received by the handler and can carry additional information '
- 'about the\n'
- 'exceptional condition.\n'
+ 'class of the instance or a *non-virtual base class* thereof. '
+ 'The\n'
+ 'instance can be received by the handler and can carry '
+ 'additional\n'
+ 'information about the exceptional condition.\n'
'\n'
'Note:\n'
'\n'
@@ -5356,20 +5496,32 @@
'binding\n'
'operations.\n'
'\n'
- 'The following constructs bind names: formal parameters to '
- 'functions,\n'
- '"import" statements, class and function definitions (these bind '
- 'the\n'
- 'class or function name in the defining block), and targets that '
- 'are\n'
- 'identifiers if occurring in an assignment, "for" loop header, '
- 'or after\n'
- '"as" in a "with" statement or "except" clause. The "import" '
- 'statement\n'
- 'of the form "from ... import *" binds all names defined in the\n'
- 'imported module, except those beginning with an underscore. '
- 'This form\n'
- 'may only be used at the module level.\n'
+ 'The following constructs bind names:\n'
+ '\n'
+ '* formal parameters to functions,\n'
+ '\n'
+ '* class definitions,\n'
+ '\n'
+ '* function definitions,\n'
+ '\n'
+ '* assignment expressions,\n'
+ '\n'
+ '* targets that are identifiers if occurring in an assignment:\n'
+ '\n'
+ ' * "for" loop header,\n'
+ '\n'
+ ' * after "as" in a "with" statement, "except" clause or in the '
+ 'as-\n'
+ ' pattern in structural pattern matching,\n'
+ '\n'
+ ' * in a capture pattern in structural pattern matching\n'
+ '\n'
+ '* "import" statements.\n'
+ '\n'
+ 'The "import" statement of the form "from ... import *" binds '
+ 'all names\n'
+ 'defined in the imported module, except those beginning with an\n'
+ 'underscore. This form may only be used at the module level.\n'
'\n'
'A target occurring in a "del" statement is also considered '
'bound for\n'
@@ -5439,12 +5591,13 @@
'be\n'
'determined by scanning the entire text of the block for name '
'binding\n'
- 'operations.\n'
+ 'operations. See the FAQ entry on UnboundLocalError for '
+ 'examples.\n'
'\n'
'If the "global" statement occurs within a block, all uses of '
- 'the name\n'
- 'specified in the statement refer to the binding of that name in '
- 'the\n'
+ 'the names\n'
+ 'specified in the statement refer to the bindings of those names '
+ 'in the\n'
'top-level namespace. Names are resolved in the top-level '
'namespace by\n'
'searching the global namespace, i.e. the namespace of the '
@@ -5453,9 +5606,10 @@
'namespace\n'
'of the module "builtins". The global namespace is searched '
'first. If\n'
- 'the name is not found there, the builtins namespace is '
- 'searched. The\n'
- '"global" statement must precede all uses of the name.\n'
+ 'the names are not found there, the builtins namespace is '
+ 'searched.\n'
+ 'The "global" statement must precede all uses of the listed '
+ 'names.\n'
'\n'
'The "global" statement has the same scope as a name binding '
'operation\n'
@@ -5606,11 +5760,11 @@
'clause is\n'
'selected depending on the class of the instance: it must '
'reference the\n'
- 'class of the instance or a base class thereof. The instance '
- 'can be\n'
- 'received by the handler and can carry additional information '
- 'about the\n'
- 'exceptional condition.\n'
+ 'class of the instance or a *non-virtual base class* thereof. '
+ 'The\n'
+ 'instance can be received by the handler and can carry '
+ 'additional\n'
+ 'information about the exceptional condition.\n'
'\n'
'Note:\n'
'\n'
@@ -5746,33 +5900,9 @@
'\n'
'Names in the target list are not deleted when the loop is finished,\n'
'but if the sequence is empty, they will not have been assigned to at\n'
- 'all by the loop. Hint: the built-in function "range()" returns an\n'
- 'iterator of integers suitable to emulate the effect of Pascal’s "for '
- 'i\n'
- ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, 2]".\n'
- '\n'
- 'Note:\n'
- '\n'
- ' There is a subtlety when the sequence is being modified by the '
- 'loop\n'
- ' (this can only occur for mutable sequences, e.g. lists). An\n'
- ' internal counter is used to keep track of which item is used next,\n'
- ' and this is incremented on each iteration. When this counter has\n'
- ' reached the length of the sequence the loop terminates. This '
- 'means\n'
- ' that if the suite deletes the current (or a previous) item from '
- 'the\n'
- ' sequence, the next item will be skipped (since it gets the index '
- 'of\n'
- ' the current item which has already been treated). Likewise, if '
- 'the\n'
- ' suite inserts an item in the sequence before the current item, the\n'
- ' current item will be treated again the next time through the loop.\n'
- ' This can lead to nasty bugs that can be avoided by making a\n'
- ' temporary copy using a slice of the whole sequence, e.g.,\n'
- '\n'
- ' for x in a[:]:\n'
- ' if x < 0: a.remove(x)\n',
+ 'all by the loop. Hint: the built-in type "range()" represents\n'
+ 'immutable arithmetic sequences of integers. For instance, iterating\n'
+ '"range(3)" successively yields 0, 1, and then 2.\n',
'formatstrings': 'Format String Syntax\n'
'********************\n'
'\n'
@@ -6075,19 +6205,19 @@
'complex\n'
'types. For integers, when binary, octal, or hexadecimal '
'output is\n'
- 'used, this option adds the prefix respective "\'0b\'", '
- '"\'0o\'", or "\'0x\'"\n'
- 'to the output value. For float and complex the alternate '
- 'form causes\n'
- 'the result of the conversion to always contain a '
- 'decimal-point\n'
- 'character, even if no digits follow it. Normally, a '
- 'decimal-point\n'
- 'character appears in the result of these conversions only '
- 'if a digit\n'
- 'follows it. In addition, for "\'g\'" and "\'G\'" '
- 'conversions, trailing\n'
- 'zeros are not removed from the result.\n'
+ 'used, this option adds the respective prefix "\'0b\'", '
+ '"\'0o\'", "\'0x\'",\n'
+ 'or "\'0X\'" to the output value. For float and complex the '
+ 'alternate\n'
+ 'form causes the result of the conversion to always contain '
+ 'a decimal-\n'
+ 'point character, even if no digits follow it. Normally, a '
+ 'decimal-\n'
+ 'point character appears in the result of these conversions '
+ 'only if a\n'
+ 'digit follows it. In addition, for "\'g\'" and "\'G\'" '
+ 'conversions,\n'
+ 'trailing zeros are not removed from the result.\n'
'\n'
'The "\',\'" option signals the use of a comma for a '
'thousands separator.\n'
@@ -6132,19 +6262,19 @@
'"\'0\'" no\n'
'longer affects the default alignment for strings.\n'
'\n'
- 'The *precision* is a decimal number indicating how many '
+ 'The *precision* is a decimal integer indicating how many '
'digits should\n'
- 'be displayed after the decimal point for a floating point '
- 'value\n'
- 'formatted with "\'f\'" and "\'F\'", or before and after the '
- 'decimal point\n'
- 'for a floating point value formatted with "\'g\'" or '
- '"\'G\'". For non-\n'
- 'number types the field indicates the maximum field size - '
- 'in other\n'
- 'words, how many characters will be used from the field '
- 'content. The\n'
- '*precision* is not allowed for integer values.\n'
+ 'be displayed after the decimal point for presentation types '
+ '"\'f\'" and\n'
+ '"\'F\'", or before and after the decimal point for '
+ 'presentation types\n'
+ '"\'g\'" or "\'G\'". For string presentation types the '
+ 'field indicates the\n'
+ 'maximum field size - in other words, how many characters '
+ 'will be used\n'
+ 'from the field content. The *precision* is not allowed for '
+ 'integer\n'
+ 'presentation types.\n'
'\n'
'Finally, the *type* determines how the data should be '
'presented.\n'
@@ -6204,8 +6334,12 @@
'+-----------+------------------------------------------------------------+\n'
' | "\'X\'" | Hex format. Outputs the number in base '
'16, using upper- |\n'
- ' | | case letters for the digits above '
- '9. |\n'
+ ' | | case letters for the digits above 9. In '
+ 'case "\'#\'" is |\n'
+ ' | | specified, the prefix "\'0x\'" will be '
+ 'upper-cased to "\'0X\'" |\n'
+ ' | | as '
+ 'well. |\n'
' '
'+-----------+------------------------------------------------------------+\n'
' | "\'n\'" | Number. This is the same as "\'d\'", '
@@ -6562,7 +6696,6 @@
' decorators ::= decorator+\n'
' decorator ::= "@" assignment_expression '
'NEWLINE\n'
- ' dotted_name ::= identifier ("." identifier)*\n'
' parameter_list ::= defparameter ("," '
'defparameter)* "," "/" ["," [parameter_list_no_posonly]]\n'
' | parameter_list_no_posonly\n'
@@ -6587,7 +6720,7 @@
'\n'
'The function definition does not execute the function body; this '
'gets\n'
- 'executed only when the function is called. [2]\n'
+ 'executed only when the function is called. [4]\n'
'\n'
'A function definition may be wrapped by one or more *decorator*\n'
'expressions. Decorator expressions are evaluated when the '
@@ -6662,7 +6795,7 @@
'Calls.\n'
'A function call always assigns values to all parameters '
'mentioned in\n'
- 'the parameter list, either from position arguments, from '
+ 'the parameter list, either from positional arguments, from '
'keyword\n'
'arguments, or from default values. If the form “"*identifier"” '
'is\n'
@@ -6674,8 +6807,14 @@
'new\n'
'empty mapping of the same type. Parameters after “"*"” or\n'
'“"*identifier"” are keyword-only parameters and may only be '
- 'passed\n'
- 'used keyword arguments.\n'
+ 'passed by\n'
+ 'keyword arguments. Parameters before “"/"” are positional-only\n'
+ 'parameters and may only be passed by positional arguments.\n'
+ '\n'
+ 'Changed in version 3.8: The "/" function parameter syntax may be '
+ 'used\n'
+ 'to indicate positional-only parameters. See **PEP 570** for '
+ 'details.\n'
'\n'
'Parameters may have an *annotation* of the form “": '
'expression"”\n'
@@ -6688,11 +6827,20 @@
'parameter list. These annotations can be any valid Python '
'expression.\n'
'The presence of annotations does not change the semantics of a\n'
- 'function. The annotation values are available as string values '
- 'in a\n'
+ 'function. The annotation values are available as values of a\n'
'dictionary keyed by the parameters’ names in the '
'"__annotations__"\n'
- 'attribute of the function object.\n'
+ 'attribute of the function object. If the "annotations" import '
+ 'from\n'
+ '"__future__" is used, annotations are preserved as strings at '
+ 'runtime\n'
+ 'which enables postponed evaluation. Otherwise, they are '
+ 'evaluated\n'
+ 'when the function definition is executed. In this case '
+ 'annotations\n'
+ 'may be evaluated in a different order than they appear in the '
+ 'source\n'
+ 'code.\n'
'\n'
'It is also possible to create anonymous functions (functions not '
'bound\n'
@@ -6793,22 +6941,31 @@
'trailing underscore characters:\n'
'\n'
'"_*"\n'
- ' Not imported by "from module import *". The special '
- 'identifier "_"\n'
- ' is used in the interactive interpreter to store the result '
- 'of the\n'
- ' last evaluation; it is stored in the "builtins" module. '
- 'When not\n'
- ' in interactive mode, "_" has no special meaning and is not '
- 'defined.\n'
- ' See section The import statement.\n'
+ ' Not imported by "from module import *".\n'
+ '\n'
+ '"_"\n'
+ ' In a "case" pattern within a "match" statement, "_" is a '
+ 'soft\n'
+ ' keyword that denotes a wildcard.\n'
+ '\n'
+ ' Separately, the interactive interpreter makes the result of '
+ 'the\n'
+ ' last evaluation available in the variable "_". (It is '
+ 'stored in the\n'
+ ' "builtins" module, alongside built-in functions like '
+ '"print".)\n'
+ '\n'
+ ' Elsewhere, "_" is a regular identifier. It is often used to '
+ 'name\n'
+ ' “special” items, but it is not special to Python itself.\n'
'\n'
' Note:\n'
'\n'
' The name "_" is often used in conjunction with\n'
' internationalization; refer to the documentation for the\n'
' "gettext" module for more information on this '
- 'convention.\n'
+ 'convention.It is\n'
+ ' also commonly used for unused variables.\n'
'\n'
'"__*__"\n'
' System-defined names, informally known as “dunder” names. '
@@ -6963,15 +7120,23 @@
'trailing underscore characters:\n'
'\n'
'"_*"\n'
- ' Not imported by "from module import *". The special '
- 'identifier "_"\n'
- ' is used in the interactive interpreter to store the result '
+ ' Not imported by "from module import *".\n'
+ '\n'
+ '"_"\n'
+ ' In a "case" pattern within a "match" statement, "_" is a '
+ 'soft\n'
+ ' keyword that denotes a wildcard.\n'
+ '\n'
+ ' Separately, the interactive interpreter makes the result '
'of the\n'
- ' last evaluation; it is stored in the "builtins" module. '
- 'When not\n'
- ' in interactive mode, "_" has no special meaning and is not '
- 'defined.\n'
- ' See section The import statement.\n'
+ ' last evaluation available in the variable "_". (It is '
+ 'stored in the\n'
+ ' "builtins" module, alongside built-in functions like '
+ '"print".)\n'
+ '\n'
+ ' Elsewhere, "_" is a regular identifier. It is often used '
+ 'to name\n'
+ ' “special” items, but it is not special to Python itself.\n'
'\n'
' Note:\n'
'\n'
@@ -6979,7 +7144,8 @@
' internationalization; refer to the documentation for '
'the\n'
' "gettext" module for more information on this '
- 'convention.\n'
+ 'convention.It is\n'
+ ' also commonly used for unused variables.\n'
'\n'
'"__*__"\n'
' System-defined names, informally known as “dunder” names. '
@@ -7051,7 +7217,7 @@
' | "from" relative_module "import" "(" '
'identifier ["as" identifier]\n'
' ("," identifier ["as" identifier])* [","] ")"\n'
- ' | "from" module "import" "*"\n'
+ ' | "from" relative_module "import" "*"\n'
' module ::= (identifier ".")* identifier\n'
' relative_module ::= "."* module | "."+\n'
'\n'
@@ -7071,7 +7237,7 @@
'the clauses had been separated out into individual import '
'statements.\n'
'\n'
- 'The details of the first step, finding and loading modules are\n'
+ 'The details of the first step, finding and loading modules, are\n'
'described in greater detail in the section on the import system, '
'which\n'
'also describes the various types of packages and modules that can '
@@ -7130,12 +7296,12 @@
'Examples:\n'
'\n'
' import foo # foo imported and bound locally\n'
- ' import foo.bar.baz # foo.bar.baz imported, foo bound '
- 'locally\n'
- ' import foo.bar.baz as fbb # foo.bar.baz imported and bound as '
- 'fbb\n'
- ' from foo.bar import baz # foo.bar.baz imported and bound as '
- 'baz\n'
+ ' import foo.bar.baz # foo, foo.bar, and foo.bar.baz '
+ 'imported, foo bound locally\n'
+ ' import foo.bar.baz as fbb # foo, foo.bar, and foo.bar.baz '
+ 'imported, foo.bar.baz bound as fbb\n'
+ ' from foo.bar import baz # foo, foo.bar, and foo.bar.baz '
+ 'imported, foo.bar.baz bound as baz\n'
' from foo import attr # foo imported and foo.attr bound as '
'attr\n'
'\n'
@@ -7395,10 +7561,7 @@
'lambda': 'Lambdas\n'
'*******\n'
'\n'
- ' lambda_expr ::= "lambda" [parameter_list] ":" '
- 'expression\n'
- ' lambda_expr_nocond ::= "lambda" [parameter_list] ":" '
- 'expression_nocond\n'
+ ' lambda_expr ::= "lambda" [parameter_list] ":" expression\n'
'\n'
'Lambda expressions (sometimes called lambda forms) are used to '
'create\n'
@@ -7443,20 +7606,32 @@
'*Names* refer to objects. Names are introduced by name binding\n'
'operations.\n'
'\n'
- 'The following constructs bind names: formal parameters to '
- 'functions,\n'
- '"import" statements, class and function definitions (these bind '
- 'the\n'
- 'class or function name in the defining block), and targets that '
- 'are\n'
- 'identifiers if occurring in an assignment, "for" loop header, or '
- 'after\n'
- '"as" in a "with" statement or "except" clause. The "import" '
- 'statement\n'
- 'of the form "from ... import *" binds all names defined in the\n'
- 'imported module, except those beginning with an underscore. This '
- 'form\n'
- 'may only be used at the module level.\n'
+ 'The following constructs bind names:\n'
+ '\n'
+ '* formal parameters to functions,\n'
+ '\n'
+ '* class definitions,\n'
+ '\n'
+ '* function definitions,\n'
+ '\n'
+ '* assignment expressions,\n'
+ '\n'
+ '* targets that are identifiers if occurring in an assignment:\n'
+ '\n'
+ ' * "for" loop header,\n'
+ '\n'
+ ' * after "as" in a "with" statement, "except" clause or in the '
+ 'as-\n'
+ ' pattern in structural pattern matching,\n'
+ '\n'
+ ' * in a capture pattern in structural pattern matching\n'
+ '\n'
+ '* "import" statements.\n'
+ '\n'
+ 'The "import" statement of the form "from ... import *" binds all '
+ 'names\n'
+ 'defined in the imported module, except those beginning with an\n'
+ 'underscore. This form may only be used at the module level.\n'
'\n'
'A target occurring in a "del" statement is also considered bound '
'for\n'
@@ -7523,11 +7698,11 @@
'within a code block. The local variables of a code block can be\n'
'determined by scanning the entire text of the block for name '
'binding\n'
- 'operations.\n'
+ 'operations. See the FAQ entry on UnboundLocalError for examples.\n'
'\n'
'If the "global" statement occurs within a block, all uses of the '
- 'name\n'
- 'specified in the statement refer to the binding of that name in '
+ 'names\n'
+ 'specified in the statement refer to the bindings of those names in '
'the\n'
'top-level namespace. Names are resolved in the top-level '
'namespace by\n'
@@ -7536,9 +7711,9 @@
'namespace\n'
'of the module "builtins". The global namespace is searched '
'first. If\n'
- 'the name is not found there, the builtins namespace is searched. '
- 'The\n'
- '"global" statement must precede all uses of the name.\n'
+ 'the names are not found there, the builtins namespace is '
+ 'searched.\n'
+ 'The "global" statement must precede all uses of the listed names.\n'
'\n'
'The "global" statement has the same scope as a name binding '
'operation\n'
@@ -7873,9 +8048,9 @@
' of the object truncated to an "Integral" (typically an '
'"int").\n'
'\n'
- ' If "__int__()" is not defined then the built-in function '
- '"int()"\n'
- ' falls back to "__trunc__()".\n',
+ ' The built-in function "int()" falls back to '
+ '"__trunc__()" if\n'
+ ' neither "__int__()" nor "__index__()" is defined.\n',
'objects': 'Objects, values and types\n'
'*************************\n'
'\n'
@@ -8009,8 +8184,8 @@
'\n'
'The following table summarizes the operator precedence '
'in Python, from\n'
- 'lowest precedence (least binding) to highest precedence '
- '(most\n'
+ 'highest precedence (most binding) to lowest precedence '
+ '(least\n'
'binding). Operators in the same box have the same '
'precedence. Unless\n'
'the syntax is explicitly given, operators are binary. '
@@ -8029,71 +8204,71 @@
'| Operator | '
'Description |\n'
'|=================================================|=======================================|\n'
- '| ":=" | '
- 'Assignment expression |\n'
- '+-------------------------------------------------+---------------------------------------+\n'
- '| "lambda" | '
- 'Lambda expression |\n'
+ '| "(expressions...)", "[expressions...]", "{key: | '
+ 'Binding or parenthesized expression, |\n'
+ '| value...}", "{expressions...}" | list '
+ 'display, dictionary display, set |\n'
+ '| | '
+ 'display |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "if" – "else" | '
- 'Conditional expression |\n'
+ '| "x[index]", "x[index:index]", | '
+ 'Subscription, slicing, call, |\n'
+ '| "x(arguments...)", "x.attribute" | '
+ 'attribute reference |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "or" | '
- 'Boolean OR |\n'
+ '| "await x" | '
+ 'Await expression |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "and" | '
- 'Boolean AND |\n'
+ '| "**" | '
+ 'Exponentiation [5] |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "not" "x" | '
- 'Boolean NOT |\n'
+ '| "+x", "-x", "~x" | '
+ 'Positive, negative, bitwise NOT |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "in", "not in", "is", "is not", "<", "<=", ">", | '
- 'Comparisons, including membership |\n'
- '| ">=", "!=", "==" | '
- 'tests and identity tests |\n'
+ '| "*", "@", "/", "//", "%" | '
+ 'Multiplication, matrix |\n'
+ '| | '
+ 'multiplication, division, floor |\n'
+ '| | '
+ 'division, remainder [6] |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "|" | '
- 'Bitwise OR |\n'
+ '| "+", "-" | '
+ 'Addition and subtraction |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "^" | '
- 'Bitwise XOR |\n'
+ '| "<<", ">>" | '
+ 'Shifts |\n'
'+-------------------------------------------------+---------------------------------------+\n'
'| "&" | '
'Bitwise AND |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "<<", ">>" | '
- 'Shifts |\n'
+ '| "^" | '
+ 'Bitwise XOR |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "+", "-" | '
- 'Addition and subtraction |\n'
+ '| "|" | '
+ 'Bitwise OR |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "*", "@", "/", "//", "%" | '
- 'Multiplication, matrix |\n'
- '| | '
- 'multiplication, division, floor |\n'
- '| | '
- 'division, remainder [5] |\n'
+ '| "in", "not in", "is", "is not", "<", "<=", ">", | '
+ 'Comparisons, including membership |\n'
+ '| ">=", "!=", "==" | '
+ 'tests and identity tests |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "+x", "-x", "~x" | '
- 'Positive, negative, bitwise NOT |\n'
+ '| "not x" | '
+ 'Boolean NOT |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "**" | '
- 'Exponentiation [6] |\n'
+ '| "and" | '
+ 'Boolean AND |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "await" "x" | '
- 'Await expression |\n'
+ '| "or" | '
+ 'Boolean OR |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "x[index]", "x[index:index]", | '
- 'Subscription, slicing, call, |\n'
- '| "x(arguments...)", "x.attribute" | '
- 'attribute reference |\n'
+ '| "if" – "else" | '
+ 'Conditional expression |\n'
'+-------------------------------------------------+---------------------------------------+\n'
- '| "(expressions...)", "[expressions...]", "{key: | '
- 'Binding or parenthesized expression, |\n'
- '| value...}", "{expressions...}" | list '
- 'display, dictionary display, set |\n'
- '| | '
- 'display |\n'
+ '| "lambda" | '
+ 'Lambda expression |\n'
+ '+-------------------------------------------------+---------------------------------------+\n'
+ '| ":=" | '
+ 'Assignment expression |\n'
'+-------------------------------------------------+---------------------------------------+\n'
'\n'
'-[ Footnotes ]-\n'
@@ -8174,14 +8349,14 @@
'Check their\n'
' documentation for more info.\n'
'\n'
- '[5] The "%" operator is also used for string formatting; '
- 'the same\n'
- ' precedence applies.\n'
- '\n'
- '[6] The power operator "**" binds less tightly than an '
+ '[5] The power operator "**" binds less tightly than an '
'arithmetic or\n'
' bitwise unary operator on its right, that is, '
- '"2**-1" is "0.5".\n',
+ '"2**-1" is "0.5".\n'
+ '\n'
+ '[6] The "%" operator is also used for string formatting; '
+ 'the same\n'
+ ' precedence applies.\n',
'pass': 'The "pass" statement\n'
'********************\n'
'\n'
@@ -8229,18 +8404,21 @@
'"ZeroDivisionError".\n'
'Raising a negative number to a fractional power results in a '
'"complex"\n'
- 'number. (In earlier versions it raised a "ValueError".)\n',
+ 'number. (In earlier versions it raised a "ValueError".)\n'
+ '\n'
+ 'This operation can be customized using the special "__pow__()" '
+ 'method.\n',
'raise': 'The "raise" statement\n'
'*********************\n'
'\n'
' raise_stmt ::= "raise" [expression ["from" expression]]\n'
'\n'
- 'If no expressions are present, "raise" re-raises the last '
- 'exception\n'
- 'that was active in the current scope. If no exception is active '
- 'in\n'
- 'the current scope, a "RuntimeError" exception is raised indicating\n'
- 'that this is an error.\n'
+ 'If no expressions are present, "raise" re-raises the exception that '
+ 'is\n'
+ 'currently being handled, which is also known as the *active\n'
+ 'exception*. If there isn’t currently an active exception, a\n'
+ '"RuntimeError" exception is raised indicating that this is an '
+ 'error.\n'
'\n'
'Otherwise, "raise" evaluates the first expression as the exception\n'
'object. It must be either a subclass or an instance of\n'
@@ -8266,12 +8444,18 @@
'\n'
'The "from" clause is used for exception chaining: if given, the '
'second\n'
- '*expression* must be another exception class or instance, which '
- 'will\n'
- 'then be attached to the raised exception as the "__cause__" '
- 'attribute\n'
- '(which is writable). If the raised exception is not handled, both\n'
- 'exceptions will be printed:\n'
+ '*expression* must be another exception class or instance. If the\n'
+ 'second expression is an exception instance, it will be attached to '
+ 'the\n'
+ 'raised exception as the "__cause__" attribute (which is writable). '
+ 'If\n'
+ 'the expression is an exception class, the class will be '
+ 'instantiated\n'
+ 'and the resulting exception instance will be attached to the '
+ 'raised\n'
+ 'exception as the "__cause__" attribute. If the raised exception is '
+ 'not\n'
+ 'handled, both exceptions will be printed:\n'
'\n'
' >>> try:\n'
' ... print(1 / 0)\n'
@@ -8289,11 +8473,14 @@
' File "", line 4, in \n'
' RuntimeError: Something bad happened\n'
'\n'
- 'A similar mechanism works implicitly if an exception is raised '
- 'inside\n'
- 'an exception handler or a "finally" clause: the previous exception '
- 'is\n'
- 'then attached as the new exception’s "__context__" attribute:\n'
+ 'A similar mechanism works implicitly if a new exception is raised '
+ 'when\n'
+ 'an exception is already being handled. An exception may be '
+ 'handled\n'
+ 'when an "except" or "finally" clause, or a "with" statement, is '
+ 'used.\n'
+ 'The previous exception is then attached as the new exception’s\n'
+ '"__context__" attribute:\n'
'\n'
' >>> try:\n'
' ... print(1 / 0)\n'
@@ -8375,61 +8562,62 @@
'\n'
'The following methods can be defined to implement '
'container objects.\n'
- 'Containers usually are sequences (such as lists or tuples) '
- 'or mappings\n'
- '(like dictionaries), but can represent other containers as '
- 'well. The\n'
- 'first set of methods is used either to emulate a sequence '
- 'or to\n'
- 'emulate a mapping; the difference is that for a sequence, '
- 'the\n'
- 'allowable keys should be the integers *k* for which "0 <= '
- 'k < N" where\n'
- '*N* is the length of the sequence, or slice objects, which '
- 'define a\n'
- 'range of items. It is also recommended that mappings '
- 'provide the\n'
- 'methods "keys()", "values()", "items()", "get()", '
- '"clear()",\n'
- '"setdefault()", "pop()", "popitem()", "copy()", and '
- '"update()"\n'
- 'behaving similar to those for Python’s standard dictionary '
+ 'Containers usually are *sequences* (such as "lists" or '
+ '"tuples") or\n'
+ '*mappings* (like "dictionaries"), but can represent other '
+ 'containers\n'
+ 'as well. The first set of methods is used either to '
+ 'emulate a\n'
+ 'sequence or to emulate a mapping; the difference is that '
+ 'for a\n'
+ 'sequence, the allowable keys should be the integers *k* '
+ 'for which "0\n'
+ '<= k < N" where *N* is the length of the sequence, or '
+ '"slice" objects,\n'
+ 'which define a range of items. It is also recommended '
+ 'that mappings\n'
+ 'provide the methods "keys()", "values()", "items()", '
+ '"get()",\n'
+ '"clear()", "setdefault()", "pop()", "popitem()", "copy()", '
+ 'and\n'
+ '"update()" behaving similar to those for Python’s '
+ 'standard\n'
+ '"dictionary" objects. The "collections.abc" module '
+ 'provides a\n'
+ '"MutableMapping" *abstract base class* to help create '
+ 'those methods\n'
+ 'from a base set of "__getitem__()", "__setitem__()", '
+ '"__delitem__()",\n'
+ 'and "keys()". Mutable sequences should provide methods '
+ '"append()",\n'
+ '"count()", "index()", "extend()", "insert()", "pop()", '
+ '"remove()",\n'
+ '"reverse()" and "sort()", like Python standard "list" '
'objects.\n'
- 'The "collections.abc" module provides a "MutableMapping" '
- 'abstract base\n'
- 'class to help create those methods from a base set of '
- '"__getitem__()",\n'
- '"__setitem__()", "__delitem__()", and "keys()". Mutable '
- 'sequences\n'
- 'should provide methods "append()", "count()", "index()", '
- '"extend()",\n'
- '"insert()", "pop()", "remove()", "reverse()" and "sort()", '
- 'like Python\n'
- 'standard list objects. Finally, sequence types should '
- 'implement\n'
- 'addition (meaning concatenation) and multiplication '
+ 'Finally, sequence types should implement addition '
'(meaning\n'
- 'repetition) by defining the methods "__add__()", '
- '"__radd__()",\n'
- '"__iadd__()", "__mul__()", "__rmul__()" and "__imul__()" '
- 'described\n'
- 'below; they should not define other numerical operators. '
+ 'concatenation) and multiplication (meaning repetition) by '
+ 'defining the\n'
+ 'methods "__add__()", "__radd__()", "__iadd__()", '
+ '"__mul__()",\n'
+ '"__rmul__()" and "__imul__()" described below; they should '
+ 'not define\n'
+ 'other numerical operators. It is recommended that both '
+ 'mappings and\n'
+ 'sequences implement the "__contains__()" method to allow '
+ 'efficient use\n'
+ 'of the "in" operator; for mappings, "in" should search the '
+ 'mapping’s\n'
+ 'keys; for sequences, it should search through the values. '
'It is\n'
- 'recommended that both mappings and sequences implement '
+ 'further recommended that both mappings and sequences '
+ 'implement the\n'
+ '"__iter__()" method to allow efficient iteration through '
'the\n'
- '"__contains__()" method to allow efficient use of the "in" '
- 'operator;\n'
- 'for mappings, "in" should search the mapping’s keys; for '
- 'sequences, it\n'
- 'should search through the values. It is further '
- 'recommended that both\n'
- 'mappings and sequences implement the "__iter__()" method '
- 'to allow\n'
- 'efficient iteration through the container; for mappings, '
- '"__iter__()"\n'
- 'should iterate through the object’s keys; for sequences, '
- 'it should\n'
- 'iterate through the values.\n'
+ 'container; for mappings, "__iter__()" should iterate '
+ 'through the\n'
+ 'object’s keys; for sequences, it should iterate through '
+ 'the values.\n'
'\n'
'object.__len__(self)\n'
'\n'
@@ -8488,22 +8676,24 @@
'object.__getitem__(self, key)\n'
'\n'
' Called to implement evaluation of "self[key]". For '
- 'sequence types,\n'
- ' the accepted keys should be integers and slice '
- 'objects. Note that\n'
- ' the special interpretation of negative indexes (if the '
- 'class wishes\n'
- ' to emulate a sequence type) is up to the '
- '"__getitem__()" method. If\n'
- ' *key* is of an inappropriate type, "TypeError" may be '
- 'raised; if of\n'
- ' a value outside the set of indexes for the sequence '
- '(after any\n'
- ' special interpretation of negative values), '
- '"IndexError" should be\n'
- ' raised. For mapping types, if *key* is missing (not in '
+ '*sequence*\n'
+ ' types, the accepted keys should be integers and slice '
+ 'objects.\n'
+ ' Note that the special interpretation of negative '
+ 'indexes (if the\n'
+ ' class wishes to emulate a *sequence* type) is up to '
'the\n'
- ' container), "KeyError" should be raised.\n'
+ ' "__getitem__()" method. If *key* is of an inappropriate '
+ 'type,\n'
+ ' "TypeError" may be raised; if of a value outside the '
+ 'set of indexes\n'
+ ' for the sequence (after any special interpretation of '
+ 'negative\n'
+ ' values), "IndexError" should be raised. For *mapping* '
+ 'types, if\n'
+ ' *key* is missing (not in the container), "KeyError" '
+ 'should be\n'
+ ' raised.\n'
'\n'
' Note:\n'
'\n'
@@ -8513,6 +8703,15 @@
'of the\n'
' sequence.\n'
'\n'
+ ' Note:\n'
+ '\n'
+ ' When subscripting a *class*, the special class '
+ 'method\n'
+ ' "__class_getitem__()" may be called instead of '
+ '"__getitem__()".\n'
+ ' See __class_getitem__ versus __getitem__ for more '
+ 'details.\n'
+ '\n'
'object.__setitem__(self, key, value)\n'
'\n'
' Called to implement assignment to "self[key]". Same '
@@ -8548,19 +8747,13 @@
'\n'
'object.__iter__(self)\n'
'\n'
- ' This method is called when an iterator is required for '
- 'a container.\n'
- ' This method should return a new iterator object that '
- 'can iterate\n'
- ' over all the objects in the container. For mappings, '
- 'it should\n'
- ' iterate over the keys of the container.\n'
- '\n'
- ' Iterator objects also need to implement this method; '
- 'they are\n'
- ' required to return themselves. For more information on '
- 'iterator\n'
- ' objects, see Iterator Types.\n'
+ ' This method is called when an *iterator* is required '
+ 'for a\n'
+ ' container. This method should return a new iterator '
+ 'object that can\n'
+ ' iterate over all the objects in the container. For '
+ 'mappings, it\n'
+ ' should iterate over the keys of the container.\n'
'\n'
'object.__reversed__(self)\n'
'\n'
@@ -8623,6 +8816,10 @@
'the\n'
'second argument.\n'
'\n'
+ 'This operation can be customized using the special '
+ '"__lshift__()" and\n'
+ '"__rshift__()" methods.\n'
+ '\n'
'A right shift by *n* bits is defined as floor division by '
'"pow(2,n)".\n'
'A left shift by *n* bits is defined as multiplication with '
@@ -8738,31 +8935,7 @@
' still alive. The list is in definition order. Example:\n'
'\n'
' >>> int.__subclasses__()\n'
- " []\n"
- '\n'
- '-[ Footnotes ]-\n'
- '\n'
- '[1] Additional information on these special methods may be '
- 'found in\n'
- ' the Python Reference Manual (Basic customization).\n'
- '\n'
- '[2] As a consequence, the list "[1, 2]" is considered equal '
- 'to "[1.0,\n'
- ' 2.0]", and similarly for tuples.\n'
- '\n'
- '[3] They must have since the parser can’t tell the type of '
- 'the\n'
- ' operands.\n'
- '\n'
- '[4] Cased characters are those with general category '
- 'property being\n'
- ' one of “Lu” (Letter, uppercase), “Ll” (Letter, '
- 'lowercase), or “Lt”\n'
- ' (Letter, titlecase).\n'
- '\n'
- '[5] To format only a tuple you should therefore provide a '
- 'singleton\n'
- ' tuple whose only element is the tuple to be formatted.\n',
+ " []\n",
'specialnames': 'Special method names\n'
'********************\n'
'\n'
@@ -8831,19 +9004,19 @@
' invoking the superclass’s "__new__()" method using\n'
' "super().__new__(cls[, ...])" with appropriate arguments '
'and then\n'
- ' modifying the newly-created instance as necessary before '
+ ' modifying the newly created instance as necessary before '
'returning\n'
' it.\n'
'\n'
' If "__new__()" is invoked during object construction and '
'it returns\n'
- ' an instance or subclass of *cls*, then the new '
- 'instance’s\n'
- ' "__init__()" method will be invoked like "__init__(self[, '
- '...])",\n'
- ' where *self* is the new instance and the remaining '
- 'arguments are\n'
- ' the same as were passed to the object constructor.\n'
+ ' an instance of *cls*, then the new instance’s '
+ '"__init__()" method\n'
+ ' will be invoked like "__init__(self[, ...])", where '
+ '*self* is the\n'
+ ' new instance and the remaining arguments are the same as '
+ 'were\n'
+ ' passed to the object constructor.\n'
'\n'
' If "__new__()" does not return an instance of *cls*, then '
'the new\n'
@@ -9135,15 +9308,17 @@
'on members\n'
' of hashed collections including "set", "frozenset", and '
'"dict".\n'
- ' "__hash__()" should return an integer. The only required '
- 'property\n'
- ' is that objects which compare equal have the same hash '
- 'value; it is\n'
- ' advised to mix together the hash values of the components '
- 'of the\n'
- ' object that also play a part in comparison of objects by '
- 'packing\n'
- ' them into a tuple and hashing the tuple. Example:\n'
+ ' The "__hash__()" method should return an integer. The '
+ 'only required\n'
+ ' property is that objects which compare equal have the '
+ 'same hash\n'
+ ' value; it is advised to mix together the hash values of '
+ 'the\n'
+ ' components of the object that also play a part in '
+ 'comparison of\n'
+ ' objects by packing them into a tuple and hashing the '
+ 'tuple.\n'
+ ' Example:\n'
'\n'
' def __hash__(self):\n'
' return hash((self.name, self.nick, self.color))\n'
@@ -9173,7 +9348,7 @@
' hashable collections. If a class defines mutable objects '
'and\n'
' implements an "__eq__()" method, it should not implement\n'
- ' "__hash__()", since the implementation of hashable '
+ ' "__hash__()", since the implementation of *hashable* '
'collections\n'
' requires that a key’s hash value is immutable (if the '
'object’s hash\n'
@@ -9230,7 +9405,7 @@
'is\n'
' intended to provide protection against a '
'denial-of-service caused\n'
- ' by carefully-chosen inputs that exploit the worst case\n'
+ ' by carefully chosen inputs that exploit the worst case\n'
' performance of a dict insertion, O(n^2) complexity. '
'See\n'
' http://www.ocert.org/advisories/ocert-2011-003.html '
@@ -9511,32 +9686,6 @@
'of the\n'
' owner class.\n'
'\n'
- 'object.__set_name__(self, owner, name)\n'
- '\n'
- ' Called at the time the owning class *owner* is created. '
- 'The\n'
- ' descriptor has been assigned to *name*.\n'
- '\n'
- ' Note:\n'
- '\n'
- ' "__set_name__()" is only called implicitly as part of '
- 'the "type"\n'
- ' constructor, so it will need to be called explicitly '
- 'with the\n'
- ' appropriate parameters when a descriptor is added to a '
- 'class\n'
- ' after initial creation:\n'
- '\n'
- ' class A:\n'
- ' pass\n'
- ' descr = custom_descriptor()\n'
- ' A.attr = descr\n'
- " descr.__set_name__(A, 'attr')\n"
- '\n'
- ' See Creating the class object for more details.\n'
- '\n'
- ' New in version 3.6.\n'
- '\n'
'The attribute "__objclass__" is interpreted by the "inspect" '
'module as\n'
'specifying the class where this object was defined (setting '
@@ -9607,16 +9756,16 @@
'"super(B,\n'
' obj).m()" searches "obj.__class__.__mro__" for the base '
'class "A"\n'
- ' immediately preceding "B" and then invokes the descriptor '
+ ' immediately following "B" and then invokes the descriptor '
'with the\n'
' call: "A.__dict__[\'m\'].__get__(obj, obj.__class__)".\n'
'\n'
'For instance bindings, the precedence of descriptor '
'invocation depends\n'
- 'on the which descriptor methods are defined. A descriptor '
- 'can define\n'
- 'any combination of "__get__()", "__set__()" and '
- '"__delete__()". If it\n'
+ 'on which descriptor methods are defined. A descriptor can '
+ 'define any\n'
+ 'combination of "__get__()", "__set__()" and "__delete__()". '
+ 'If it\n'
'does not define "__get__()", then accessing the attribute '
'will return\n'
'the descriptor object itself unless there is a value in the '
@@ -9637,13 +9786,14 @@
'be\n'
'overridden by instances.\n'
'\n'
- 'Python methods (including "staticmethod()" and '
- '"classmethod()") are\n'
- 'implemented as non-data descriptors. Accordingly, instances '
- 'can\n'
- 'redefine and override methods. This allows individual '
- 'instances to\n'
- 'acquire behaviors that differ from other instances of the '
+ 'Python methods (including those decorated with '
+ '"@staticmethod" and\n'
+ '"@classmethod") are implemented as non-data descriptors. '
+ 'Accordingly,\n'
+ 'instances can redefine and override methods. This allows '
+ 'individual\n'
+ 'instances to acquire behaviors that differ from other '
+ 'instances of the\n'
'same class.\n'
'\n'
'The "property()" function is implemented as a data '
@@ -9657,12 +9807,12 @@
'\n'
'*__slots__* allow us to explicitly declare data members '
'(like\n'
- 'properties) and deny the creation of *__dict__* and '
+ 'properties) and deny the creation of "__dict__" and '
'*__weakref__*\n'
'(unless explicitly declared in *__slots__* or available in a '
'parent.)\n'
'\n'
- 'The space saved over using *__dict__* can be significant. '
+ 'The space saved over using "__dict__" can be significant. '
'Attribute\n'
'lookup speed can be significantly improved as well.\n'
'\n'
@@ -9674,7 +9824,7 @@
'*__slots__*\n'
' reserves space for the declared variables and prevents '
'the\n'
- ' automatic creation of *__dict__* and *__weakref__* for '
+ ' automatic creation of "__dict__" and *__weakref__* for '
'each\n'
' instance.\n'
'\n'
@@ -9683,11 +9833,11 @@
'~~~~~~~~~~~~~~~~~~~~~~~~~~\n'
'\n'
'* When inheriting from a class without *__slots__*, the '
- '*__dict__* and\n'
+ '"__dict__" and\n'
' *__weakref__* attribute of the instances will always be '
'accessible.\n'
'\n'
- '* Without a *__dict__* variable, instances cannot be '
+ '* Without a "__dict__" variable, instances cannot be '
'assigned new\n'
' variables not listed in the *__slots__* definition. '
'Attempts to\n'
@@ -9700,28 +9850,28 @@
'\n'
'* Without a *__weakref__* variable for each instance, '
'classes defining\n'
- ' *__slots__* do not support weak references to its '
- 'instances. If weak\n'
- ' reference support is needed, then add "\'__weakref__\'" to '
- 'the\n'
+ ' *__slots__* do not support "weak references" to its '
+ 'instances. If\n'
+ ' weak reference support is needed, then add '
+ '"\'__weakref__\'" to the\n'
' sequence of strings in the *__slots__* declaration.\n'
'\n'
'* *__slots__* are implemented at the class level by '
'creating\n'
- ' descriptors (Implementing Descriptors) for each variable '
- 'name. As a\n'
- ' result, class attributes cannot be used to set default '
- 'values for\n'
- ' instance variables defined by *__slots__*; otherwise, the '
- 'class\n'
- ' attribute would overwrite the descriptor assignment.\n'
+ ' descriptors for each variable name. As a result, class '
+ 'attributes\n'
+ ' cannot be used to set default values for instance '
+ 'variables defined\n'
+ ' by *__slots__*; otherwise, the class attribute would '
+ 'overwrite the\n'
+ ' descriptor assignment.\n'
'\n'
'* The action of a *__slots__* declaration is not limited to '
'the class\n'
' where it is defined. *__slots__* declared in parents are '
'available\n'
' in child classes. However, child subclasses will get a '
- '*__dict__*\n'
+ '"__dict__"\n'
' and *__weakref__* unless they also define *__slots__* '
'(which should\n'
' only contain names of any *additional* slots).\n'
@@ -9736,18 +9886,24 @@
'future, a\n'
' check may be added to prevent this.\n'
'\n'
- '* Nonempty *__slots__* does not work for classes derived '
- 'from\n'
- ' “variable-length” built-in types such as "int", "bytes" '
- 'and "tuple".\n'
+ '* "TypeError" will be raised if nonempty *__slots__* are '
+ 'defined for a\n'
+ ' class derived from a ""variable-length" built-in type" '
+ 'such as\n'
+ ' "int", "bytes", and "tuple".\n'
'\n'
- '* Any non-string iterable may be assigned to *__slots__*. '
- 'Mappings may\n'
- ' also be used; however, in the future, special meaning may '
- 'be\n'
- ' assigned to the values corresponding to each key.\n'
+ '* Any non-string *iterable* may be assigned to *__slots__*.\n'
+ '\n'
+ '* If a "dictionary" is used to assign *__slots__*, the '
+ 'dictionary keys\n'
+ ' will be used as the slot names. The values of the '
+ 'dictionary can be\n'
+ ' used to provide per-attribute docstrings that will be '
+ 'recognised by\n'
+ ' "inspect.getdoc()" and displayed in the output of '
+ '"help()".\n'
'\n'
- '* *__class__* assignment works only if both classes have the '
+ '* "__class__" assignment works only if both classes have the '
'same\n'
' *__slots__*.\n'
'\n'
@@ -9759,9 +9915,9 @@
'violations\n'
' raise "TypeError".\n'
'\n'
- '* If an iterator is used for *__slots__* then a descriptor '
- 'is created\n'
- ' for each of the iterator’s values. However, the '
+ '* If an *iterator* is used for *__slots__* then a '
+ '*descriptor* is\n'
+ ' created for each of the iterator’s values. However, the '
'*__slots__*\n'
' attribute will be an empty iterator.\n'
'\n'
@@ -9770,15 +9926,15 @@
'==========================\n'
'\n'
'Whenever a class inherits from another class, '
- '*__init_subclass__* is\n'
- 'called on that class. This way, it is possible to write '
- 'classes which\n'
- 'change the behavior of subclasses. This is closely related '
- 'to class\n'
- 'decorators, but where class decorators only affect the '
- 'specific class\n'
- 'they’re applied to, "__init_subclass__" solely applies to '
- 'future\n'
+ '"__init_subclass__()" is\n'
+ 'called on the parent class. This way, it is possible to '
+ 'write classes\n'
+ 'which change the behavior of subclasses. This is closely '
+ 'related to\n'
+ 'class decorators, but where class decorators only affect the '
+ 'specific\n'
+ 'class they’re applied to, "__init_subclass__" solely applies '
+ 'to future\n'
'subclasses of the class defining the method.\n'
'\n'
'classmethod object.__init_subclass__(cls)\n'
@@ -9826,6 +9982,38 @@
'\n'
' New in version 3.6.\n'
'\n'
+ 'When a class is created, "type.__new__()" scans the class '
+ 'variables\n'
+ 'and makes callbacks to those with a "__set_name__()" hook.\n'
+ '\n'
+ 'object.__set_name__(self, owner, name)\n'
+ '\n'
+ ' Automatically called at the time the owning class *owner* '
+ 'is\n'
+ ' created. The object has been assigned to *name* in that '
+ 'class:\n'
+ '\n'
+ ' class A:\n'
+ ' x = C() # Automatically calls: x.__set_name__(A, '
+ "'x')\n"
+ '\n'
+ ' If the class variable is assigned after the class is '
+ 'created,\n'
+ ' "__set_name__()" will not be called automatically. If '
+ 'needed,\n'
+ ' "__set_name__()" can be called directly:\n'
+ '\n'
+ ' class A:\n'
+ ' pass\n'
+ '\n'
+ ' c = C()\n'
+ ' A.x = c # The hook is not called\n'
+ " c.__set_name__(A, 'x') # Manually invoke the hook\n"
+ '\n'
+ ' See Creating the class object for more details.\n'
+ '\n'
+ ' New in version 3.6.\n'
+ '\n'
'\n'
'Metaclasses\n'
'-----------\n'
@@ -9938,10 +10126,10 @@
'come from\n'
'the class definition). The "__prepare__" method should be '
'implemented\n'
- 'as a "classmethod()". The namespace returned by '
- '"__prepare__" is\n'
- 'passed in to "__new__", but when the final class object is '
- 'created the\n'
+ 'as a "classmethod". The namespace returned by "__prepare__" '
+ 'is passed\n'
+ 'in to "__new__", but when the final class object is created '
+ 'the\n'
'namespace is copied into a new "dict".\n'
'\n'
'If the metaclass has no "__prepare__" attribute, then the '
@@ -10021,22 +10209,21 @@
'When using the default metaclass "type", or any metaclass '
'that\n'
'ultimately calls "type.__new__", the following additional\n'
- 'customisation steps are invoked after creating the class '
+ 'customization steps are invoked after creating the class '
'object:\n'
'\n'
- '* first, "type.__new__" collects all of the descriptors in '
- 'the class\n'
- ' namespace that define a "__set_name__()" method;\n'
+ '1. The "type.__new__" method collects all of the attributes '
+ 'in the\n'
+ ' class namespace that define a "__set_name__()" method;\n'
'\n'
- '* second, all of these "__set_name__" methods are called '
- 'with the\n'
- ' class being defined and the assigned name of that '
- 'particular\n'
- ' descriptor;\n'
+ '2. Those "__set_name__" methods are called with the class '
+ 'being\n'
+ ' defined and the assigned name of that particular '
+ 'attribute;\n'
'\n'
- '* finally, the "__init_subclass__()" hook is called on the '
- 'immediate\n'
- ' parent of the new class in its method resolution order.\n'
+ '3. The "__init_subclass__()" hook is called on the immediate '
+ 'parent of\n'
+ ' the new class in its method resolution order.\n'
'\n'
'After the class object is created, it is passed to the '
'class\n'
@@ -10129,9 +10316,33 @@
'Emulating generic types\n'
'=======================\n'
'\n'
- 'One can implement the generic class syntax as specified by '
- '**PEP 484**\n'
- '(for example "List[int]") by defining a special method:\n'
+ 'When using *type annotations*, it is often useful to '
+ '*parameterize* a\n'
+ '*generic type* using Python’s square-brackets notation. For '
+ 'example,\n'
+ 'the annotation "list[int]" might be used to signify a "list" '
+ 'in which\n'
+ 'all the elements are of type "int".\n'
+ '\n'
+ 'See also:\n'
+ '\n'
+ ' **PEP 484** - Type Hints\n'
+ ' Introducing Python’s framework for type annotations\n'
+ '\n'
+ ' Generic Alias Types\n'
+ ' Documentation for objects representing parameterized '
+ 'generic\n'
+ ' classes\n'
+ '\n'
+ ' Generics, user-defined generics and "typing.Generic"\n'
+ ' Documentation on how to implement generic classes that '
+ 'can be\n'
+ ' parameterized at runtime and understood by static '
+ 'type-checkers.\n'
+ '\n'
+ 'A class can *generally* only be parameterized if it defines '
+ 'the\n'
+ 'special class method "__class_getitem__()".\n'
'\n'
'classmethod object.__class_getitem__(cls, key)\n'
'\n'
@@ -10139,18 +10350,144 @@
'generic class\n'
' by type arguments found in *key*.\n'
'\n'
- 'This method is looked up on the class object itself, and '
- 'when defined\n'
- 'in the class body, this method is implicitly a class '
- 'method. Note,\n'
- 'this mechanism is primarily reserved for use with static '
- 'type hints,\n'
- 'other usage is discouraged.\n'
+ ' When defined on a class, "__class_getitem__()" is '
+ 'automatically a\n'
+ ' class method. As such, there is no need for it to be '
+ 'decorated with\n'
+ ' "@classmethod" when it is defined.\n'
+ '\n'
+ '\n'
+ 'The purpose of *__class_getitem__*\n'
+ '----------------------------------\n'
+ '\n'
+ 'The purpose of "__class_getitem__()" is to allow runtime\n'
+ 'parameterization of standard-library generic classes in '
+ 'order to more\n'
+ 'easily apply *type hints* to these classes.\n'
+ '\n'
+ 'To implement custom generic classes that can be '
+ 'parameterized at\n'
+ 'runtime and understood by static type-checkers, users should '
+ 'either\n'
+ 'inherit from a standard library class that already '
+ 'implements\n'
+ '"__class_getitem__()", or inherit from "typing.Generic", '
+ 'which has its\n'
+ 'own implementation of "__class_getitem__()".\n'
+ '\n'
+ 'Custom implementations of "__class_getitem__()" on classes '
+ 'defined\n'
+ 'outside of the standard library may not be understood by '
+ 'third-party\n'
+ 'type-checkers such as mypy. Using "__class_getitem__()" on '
+ 'any class\n'
+ 'for purposes other than type hinting is discouraged.\n'
+ '\n'
+ '\n'
+ '*__class_getitem__* versus *__getitem__*\n'
+ '----------------------------------------\n'
+ '\n'
+ 'Usually, the subscription of an object using square brackets '
+ 'will call\n'
+ 'the "__getitem__()" instance method defined on the object’s '
+ 'class.\n'
+ 'However, if the object being subscribed is itself a class, '
+ 'the class\n'
+ 'method "__class_getitem__()" may be called instead.\n'
+ '"__class_getitem__()" should return a GenericAlias object if '
+ 'it is\n'
+ 'properly defined.\n'
+ '\n'
+ 'Presented with the *expression* "obj[x]", the Python '
+ 'interpreter\n'
+ 'follows something like the following process to decide '
+ 'whether\n'
+ '"__getitem__()" or "__class_getitem__()" should be called:\n'
+ '\n'
+ ' from inspect import isclass\n'
+ '\n'
+ ' def subscribe(obj, x):\n'
+ ' """Return the result of the expression `obj[x]`"""\n'
+ '\n'
+ ' class_of_obj = type(obj)\n'
+ '\n'
+ ' # If the class of obj defines __getitem__,\n'
+ ' # call class_of_obj.__getitem__(obj, x)\n'
+ " if hasattr(class_of_obj, '__getitem__'):\n"
+ ' return class_of_obj.__getitem__(obj, x)\n'
+ '\n'
+ ' # Else, if obj is a class and defines '
+ '__class_getitem__,\n'
+ ' # call obj.__class_getitem__(x)\n'
+ ' elif isclass(obj) and hasattr(obj, '
+ "'__class_getitem__'):\n"
+ ' return obj.__class_getitem__(x)\n'
+ '\n'
+ ' # Else, raise an exception\n'
+ ' else:\n'
+ ' raise TypeError(\n'
+ ' f"\'{class_of_obj.__name__}\' object is not '
+ 'subscriptable"\n'
+ ' )\n'
+ '\n'
+ 'In Python, all classes are themselves instances of other '
+ 'classes. The\n'
+ 'class of a class is known as that class’s *metaclass*, and '
+ 'most\n'
+ 'classes have the "type" class as their metaclass. "type" '
+ 'does not\n'
+ 'define "__getitem__()", meaning that expressions such as '
+ '"list[int]",\n'
+ '"dict[str, float]" and "tuple[str, bytes]" all result in\n'
+ '"__class_getitem__()" being called:\n'
+ '\n'
+ ' >>> # list has class "type" as its metaclass, like most '
+ 'classes:\n'
+ ' >>> type(list)\n'
+ " \n"
+ ' >>> type(dict) == type(list) == type(tuple) == type(str) '
+ '== type(bytes)\n'
+ ' True\n'
+ ' >>> # "list[int]" calls "list.__class_getitem__(int)"\n'
+ ' >>> list[int]\n'
+ ' list[int]\n'
+ ' >>> # list.__class_getitem__ returns a GenericAlias '
+ 'object:\n'
+ ' >>> type(list[int])\n'
+ " \n"
+ '\n'
+ 'However, if a class has a custom metaclass that defines\n'
+ '"__getitem__()", subscribing the class may result in '
+ 'different\n'
+ 'behaviour. An example of this can be found in the "enum" '
+ 'module:\n'
+ '\n'
+ ' >>> from enum import Enum\n'
+ ' >>> class Menu(Enum):\n'
+ ' ... """A breakfast menu"""\n'
+ " ... SPAM = 'spam'\n"
+ " ... BACON = 'bacon'\n"
+ ' ...\n'
+ ' >>> # Enum classes have a custom metaclass:\n'
+ ' >>> type(Menu)\n'
+ " \n"
+ ' >>> # EnumMeta defines __getitem__,\n'
+ ' >>> # so __class_getitem__ is not called,\n'
+ ' >>> # and the result is not a GenericAlias object:\n'
+ " >>> Menu['SPAM']\n"
+ " \n"
+ " >>> type(Menu['SPAM'])\n"
+ " \n"
'\n'
'See also:\n'
'\n'
- ' **PEP 560** - Core support for typing module and generic '
+ ' **PEP 560** - Core Support for typing module and generic '
'types\n'
+ ' Introducing "__class_getitem__()", and outlining when '
+ 'a\n'
+ ' subscription results in "__class_getitem__()" being '
+ 'called\n'
+ ' instead of "__getitem__()"\n'
'\n'
'\n'
'Emulating callable objects\n'
@@ -10169,60 +10506,60 @@
'\n'
'The following methods can be defined to implement container '
'objects.\n'
- 'Containers usually are sequences (such as lists or tuples) '
- 'or mappings\n'
- '(like dictionaries), but can represent other containers as '
- 'well. The\n'
- 'first set of methods is used either to emulate a sequence or '
- 'to\n'
- 'emulate a mapping; the difference is that for a sequence, '
- 'the\n'
- 'allowable keys should be the integers *k* for which "0 <= k '
- '< N" where\n'
- '*N* is the length of the sequence, or slice objects, which '
- 'define a\n'
- 'range of items. It is also recommended that mappings '
- 'provide the\n'
- 'methods "keys()", "values()", "items()", "get()", '
- '"clear()",\n'
- '"setdefault()", "pop()", "popitem()", "copy()", and '
- '"update()"\n'
- 'behaving similar to those for Python’s standard dictionary '
+ 'Containers usually are *sequences* (such as "lists" or '
+ '"tuples") or\n'
+ '*mappings* (like "dictionaries"), but can represent other '
+ 'containers\n'
+ 'as well. The first set of methods is used either to emulate '
+ 'a\n'
+ 'sequence or to emulate a mapping; the difference is that for '
+ 'a\n'
+ 'sequence, the allowable keys should be the integers *k* for '
+ 'which "0\n'
+ '<= k < N" where *N* is the length of the sequence, or '
+ '"slice" objects,\n'
+ 'which define a range of items. It is also recommended that '
+ 'mappings\n'
+ 'provide the methods "keys()", "values()", "items()", '
+ '"get()",\n'
+ '"clear()", "setdefault()", "pop()", "popitem()", "copy()", '
+ 'and\n'
+ '"update()" behaving similar to those for Python’s standard\n'
+ '"dictionary" objects. The "collections.abc" module provides '
+ 'a\n'
+ '"MutableMapping" *abstract base class* to help create those '
+ 'methods\n'
+ 'from a base set of "__getitem__()", "__setitem__()", '
+ '"__delitem__()",\n'
+ 'and "keys()". Mutable sequences should provide methods '
+ '"append()",\n'
+ '"count()", "index()", "extend()", "insert()", "pop()", '
+ '"remove()",\n'
+ '"reverse()" and "sort()", like Python standard "list" '
'objects.\n'
- 'The "collections.abc" module provides a "MutableMapping" '
- 'abstract base\n'
- 'class to help create those methods from a base set of '
- '"__getitem__()",\n'
- '"__setitem__()", "__delitem__()", and "keys()". Mutable '
- 'sequences\n'
- 'should provide methods "append()", "count()", "index()", '
- '"extend()",\n'
- '"insert()", "pop()", "remove()", "reverse()" and "sort()", '
- 'like Python\n'
- 'standard list objects. Finally, sequence types should '
- 'implement\n'
- 'addition (meaning concatenation) and multiplication '
- '(meaning\n'
- 'repetition) by defining the methods "__add__()", '
- '"__radd__()",\n'
- '"__iadd__()", "__mul__()", "__rmul__()" and "__imul__()" '
- 'described\n'
- 'below; they should not define other numerical operators. It '
- 'is\n'
- 'recommended that both mappings and sequences implement the\n'
- '"__contains__()" method to allow efficient use of the "in" '
- 'operator;\n'
- 'for mappings, "in" should search the mapping’s keys; for '
- 'sequences, it\n'
- 'should search through the values. It is further recommended '
- 'that both\n'
- 'mappings and sequences implement the "__iter__()" method to '
- 'allow\n'
- 'efficient iteration through the container; for mappings, '
- '"__iter__()"\n'
- 'should iterate through the object’s keys; for sequences, it '
- 'should\n'
- 'iterate through the values.\n'
+ 'Finally, sequence types should implement addition (meaning\n'
+ 'concatenation) and multiplication (meaning repetition) by '
+ 'defining the\n'
+ 'methods "__add__()", "__radd__()", "__iadd__()", '
+ '"__mul__()",\n'
+ '"__rmul__()" and "__imul__()" described below; they should '
+ 'not define\n'
+ 'other numerical operators. It is recommended that both '
+ 'mappings and\n'
+ 'sequences implement the "__contains__()" method to allow '
+ 'efficient use\n'
+ 'of the "in" operator; for mappings, "in" should search the '
+ 'mapping’s\n'
+ 'keys; for sequences, it should search through the values. '
+ 'It is\n'
+ 'further recommended that both mappings and sequences '
+ 'implement the\n'
+ '"__iter__()" method to allow efficient iteration through '
+ 'the\n'
+ 'container; for mappings, "__iter__()" should iterate through '
+ 'the\n'
+ 'object’s keys; for sequences, it should iterate through the '
+ 'values.\n'
'\n'
'object.__len__(self)\n'
'\n'
@@ -10280,22 +10617,23 @@
'object.__getitem__(self, key)\n'
'\n'
' Called to implement evaluation of "self[key]". For '
- 'sequence types,\n'
- ' the accepted keys should be integers and slice objects. '
- 'Note that\n'
- ' the special interpretation of negative indexes (if the '
- 'class wishes\n'
- ' to emulate a sequence type) is up to the "__getitem__()" '
- 'method. If\n'
- ' *key* is of an inappropriate type, "TypeError" may be '
- 'raised; if of\n'
- ' a value outside the set of indexes for the sequence '
- '(after any\n'
- ' special interpretation of negative values), "IndexError" '
+ '*sequence*\n'
+ ' types, the accepted keys should be integers and slice '
+ 'objects.\n'
+ ' Note that the special interpretation of negative indexes '
+ '(if the\n'
+ ' class wishes to emulate a *sequence* type) is up to the\n'
+ ' "__getitem__()" method. If *key* is of an inappropriate '
+ 'type,\n'
+ ' "TypeError" may be raised; if of a value outside the set '
+ 'of indexes\n'
+ ' for the sequence (after any special interpretation of '
+ 'negative\n'
+ ' values), "IndexError" should be raised. For *mapping* '
+ 'types, if\n'
+ ' *key* is missing (not in the container), "KeyError" '
'should be\n'
- ' raised. For mapping types, if *key* is missing (not in '
- 'the\n'
- ' container), "KeyError" should be raised.\n'
+ ' raised.\n'
'\n'
' Note:\n'
'\n'
@@ -10305,6 +10643,14 @@
'the\n'
' sequence.\n'
'\n'
+ ' Note:\n'
+ '\n'
+ ' When subscripting a *class*, the special class method\n'
+ ' "__class_getitem__()" may be called instead of '
+ '"__getitem__()".\n'
+ ' See __class_getitem__ versus __getitem__ for more '
+ 'details.\n'
+ '\n'
'object.__setitem__(self, key, value)\n'
'\n'
' Called to implement assignment to "self[key]". Same note '
@@ -10340,19 +10686,13 @@
'\n'
'object.__iter__(self)\n'
'\n'
- ' This method is called when an iterator is required for a '
- 'container.\n'
- ' This method should return a new iterator object that can '
- 'iterate\n'
- ' over all the objects in the container. For mappings, it '
- 'should\n'
- ' iterate over the keys of the container.\n'
- '\n'
- ' Iterator objects also need to implement this method; they '
- 'are\n'
- ' required to return themselves. For more information on '
- 'iterator\n'
- ' objects, see Iterator Types.\n'
+ ' This method is called when an *iterator* is required for '
+ 'a\n'
+ ' container. This method should return a new iterator '
+ 'object that can\n'
+ ' iterate over all the objects in the container. For '
+ 'mappings, it\n'
+ ' should iterate over the keys of the container.\n'
'\n'
'object.__reversed__(self)\n'
'\n'
@@ -10595,9 +10935,9 @@
' of the object truncated to an "Integral" (typically an '
'"int").\n'
'\n'
- ' If "__int__()" is not defined then the built-in function '
- '"int()"\n'
- ' falls back to "__trunc__()".\n'
+ ' The built-in function "int()" falls back to "__trunc__()" '
+ 'if\n'
+ ' neither "__int__()" nor "__index__()" is defined.\n'
'\n'
'\n'
'With Statement Context Managers\n'
@@ -10672,22 +11012,22 @@
'y)" is\n'
'typically invalid without special support in "MyClass". To '
'be able to\n'
- 'use that kind of patterns, the class needs to define a\n'
- '*__match_args__* attribute.\n'
+ 'use that kind of pattern, the class needs to define a '
+ '*__match_args__*\n'
+ 'attribute.\n'
'\n'
'object.__match_args__\n'
'\n'
- ' This class variable can be assigned a tuple or list of '
- 'strings.\n'
- ' When this class is used in a class pattern with '
- 'positional\n'
- ' arguments, each positional argument will be converted '
- 'into a\n'
- ' keyword argument, using the corresponding value in '
- '*__match_args__*\n'
- ' as the keyword. The absence of this attribute is '
- 'equivalent to\n'
- ' setting it to "()".\n'
+ ' This class variable can be assigned a tuple of strings. '
+ 'When this\n'
+ ' class is used in a class pattern with positional '
+ 'arguments, each\n'
+ ' positional argument will be converted into a keyword '
+ 'argument,\n'
+ ' using the corresponding value in *__match_args__* as the '
+ 'keyword.\n'
+ ' The absence of this attribute is equivalent to setting it '
+ 'to "()".\n'
'\n'
'For example, if "MyClass.__match_args__" is "("left", '
'"center",\n'
@@ -10879,37 +11219,41 @@
'*start* and\n'
' *end* are interpreted as in slice notation.\n'
'\n'
+ ' If *sub* is empty, returns the number of empty strings '
+ 'between\n'
+ ' characters which is the length of the string plus one.\n'
+ '\n'
"str.encode(encoding='utf-8', errors='strict')\n"
'\n'
- ' Return an encoded version of the string as a bytes '
- 'object. Default\n'
- ' encoding is "\'utf-8\'". *errors* may be given to set a '
- 'different\n'
- ' error handling scheme. The default for *errors* is '
- '"\'strict\'",\n'
- ' meaning that encoding errors raise a "UnicodeError". '
+ ' Return the string encoded to "bytes".\n'
+ '\n'
+ ' *encoding* defaults to "\'utf-8\'"; see Standard '
+ 'Encodings for\n'
+ ' possible values.\n'
+ '\n'
+ ' *errors* controls how encoding errors are handled. If '
+ '"\'strict\'"\n'
+ ' (the default), a "UnicodeError" exception is raised. '
'Other possible\n'
' values are "\'ignore\'", "\'replace\'", '
'"\'xmlcharrefreplace\'",\n'
' "\'backslashreplace\'" and any other name registered '
'via\n'
- ' "codecs.register_error()", see section Error Handlers. '
- 'For a list\n'
- ' of possible encodings, see section Standard Encodings.\n'
+ ' "codecs.register_error()". See Error Handlers for '
+ 'details.\n'
'\n'
- ' By default, the *errors* argument is not checked for '
- 'best\n'
- ' performances, but only used at the first encoding '
- 'error. Enable the\n'
- ' Python Development Mode, or use a debug build to check '
- '*errors*.\n'
+ ' For performance reasons, the value of *errors* is not '
+ 'checked for\n'
+ ' validity unless an encoding error actually occurs, '
+ 'Python\n'
+ ' Development Mode is enabled or a debug build is used.\n'
'\n'
- ' Changed in version 3.1: Support for keyword arguments '
- 'added.\n'
+ ' Changed in version 3.1: Added support for keyword '
+ 'arguments.\n'
'\n'
- ' Changed in version 3.9: The *errors* is now checked in '
- 'development\n'
- ' mode and in debug mode.\n'
+ ' Changed in version 3.9: The value of the *errors* '
+ 'argument is now\n'
+ ' checked in Python Development Mode and in debug mode.\n'
'\n'
'str.endswith(suffix[, start[, end]])\n'
'\n'
@@ -11128,9 +11472,9 @@
' >>> from keyword import iskeyword\n'
'\n'
" >>> 'hello'.isidentifier(), iskeyword('hello')\n"
- ' True, False\n'
+ ' (True, False)\n'
" >>> 'def'.isidentifier(), iskeyword('def')\n"
- ' True, True\n'
+ ' (True, True)\n'
'\n'
'str.islower()\n'
'\n'
@@ -11481,7 +11825,7 @@
" >>> ' 1 2 3 '.split()\n"
" ['1', '2', '3']\n"
'\n'
- 'str.splitlines([keepends])\n'
+ 'str.splitlines(keepends=False)\n'
'\n'
' Return a list of the lines in the string, breaking at '
'line\n'
@@ -11653,9 +11997,13 @@
' >>> "they\'re bill\'s friends from the UK".title()\n'
' "They\'Re Bill\'S Friends From The Uk"\n'
'\n'
- ' A workaround for apostrophes can be constructed using '
- 'regular\n'
- ' expressions:\n'
+ ' The "string.capwords()" function does not have this '
+ 'problem, as it\n'
+ ' splits words on spaces only.\n'
+ '\n'
+ ' Alternatively, a workaround for apostrophes can be '
+ 'constructed\n'
+ ' using regular expressions:\n'
'\n'
' >>> import re\n'
' >>> def titlecase(s):\n'
@@ -11777,12 +12125,15 @@
'single quotes ("\'") or double quotes ("""). They can also be '
'enclosed\n'
'in matching groups of three single or double quotes (these are\n'
- 'generally referred to as *triple-quoted strings*). The '
- 'backslash\n'
- '("\\") character is used to escape characters that otherwise have '
- 'a\n'
- 'special meaning, such as newline, backslash itself, or the quote\n'
+ 'generally referred to as *triple-quoted strings*). The backslash '
+ '("\\")\n'
+ 'character is used to give special meaning to otherwise ordinary\n'
+ 'characters like "n", which means ‘newline’ when escaped ("\\n"). '
+ 'It can\n'
+ 'also be used to escape characters that otherwise have a special\n'
+ 'meaning, such as newline, backslash itself, or the quote '
'character.\n'
+ 'See escape sequences below for examples.\n'
'\n'
'Bytes literals are always prefixed with "\'b\'" or "\'B\'"; they '
'produce\n'
@@ -11839,8 +12190,8 @@
'| Escape Sequence | Meaning | Notes '
'|\n'
'|===================|===================================|=========|\n'
- '| "\\newline" | Backslash and newline ignored '
- '| |\n'
+ '| "\\" | Backslash and newline ignored | '
+ '(1) |\n'
'+-------------------+-----------------------------------+---------+\n'
'| "\\\\" | Backslash ("\\") '
'| |\n'
@@ -11873,10 +12224,10 @@
'| |\n'
'+-------------------+-----------------------------------+---------+\n'
'| "\\ooo" | Character with octal value *ooo* | '
- '(1,3) |\n'
+ '(2,4) |\n'
'+-------------------+-----------------------------------+---------+\n'
'| "\\xhh" | Character with hex value *hh* | '
- '(2,3) |\n'
+ '(3,4) |\n'
'+-------------------+-----------------------------------+---------+\n'
'\n'
'Escape sequences only recognized in string literals are:\n'
@@ -11886,39 +12237,51 @@
'|\n'
'|===================|===================================|=========|\n'
'| "\\N{name}" | Character named *name* in the | '
- '(4) |\n'
+ '(5) |\n'
'| | Unicode database | '
'|\n'
'+-------------------+-----------------------------------+---------+\n'
'| "\\uxxxx" | Character with 16-bit hex value | '
- '(5) |\n'
+ '(6) |\n'
'| | *xxxx* | '
'|\n'
'+-------------------+-----------------------------------+---------+\n'
'| "\\Uxxxxxxxx" | Character with 32-bit hex value | '
- '(6) |\n'
+ '(7) |\n'
'| | *xxxxxxxx* | '
'|\n'
'+-------------------+-----------------------------------+---------+\n'
'\n'
'Notes:\n'
'\n'
- '1. As in Standard C, up to three octal digits are accepted.\n'
+ '1. A backslash can be added at the end of a line to ignore the\n'
+ ' newline:\n'
'\n'
- '2. Unlike in Standard C, exactly two hex digits are required.\n'
+ " >>> 'This string will not include \\\n"
+ " ... backslashes or newline characters.'\n"
+ " 'This string will not include backslashes or newline "
+ "characters.'\n"
+ '\n'
+ ' The same result can be achieved using triple-quoted strings, '
+ 'or\n'
+ ' parentheses and string literal concatenation.\n'
'\n'
- '3. In a bytes literal, hexadecimal and octal escapes denote the '
+ '2. As in Standard C, up to three octal digits are accepted.\n'
+ '\n'
+ '3. Unlike in Standard C, exactly two hex digits are required.\n'
+ '\n'
+ '4. In a bytes literal, hexadecimal and octal escapes denote the '
'byte\n'
' with the given value. In a string literal, these escapes '
'denote a\n'
' Unicode character with the given value.\n'
'\n'
- '4. Changed in version 3.3: Support for name aliases [1] has been\n'
+ '5. Changed in version 3.3: Support for name aliases [1] has been\n'
' added.\n'
'\n'
- '5. Exactly four hex digits are required.\n'
+ '6. Exactly four hex digits are required.\n'
'\n'
- '6. Any Unicode character can be encoded this way. Exactly eight '
+ '7. Any Unicode character can be encoded this way. Exactly eight '
'hex\n'
' digits are required.\n'
'\n'
@@ -11962,67 +12325,86 @@
'subscriptions': 'Subscriptions\n'
'*************\n'
'\n'
- 'Subscription of a sequence (string, tuple or list) or '
- 'mapping\n'
- '(dictionary) object usually selects an item from the '
- 'collection:\n'
+ 'The subscription of an instance of a container class will '
+ 'generally\n'
+ 'select an element from the container. The subscription of a '
+ '*generic\n'
+ 'class* will generally return a GenericAlias object.\n'
'\n'
' subscription ::= primary "[" expression_list "]"\n'
'\n'
+ 'When an object is subscripted, the interpreter will '
+ 'evaluate the\n'
+ 'primary and the expression list.\n'
+ '\n'
'The primary must evaluate to an object that supports '
- 'subscription\n'
- '(lists or dictionaries for example). User-defined objects '
- 'can support\n'
- 'subscription by defining a "__getitem__()" method.\n'
+ 'subscription. An\n'
+ 'object may support subscription through defining one or '
+ 'both of\n'
+ '"__getitem__()" and "__class_getitem__()". When the primary '
+ 'is\n'
+ 'subscripted, the evaluated result of the expression list '
+ 'will be\n'
+ 'passed to one of these methods. For more details on when\n'
+ '"__class_getitem__" is called instead of "__getitem__", '
+ 'see\n'
+ '__class_getitem__ versus __getitem__.\n'
+ '\n'
+ 'If the expression list contains at least one comma, it will '
+ 'evaluate\n'
+ 'to a "tuple" containing the items of the expression list. '
+ 'Otherwise,\n'
+ 'the expression list will evaluate to the value of the '
+ 'list’s sole\n'
+ 'member.\n'
'\n'
'For built-in objects, there are two types of objects that '
'support\n'
- 'subscription:\n'
+ 'subscription via "__getitem__()":\n'
'\n'
- 'If the primary is a mapping, the expression list must '
- 'evaluate to an\n'
- 'object whose value is one of the keys of the mapping, and '
+ '1. Mappings. If the primary is a *mapping*, the expression '
+ 'list must\n'
+ ' evaluate to an object whose value is one of the keys of '
'the\n'
- 'subscription selects the value in the mapping that '
- 'corresponds to that\n'
- 'key. (The expression list is a tuple except if it has '
- 'exactly one\n'
- 'item.)\n'
- '\n'
- 'If the primary is a sequence, the expression list must '
- 'evaluate to an\n'
- 'integer or a slice (as discussed in the following '
- 'section).\n'
+ ' mapping, and the subscription selects the value in the '
+ 'mapping that\n'
+ ' corresponds to that key. An example of a builtin mapping '
+ 'class is\n'
+ ' the "dict" class.\n'
+ '\n'
+ '2. Sequences. If the primary is a *sequence*, the '
+ 'expression list must\n'
+ ' evaluate to an "int" or a "slice" (as discussed in the '
+ 'following\n'
+ ' section). Examples of builtin sequence classes include '
+ 'the "str",\n'
+ ' "list" and "tuple" classes.\n'
'\n'
'The formal syntax makes no special provision for negative '
'indices in\n'
- 'sequences; however, built-in sequences all provide a '
+ '*sequences*. However, built-in sequences all provide a '
'"__getitem__()"\n'
'method that interprets negative indices by adding the '
'length of the\n'
- 'sequence to the index (so that "x[-1]" selects the last '
- 'item of "x").\n'
- 'The resulting value must be a nonnegative integer less than '
- 'the number\n'
- 'of items in the sequence, and the subscription selects the '
- 'item whose\n'
- 'index is that value (counting from zero). Since the support '
- 'for\n'
- 'negative indices and slicing occurs in the object’s '
- '"__getitem__()"\n'
- 'method, subclasses overriding this method will need to '
- 'explicitly add\n'
- 'that support.\n'
- '\n'
- 'A string’s items are characters. A character is not a '
- 'separate data\n'
- 'type but a string of exactly one character.\n'
- '\n'
- 'Subscription of certain *classes* or *types* creates a '
- 'generic alias.\n'
- 'In this case, user-defined classes can support subscription '
- 'by\n'
- 'providing a "__class_getitem__()" classmethod.\n',
+ 'sequence to the index so that, for example, "x[-1]" selects '
+ 'the last\n'
+ 'item of "x". The resulting value must be a nonnegative '
+ 'integer less\n'
+ 'than the number of items in the sequence, and the '
+ 'subscription selects\n'
+ 'the item whose index is that value (counting from zero). '
+ 'Since the\n'
+ 'support for negative indices and slicing occurs in the '
+ 'object’s\n'
+ '"__getitem__()" method, subclasses overriding this method '
+ 'will need to\n'
+ 'explicitly add that support.\n'
+ '\n'
+ 'A "string" is a special kind of sequence whose items are '
+ '*characters*.\n'
+ 'A character is not a separate data type but a string of '
+ 'exactly one\n'
+ 'character.\n',
'truth': 'Truth Value Testing\n'
'*******************\n'
'\n'
@@ -12078,10 +12460,10 @@
'exception. For an except clause with an expression, that expression\n'
'is evaluated, and the clause matches the exception if the resulting\n'
'object is “compatible” with the exception. An object is compatible\n'
- 'with an exception if it is the class or a base class of the '
- 'exception\n'
- 'object, or a tuple containing an item that is the class or a base\n'
- 'class of the exception object.\n'
+ 'with an exception if the object is the class or a *non-virtual base\n'
+ 'class* of the exception object, or a tuple containing an item that '
+ 'is\n'
+ 'the class or a non-virtual base class of the exception object.\n'
'\n'
'If no except clause matches the exception, the search for an '
'exception\n'
@@ -12420,7 +12802,7 @@
' points. All the code points in the range "U+0000 - '
'U+10FFFF"\n'
' can be represented in a string. Python doesn’t have a '
- '*char*\n'
+ '"char"\n'
' type; instead, every code point in the string is '
'represented\n'
' as a string object with length "1". The built-in '
@@ -12680,7 +13062,13 @@
'| |\n'
' | | and "\'return\'" for the '
'return | |\n'
- ' | | annotation, if provided. '
+ ' | | annotation, if provided. For '
+ '| |\n'
+ ' | | more information on working '
+ '| |\n'
+ ' | | with this attribute, see '
+ '| |\n'
+ ' | | Annotations Best Practices. '
'| |\n'
' '
'+---------------------------+---------------------------------+-------------+\n'
@@ -12801,20 +13189,18 @@
' A function or method which uses the "yield" statement (see\n'
' section The yield statement) is called a *generator '
'function*.\n'
- ' Such a function, when called, always returns an iterator '
- 'object\n'
- ' which can be used to execute the body of the function: '
- 'calling\n'
- ' the iterator’s "iterator.__next__()" method will cause the\n'
- ' function to execute until it provides a value using the '
- '"yield"\n'
- ' statement. When the function executes a "return" statement '
- 'or\n'
- ' falls off the end, a "StopIteration" exception is raised and '
- 'the\n'
- ' iterator will have reached the end of the set of values to '
- 'be\n'
- ' returned.\n'
+ ' Such a function, when called, always returns an *iterator*\n'
+ ' object which can be used to execute the body of the '
+ 'function:\n'
+ ' calling the iterator’s "iterator.__next__()" method will '
+ 'cause\n'
+ ' the function to execute until it provides a value using the\n'
+ ' "yield" statement. When the function executes a "return"\n'
+ ' statement or falls off the end, a "StopIteration" exception '
+ 'is\n'
+ ' raised and the iterator will have reached the end of the set '
+ 'of\n'
+ ' values to be returned.\n'
'\n'
' Coroutine functions\n'
' A function or method which is defined using "async def" is\n'
@@ -12830,18 +13216,18 @@
' which uses the "yield" statement is called a *asynchronous\n'
' generator function*. Such a function, when called, returns '
'an\n'
- ' asynchronous iterator object which can be used in an "async '
- 'for"\n'
- ' statement to execute the body of the function.\n'
+ ' *asynchronous iterator* object which can be used in an '
+ '"async\n'
+ ' for" statement to execute the body of the function.\n'
'\n'
- ' Calling the asynchronous iterator’s "aiterator.__anext__()"\n'
- ' method will return an *awaitable* which when awaited will\n'
- ' execute until it provides a value using the "yield" '
- 'expression.\n'
- ' When the function executes an empty "return" statement or '
- 'falls\n'
- ' off the end, a "StopAsyncIteration" exception is raised and '
+ ' Calling the asynchronous iterator’s "aiterator.__anext__" '
+ 'method\n'
+ ' will return an *awaitable* which when awaited will execute '
+ 'until\n'
+ ' it provides a value using the "yield" expression. When the\n'
+ ' function executes an empty "return" statement or falls off '
'the\n'
+ ' end, a "StopAsyncIteration" exception is raised and the\n'
' asynchronous iterator will have reached the end of the set '
'of\n'
' values to be yielded.\n'
@@ -12905,20 +13291,34 @@
' Attribute assignment updates the module’s namespace dictionary,\n'
' e.g., "m.x = 1" is equivalent to "m.__dict__["x"] = 1".\n'
'\n'
- ' Predefined (writable) attributes: "__name__" is the module’s '
- 'name;\n'
- ' "__doc__" is the module’s documentation string, or "None" if\n'
- ' unavailable; "__annotations__" (optional) is a dictionary\n'
- ' containing *variable annotations* collected during module body\n'
- ' execution; "__file__" is the pathname of the file from which '
+ ' Predefined (writable) attributes:\n'
+ '\n'
+ ' "__name__"\n'
+ ' The module’s name.\n'
+ '\n'
+ ' "__doc__"\n'
+ ' The module’s documentation string, or "None" if '
+ 'unavailable.\n'
+ '\n'
+ ' "__file__"\n'
+ ' The pathname of the file from which the module was loaded, '
+ 'if\n'
+ ' it was loaded from a file. The "__file__" attribute may '
+ 'be\n'
+ ' missing for certain types of modules, such as C modules '
+ 'that\n'
+ ' are statically linked into the interpreter. For '
+ 'extension\n'
+ ' modules loaded dynamically from a shared library, it’s '
'the\n'
- ' module was loaded, if it was loaded from a file. The "__file__"\n'
- ' attribute may be missing for certain types of modules, such as '
- 'C\n'
- ' modules that are statically linked into the interpreter; for\n'
- ' extension modules loaded dynamically from a shared library, it '
- 'is\n'
- ' the pathname of the shared library file.\n'
+ ' pathname of the shared library file.\n'
+ '\n'
+ ' "__annotations__"\n'
+ ' A dictionary containing *variable annotations* collected\n'
+ ' during module body execution. For best practices on '
+ 'working\n'
+ ' with "__annotations__", please see Annotations Best\n'
+ ' Practices.\n'
'\n'
' Special read-only attribute: "__dict__" is the module’s '
'namespace\n'
@@ -12976,20 +13376,31 @@
'instance\n'
' (see below).\n'
'\n'
- ' Special attributes: "__name__" is the class name; "__module__" '
- 'is\n'
- ' the module name in which the class was defined; "__dict__" is '
- 'the\n'
- ' dictionary containing the class’s namespace; "__bases__" is a '
- 'tuple\n'
- ' containing the base classes, in the order of their occurrence '
- 'in\n'
- ' the base class list; "__doc__" is the class’s documentation '
- 'string,\n'
- ' or "None" if undefined; "__annotations__" (optional) is a\n'
- ' dictionary containing *variable annotations* collected during '
- 'class\n'
- ' body execution.\n'
+ ' Special attributes:\n'
+ '\n'
+ ' "__name__"\n'
+ ' The class name.\n'
+ '\n'
+ ' "__module__"\n'
+ ' The name of the module in which the class was defined.\n'
+ '\n'
+ ' "__dict__"\n'
+ ' The dictionary containing the class’s namespace.\n'
+ '\n'
+ ' "__bases__"\n'
+ ' A tuple containing the base classes, in the order of '
+ 'their\n'
+ ' occurrence in the base class list.\n'
+ '\n'
+ ' "__doc__"\n'
+ ' The class’s documentation string, or "None" if undefined.\n'
+ '\n'
+ ' "__annotations__"\n'
+ ' A dictionary containing *variable annotations* collected\n'
+ ' during class body execution. For best practices on '
+ 'working\n'
+ ' with "__annotations__", please see Annotations Best\n'
+ ' Practices.\n'
'\n'
'Class instances\n'
' A class instance is created by calling a class object (see '
@@ -13150,6 +13561,10 @@
' gives the precise instruction (this is an index into the\n'
' bytecode string of the code object).\n'
'\n'
+ ' Accessing "f_code" raises an auditing event '
+ '"object.__getattr__"\n'
+ ' with arguments "obj" and ""f_code"".\n'
+ '\n'
' Special writable attributes: "f_trace", if not "None", is a\n'
' function called for various events during code execution '
'(this\n'
@@ -13233,6 +13648,9 @@
' the exception occurred in a "try" statement with no matching\n'
' except clause or with a finally clause.\n'
'\n'
+ ' Accessing "tb_frame" raises an auditing event\n'
+ ' "object.__getattr__" with arguments "obj" and ""tb_frame"".\n'
+ '\n'
' Special writable attribute: "tb_next" is the next level in '
'the\n'
' stack trace (towards the frame where the exception occurred), '
@@ -13283,9 +13701,8 @@
' object actually returned is the wrapped object, which is not\n'
' subject to any further transformation. Static method objects '
'are\n'
- ' not themselves callable, although the objects they wrap '
- 'usually\n'
- ' are. Static method objects are created by the built-in\n'
+ ' also callable. Static method objects are created by the '
+ 'built-in\n'
' "staticmethod()" constructor.\n'
'\n'
' Class method objects\n'
@@ -13334,29 +13751,15 @@
'dictionaries or\n'
'other mutable types (that are compared by value rather than '
'by object\n'
- 'identity) may not be used as keys. Numeric types used for '
- 'keys obey\n'
- 'the normal rules for numeric comparison: if two numbers '
- 'compare equal\n'
- '(such as "1" and "1.0") then they can be used '
- 'interchangeably to index\n'
- 'the same dictionary entry. (Note however, that since '
- 'computers store\n'
- 'floating-point numbers as approximations it is usually '
- 'unwise to use\n'
- 'them as dictionary keys.)\n'
- '\n'
- 'Dictionaries can be created by placing a comma-separated '
- 'list of "key:\n'
- 'value" pairs within braces, for example: "{\'jack\': 4098, '
- "'sjoerd':\n"
- '4127}" or "{4098: \'jack\', 4127: \'sjoerd\'}", or by the '
- '"dict"\n'
- 'constructor.\n'
- '\n'
- 'class dict(**kwarg)\n'
- 'class dict(mapping, **kwarg)\n'
- 'class dict(iterable, **kwarg)\n'
+ 'identity) may not be used as keys. Values that compare equal '
+ '(such as\n'
+ '"1", "1.0", and "True") can be used interchangeably to index '
+ 'the same\n'
+ 'dictionary entry.\n'
+ '\n'
+ 'class dict(**kwargs)\n'
+ 'class dict(mapping, **kwargs)\n'
+ 'class dict(iterable, **kwargs)\n'
'\n'
' Return a new dictionary initialized from an optional '
'positional\n'
@@ -13759,7 +14162,7 @@
' New in version 3.10.\n'
'\n'
'Keys views are set-like since their entries are unique and '
- 'hashable.\n'
+ '*hashable*.\n'
'If all values are hashable, so that "(key, value)" pairs are '
'unique\n'
'and hashable, then the items view is also set-like. (Values '
@@ -13808,8 +14211,7 @@
' >>> # get back a read-only proxy for the original '
'dictionary\n'
' >>> values.mapping\n'
- " mappingproxy({'eggs': 2, 'sausage': 1, 'bacon': 1, "
- "'spam': 500})\n"
+ " mappingproxy({'bacon': 1, 'spam': 500})\n"
" >>> values.mapping['spam']\n"
' 500\n',
'typesmethods': 'Methods\n'
@@ -14006,6 +14408,14 @@
'Comparisons in\n'
'the language reference.)\n'
'\n'
+ 'Forward and reversed iterators over mutable sequences access '
+ 'values\n'
+ 'using an index. That index will continue to march forward (or\n'
+ 'backward) even if the underlying sequence is mutated. The '
+ 'iterator\n'
+ 'terminates only when an "IndexError" or a "StopIteration" is\n'
+ 'encountered (or when the index drops below zero).\n'
+ '\n'
'Notes:\n'
'\n'
'1. While the "in" and "not in" operations are used only for '
@@ -14237,7 +14647,7 @@
'| | "s[i:i] = '
'[x]") | |\n'
'+--------------------------------+----------------------------------+-----------------------+\n'
- '| "s.pop([i])" | retrieves the item at *i* '
+ '| "s.pop()" or "s.pop(i)" | retrieves the item at *i* '
'and | (2) |\n'
'| | also removes it from '
'*s* | |\n'
@@ -14477,7 +14887,8 @@
'\n'
' The arguments to the range constructor must be integers '
'(either\n'
- ' built-in "int" or any object that implements the "__index__"\n'
+ ' built-in "int" or any object that implements the '
+ '"__index__()"\n'
' special method). If the *step* argument is omitted, it '
'defaults to\n'
' "1". If the *start* argument is omitted, it defaults to "0". '
@@ -14700,7 +15111,7 @@
'| | "s[i:i] = '
'[x]") | |\n'
'+--------------------------------+----------------------------------+-----------------------+\n'
- '| "s.pop([i])" | retrieves the item at '
+ '| "s.pop()" or "s.pop(i)" | retrieves the item at '
'*i* and | (2) |\n'
'| | also removes it from '
'*s* | |\n'
@@ -14765,15 +15176,21 @@
' u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr\n'
'\n'
'The unary "-" (minus) operator yields the negation of its numeric\n'
- 'argument.\n'
+ 'argument; the operation can be overridden with the "__neg__()" '
+ 'special\n'
+ 'method.\n'
'\n'
'The unary "+" (plus) operator yields its numeric argument '
- 'unchanged.\n'
+ 'unchanged;\n'
+ 'the operation can be overridden with the "__pos__()" special '
+ 'method.\n'
'\n'
'The unary "~" (invert) operator yields the bitwise inversion of '
'its\n'
'integer argument. The bitwise inversion of "x" is defined as\n'
- '"-(x+1)". It only applies to integral numbers.\n'
+ '"-(x+1)". It only applies to integral numbers or to custom '
+ 'objects\n'
+ 'that override the "__invert__()" special method.\n'
'\n'
'In all three cases, if the argument does not have the proper type, '
'a\n'
@@ -14840,7 +15257,7 @@
' returns without an error, then "__exit__()" will always be\n'
' called. Thus, if an error occurs during the assignment to the\n'
' target list, it will be treated the same as an error occurring\n'
- ' within the suite would be. See step 6 below.\n'
+ ' within the suite would be. See step 7 below.\n'
'\n'
'6. The suite is executed.\n'
'\n'
diff --git a/Lib/queue.py b/Lib/queue.py
index 10dbcbc18ece85..55f50088460f9e 100644
--- a/Lib/queue.py
+++ b/Lib/queue.py
@@ -298,7 +298,7 @@ def get(self, block=True, timeout=None):
def put_nowait(self, item):
'''Put an item into the queue without blocking.
- This is exactly equivalent to `put(item)` and is only provided
+ This is exactly equivalent to `put(item, block=False)` and is only provided
for compatibility with the Queue class.
'''
return self.put(item, block=False)
diff --git a/Lib/random.py b/Lib/random.py
index 3a835aef0bc1d4..1310a2d9d0e071 100644
--- a/Lib/random.py
+++ b/Lib/random.py
@@ -154,8 +154,7 @@ def seed(self, a=None, version=2):
elif version == 2 and isinstance(a, (str, bytes, bytearray)):
if isinstance(a, str):
a = a.encode()
- a += _sha512(a).digest()
- a = int.from_bytes(a, 'big')
+ a = int.from_bytes(a + _sha512(a).digest(), 'big')
elif not isinstance(a, (type(None), int, float, str, bytes, bytearray)):
_warn('Seeding based on hashing is deprecated\n'
diff --git a/Lib/re.py b/Lib/re.py
index ea41217ce08c2d..1d82b5006396cd 100644
--- a/Lib/re.py
+++ b/Lib/re.py
@@ -142,9 +142,7 @@
__version__ = "2.2.1"
-@enum.global_enum
-@enum._simple_enum(enum.IntFlag, boundary=enum.KEEP)
-class RegexFlag:
+class RegexFlag(enum.IntFlag):
ASCII = A = sre_compile.SRE_FLAG_ASCII # assume ascii "locale"
IGNORECASE = I = sre_compile.SRE_FLAG_IGNORECASE # ignore case
LOCALE = L = sre_compile.SRE_FLAG_LOCALE # assume current 8-bit locale
@@ -156,6 +154,30 @@ class RegexFlag:
TEMPLATE = T = sre_compile.SRE_FLAG_TEMPLATE # disable backtracking
DEBUG = sre_compile.SRE_FLAG_DEBUG # dump pattern after compilation
+ def __repr__(self):
+ if self._name_ is not None:
+ return f're.{self._name_}'
+ value = self._value_
+ members = []
+ negative = value < 0
+ if negative:
+ value = ~value
+ for m in self.__class__:
+ if value & m._value_:
+ value &= ~m._value_
+ members.append(f're.{m._name_}')
+ if value:
+ members.append(hex(value))
+ res = '|'.join(members)
+ if negative:
+ if len(members) > 1:
+ res = f'~({res})'
+ else:
+ res = f'~{res}'
+ return res
+ __str__ = object.__str__
+globals().update(RegexFlag.__members__)
+
# sre exception
error = sre_compile.error
diff --git a/Lib/rlcompleter.py b/Lib/rlcompleter.py
index c06388e8d9c2dd..98b7930b32fab3 100644
--- a/Lib/rlcompleter.py
+++ b/Lib/rlcompleter.py
@@ -176,13 +176,20 @@ def attr_matches(self, text):
if (word[:n] == attr and
not (noprefix and word[:n+1] == noprefix)):
match = "%s.%s" % (expr, word)
- try:
- val = getattr(thisobject, word)
- except Exception:
- pass # Include even if attribute not set
+ if isinstance(getattr(type(thisobject), word, None),
+ property):
+ # bpo-44752: thisobject.word is a method decorated by
+ # `@property`. What follows applies a postfix if
+ # thisobject.word is callable, but know we know that
+ # this is not callable (because it is a property).
+ # Also, getattr(thisobject, word) will evaluate the
+ # property method, which is not desirable.
+ matches.append(match)
+ continue
+ if (value := getattr(thisobject, word, None)) is not None:
+ matches.append(self._callable_postfix(value, match))
else:
- match = self._callable_postfix(val, match)
- matches.append(match)
+ matches.append(match)
if matches or not noprefix:
break
if noprefix == '_':
diff --git a/Lib/runpy.py b/Lib/runpy.py
index caba1214262380..c7d3d8caad1611 100644
--- a/Lib/runpy.py
+++ b/Lib/runpy.py
@@ -198,9 +198,24 @@ def _run_module_as_main(mod_name, alter_argv=True):
def run_module(mod_name, init_globals=None,
run_name=None, alter_sys=False):
- """Execute a module's code without importing it
+ """Execute a module's code without importing it.
- Returns the resulting top level namespace dictionary
+ mod_name -- an absolute module name or package name.
+
+ Optional arguments:
+ init_globals -- dictionary used to pre-populate the module’s
+ globals dictionary before the code is executed.
+
+ run_name -- if not None, this will be used for setting __name__;
+ otherwise, __name__ will be set to mod_name + '__main__' if the
+ named module is a package and to just mod_name otherwise.
+
+ alter_sys -- if True, sys.argv[0] is updated with the value of
+ __file__ and sys.modules[__name__] is updated with a temporary
+ module object for the module being executed. Both are
+ restored to their original values before the function returns.
+
+ Returns the resulting module globals dictionary.
"""
mod_name, mod_spec, code = _get_module_details(mod_name)
if run_name is None:
@@ -243,14 +258,19 @@ def _get_code_from_file(run_name, fname):
return code, fname
def run_path(path_name, init_globals=None, run_name=None):
- """Execute code located at the specified filesystem location
+ """Execute code located at the specified filesystem location.
+
+ path_name -- filesystem location of a Python script, zipfile,
+ or directory containing a top level __main__.py script.
+
+ Optional arguments:
+ init_globals -- dictionary used to pre-populate the module’s
+ globals dictionary before the code is executed.
- Returns the resulting top level namespace dictionary
+ run_name -- if not None, this will be used to set __name__;
+ otherwise, '' will be used for __name__.
- The file path may refer directly to a Python script (i.e.
- one that could be directly executed with execfile) or else
- it may refer to a zipfile or directory containing a top
- level __main__.py script.
+ Returns the resulting module globals dictionary.
"""
if run_name is None:
run_name = ""
diff --git a/Lib/shutil.py b/Lib/shutil.py
index e29fe4d83e9277..482ce95a7b23ca 100644
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -251,28 +251,37 @@ def copyfile(src, dst, *, follow_symlinks=True):
if not follow_symlinks and _islink(src):
os.symlink(os.readlink(src), dst)
else:
- with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
- # macOS
- if _HAS_FCOPYFILE:
- try:
- _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
- return dst
- except _GiveupOnFastCopy:
- pass
- # Linux
- elif _USE_CP_SENDFILE:
- try:
- _fastcopy_sendfile(fsrc, fdst)
- return dst
- except _GiveupOnFastCopy:
- pass
- # Windows, see:
- # https://github.com/python/cpython/pull/7160#discussion_r195405230
- elif _WINDOWS and file_size > 0:
- _copyfileobj_readinto(fsrc, fdst, min(file_size, COPY_BUFSIZE))
- return dst
-
- copyfileobj(fsrc, fdst)
+ with open(src, 'rb') as fsrc:
+ try:
+ with open(dst, 'wb') as fdst:
+ # macOS
+ if _HAS_FCOPYFILE:
+ try:
+ _fastcopy_fcopyfile(fsrc, fdst, posix._COPYFILE_DATA)
+ return dst
+ except _GiveupOnFastCopy:
+ pass
+ # Linux
+ elif _USE_CP_SENDFILE:
+ try:
+ _fastcopy_sendfile(fsrc, fdst)
+ return dst
+ except _GiveupOnFastCopy:
+ pass
+ # Windows, see:
+ # https://github.com/python/cpython/pull/7160#discussion_r195405230
+ elif _WINDOWS and file_size > 0:
+ _copyfileobj_readinto(fsrc, fdst, min(file_size, COPY_BUFSIZE))
+ return dst
+
+ copyfileobj(fsrc, fdst)
+
+ # Issue 43219, raise a less confusing exception
+ except IsADirectoryError as e:
+ if not os.path.exists(dst):
+ raise FileNotFoundError(f'Directory does not exist: {dst}') from e
+ else:
+ raise
return dst
@@ -478,12 +487,13 @@ def _copytree(entries, src, dst, symlinks, ignore, copy_function,
# otherwise let the copy occur. copy2 will raise an error
if srcentry.is_dir():
copytree(srcobj, dstname, symlinks, ignore,
- copy_function, dirs_exist_ok=dirs_exist_ok)
+ copy_function, ignore_dangling_symlinks,
+ dirs_exist_ok)
else:
copy_function(srcobj, dstname)
elif srcentry.is_dir():
copytree(srcobj, dstname, symlinks, ignore, copy_function,
- dirs_exist_ok=dirs_exist_ok)
+ ignore_dangling_symlinks, dirs_exist_ok)
else:
# Will raise a SpecialFileError for unsupported file types
copy_function(srcobj, dstname)
@@ -507,9 +517,6 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2,
ignore_dangling_symlinks=False, dirs_exist_ok=False):
"""Recursively copy a directory tree and return the destination directory.
- dirs_exist_ok dictates whether to raise an exception in case dst or any
- missing parent directory already exists.
-
If exception(s) occur, an Error is raised with a list of reasons.
If the optional symlinks flag is true, symbolic links in the
@@ -540,6 +547,11 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2,
destination path as arguments. By default, copy2() is used, but any
function that supports the same signature (like copy()) can be used.
+ If dirs_exist_ok is false (the default) and `dst` already exists, a
+ `FileExistsError` is raised. If `dirs_exist_ok` is true, the copying
+ operation will continue if it encounters existing directories, and files
+ within the `dst` tree will be overwritten by corresponding files from the
+ `src` tree.
"""
sys.audit("shutil.copytree", src, dst)
with os.scandir(src) as itr:
@@ -637,6 +649,7 @@ def _rmtree_safe_fd(topfd, path, onerror):
if is_dir:
try:
dirfd = os.open(entry.name, os.O_RDONLY, dir_fd=topfd)
+ dirfd_closed = False
except OSError:
onerror(os.open, fullname, sys.exc_info())
else:
@@ -644,6 +657,8 @@ def _rmtree_safe_fd(topfd, path, onerror):
if os.path.samestat(orig_st, os.fstat(dirfd)):
_rmtree_safe_fd(dirfd, fullname, onerror)
try:
+ os.close(dirfd)
+ dirfd_closed = True
os.rmdir(entry.name, dir_fd=topfd)
except OSError:
onerror(os.rmdir, fullname, sys.exc_info())
@@ -657,7 +672,8 @@ def _rmtree_safe_fd(topfd, path, onerror):
except OSError:
onerror(os.path.islink, fullname, sys.exc_info())
finally:
- os.close(dirfd)
+ if not dirfd_closed:
+ os.close(dirfd)
else:
try:
os.unlink(entry.name, dir_fd=topfd)
@@ -700,6 +716,7 @@ def onerror(*args):
return
try:
fd = os.open(path, os.O_RDONLY)
+ fd_closed = False
except Exception:
onerror(os.open, path, sys.exc_info())
return
@@ -707,6 +724,8 @@ def onerror(*args):
if os.path.samestat(orig_st, os.fstat(fd)):
_rmtree_safe_fd(fd, path, onerror)
try:
+ os.close(fd)
+ fd_closed = True
os.rmdir(path)
except OSError:
onerror(os.rmdir, path, sys.exc_info())
@@ -717,7 +736,8 @@ def onerror(*args):
except OSError:
onerror(os.path.islink, path, sys.exc_info())
finally:
- os.close(fd)
+ if not fd_closed:
+ os.close(fd)
else:
try:
if _rmtree_islink(path):
@@ -868,7 +888,7 @@ def _get_uid(name):
return None
def _make_tarball(base_name, base_dir, compress="gzip", verbose=0, dry_run=0,
- owner=None, group=None, logger=None):
+ owner=None, group=None, logger=None, root_dir=None):
"""Create a (possibly compressed) tar file from all the files under
'base_dir'.
@@ -925,14 +945,20 @@ def _set_uid_gid(tarinfo):
if not dry_run:
tar = tarfile.open(archive_name, 'w|%s' % tar_compression)
+ arcname = base_dir
+ if root_dir is not None:
+ base_dir = os.path.join(root_dir, base_dir)
try:
- tar.add(base_dir, filter=_set_uid_gid)
+ tar.add(base_dir, arcname, filter=_set_uid_gid)
finally:
tar.close()
+ if root_dir is not None:
+ archive_name = os.path.abspath(archive_name)
return archive_name
-def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None):
+def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0,
+ logger=None, owner=None, group=None, root_dir=None):
"""Create a zip file from all the files under 'base_dir'.
The output zip file will be named 'base_name' + ".zip". Returns the
@@ -956,42 +982,60 @@ def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None):
if not dry_run:
with zipfile.ZipFile(zip_filename, "w",
compression=zipfile.ZIP_DEFLATED) as zf:
- path = os.path.normpath(base_dir)
- if path != os.curdir:
- zf.write(path, path)
+ arcname = os.path.normpath(base_dir)
+ if root_dir is not None:
+ base_dir = os.path.join(root_dir, base_dir)
+ base_dir = os.path.normpath(base_dir)
+ if arcname != os.curdir:
+ zf.write(base_dir, arcname)
if logger is not None:
- logger.info("adding '%s'", path)
+ logger.info("adding '%s'", base_dir)
for dirpath, dirnames, filenames in os.walk(base_dir):
+ arcdirpath = dirpath
+ if root_dir is not None:
+ arcdirpath = os.path.relpath(arcdirpath, root_dir)
+ arcdirpath = os.path.normpath(arcdirpath)
for name in sorted(dirnames):
- path = os.path.normpath(os.path.join(dirpath, name))
- zf.write(path, path)
+ path = os.path.join(dirpath, name)
+ arcname = os.path.join(arcdirpath, name)
+ zf.write(path, arcname)
if logger is not None:
logger.info("adding '%s'", path)
for name in filenames:
- path = os.path.normpath(os.path.join(dirpath, name))
+ path = os.path.join(dirpath, name)
+ path = os.path.normpath(path)
if os.path.isfile(path):
- zf.write(path, path)
+ arcname = os.path.join(arcdirpath, name)
+ zf.write(path, arcname)
if logger is not None:
logger.info("adding '%s'", path)
+ if root_dir is not None:
+ zip_filename = os.path.abspath(zip_filename)
return zip_filename
+# Maps the name of the archive format to a tuple containing:
+# * the archiving function
+# * extra keyword arguments
+# * description
+# * does it support the root_dir argument?
_ARCHIVE_FORMATS = {
- 'tar': (_make_tarball, [('compress', None)], "uncompressed tar file"),
+ 'tar': (_make_tarball, [('compress', None)],
+ "uncompressed tar file", True),
}
if _ZLIB_SUPPORTED:
_ARCHIVE_FORMATS['gztar'] = (_make_tarball, [('compress', 'gzip')],
- "gzip'ed tar-file")
- _ARCHIVE_FORMATS['zip'] = (_make_zipfile, [], "ZIP file")
+ "gzip'ed tar-file", True)
+ _ARCHIVE_FORMATS['zip'] = (_make_zipfile, [], "ZIP file", True)
if _BZ2_SUPPORTED:
_ARCHIVE_FORMATS['bztar'] = (_make_tarball, [('compress', 'bzip2')],
- "bzip2'ed tar-file")
+ "bzip2'ed tar-file", True)
if _LZMA_SUPPORTED:
_ARCHIVE_FORMATS['xztar'] = (_make_tarball, [('compress', 'xz')],
- "xz'ed tar-file")
+ "xz'ed tar-file", True)
def get_archive_formats():
"""Returns a list of supported formats for archiving and unarchiving.
@@ -1022,7 +1066,7 @@ def register_archive_format(name, function, extra_args=None, description=''):
if not isinstance(element, (tuple, list)) or len(element) !=2:
raise TypeError('extra_args elements are : (arg_name, value)')
- _ARCHIVE_FORMATS[name] = (function, extra_args, description)
+ _ARCHIVE_FORMATS[name] = (function, extra_args, description, False)
def unregister_archive_format(name):
del _ARCHIVE_FORMATS[name]
@@ -1046,36 +1090,40 @@ def make_archive(base_name, format, root_dir=None, base_dir=None, verbose=0,
uses the current owner and group.
"""
sys.audit("shutil.make_archive", base_name, format, root_dir, base_dir)
- save_cwd = os.getcwd()
- if root_dir is not None:
- if logger is not None:
- logger.debug("changing into '%s'", root_dir)
- base_name = os.path.abspath(base_name)
- if not dry_run:
- os.chdir(root_dir)
-
- if base_dir is None:
- base_dir = os.curdir
-
- kwargs = {'dry_run': dry_run, 'logger': logger}
-
try:
format_info = _ARCHIVE_FORMATS[format]
except KeyError:
raise ValueError("unknown archive format '%s'" % format) from None
+ kwargs = {'dry_run': dry_run, 'logger': logger,
+ 'owner': owner, 'group': group}
+
func = format_info[0]
for arg, val in format_info[1]:
kwargs[arg] = val
- if format != 'zip':
- kwargs['owner'] = owner
- kwargs['group'] = group
+ if base_dir is None:
+ base_dir = os.curdir
+
+ support_root_dir = format_info[3]
+ save_cwd = None
+ if root_dir is not None:
+ if support_root_dir:
+ # Support path-like base_name here for backwards-compatibility.
+ base_name = os.fspath(base_name)
+ kwargs['root_dir'] = root_dir
+ else:
+ save_cwd = os.getcwd()
+ if logger is not None:
+ logger.debug("changing into '%s'", root_dir)
+ base_name = os.path.abspath(base_name)
+ if not dry_run:
+ os.chdir(root_dir)
try:
filename = func(base_name, base_dir, **kwargs)
finally:
- if root_dir is not None:
+ if save_cwd is not None:
if logger is not None:
logger.debug("changing back to '%s'", save_cwd)
os.chdir(save_cwd)
@@ -1161,24 +1209,20 @@ def _unpack_zipfile(filename, extract_dir):
if name.startswith('/') or '..' in name:
continue
- target = os.path.join(extract_dir, *name.split('/'))
- if not target:
+ targetpath = os.path.join(extract_dir, *name.split('/'))
+ if not targetpath:
continue
- _ensure_directory(target)
+ _ensure_directory(targetpath)
if not name.endswith('/'):
# file
- data = zip.read(info.filename)
- f = open(target, 'wb')
- try:
- f.write(data)
- finally:
- f.close()
- del data
+ with zip.open(name, 'r') as source, \
+ open(targetpath, 'wb') as target:
+ copyfileobj(source, target)
finally:
zip.close()
-def _unpack_tarfile(filename, extract_dir):
+def _unpack_tarfile(filename, extract_dir, *, filter=None):
"""Unpack tar/tar.gz/tar.bz2/tar.xz `filename` to `extract_dir`
"""
import tarfile # late import for breaking circular dependency
@@ -1188,10 +1232,15 @@ def _unpack_tarfile(filename, extract_dir):
raise ReadError(
"%s is not a compressed or uncompressed tar file" % filename)
try:
- tarobj.extractall(extract_dir)
+ tarobj.extractall(extract_dir, filter=filter)
finally:
tarobj.close()
+# Maps the name of the unpack format to a tuple containing:
+# * extensions
+# * the unpacking function
+# * extra keyword arguments
+# * description
_UNPACK_FORMATS = {
'tar': (['.tar'], _unpack_tarfile, [], "uncompressed tar file"),
'zip': (['.zip'], _unpack_zipfile, [], "ZIP file"),
@@ -1216,7 +1265,7 @@ def _find_unpack_format(filename):
return name
return None
-def unpack_archive(filename, extract_dir=None, format=None):
+def unpack_archive(filename, extract_dir=None, format=None, *, filter=None):
"""Unpack an archive.
`filename` is the name of the archive.
@@ -1230,6 +1279,9 @@ def unpack_archive(filename, extract_dir=None, format=None):
was registered for that extension.
In case none is found, a ValueError is raised.
+
+ If `filter` is given, it is passed to the underlying
+ extraction function.
"""
sys.audit("shutil.unpack_archive", filename, extract_dir, format)
@@ -1239,6 +1291,10 @@ def unpack_archive(filename, extract_dir=None, format=None):
extract_dir = os.fspath(extract_dir)
filename = os.fspath(filename)
+ if filter is None:
+ filter_kwargs = {}
+ else:
+ filter_kwargs = {'filter': filter}
if format is not None:
try:
format_info = _UNPACK_FORMATS[format]
@@ -1246,7 +1302,7 @@ def unpack_archive(filename, extract_dir=None, format=None):
raise ValueError("Unknown unpack format '{0}'".format(format)) from None
func = format_info[1]
- func(filename, extract_dir, **dict(format_info[2]))
+ func(filename, extract_dir, **dict(format_info[2]), **filter_kwargs)
else:
# we need to look at the registered unpackers supported extensions
format = _find_unpack_format(filename)
@@ -1254,7 +1310,7 @@ def unpack_archive(filename, extract_dir=None, format=None):
raise ReadError("Unknown archive format '{0}'".format(filename))
func = _UNPACK_FORMATS[format][1]
- kwargs = dict(_UNPACK_FORMATS[format][2])
+ kwargs = dict(_UNPACK_FORMATS[format][2]) | filter_kwargs
func(filename, extract_dir, **kwargs)
diff --git a/Lib/signal.py b/Lib/signal.py
index d4a6d6fe2ada8b..50b215b29d2fad 100644
--- a/Lib/signal.py
+++ b/Lib/signal.py
@@ -1,6 +1,5 @@
import _signal
from _signal import *
-from functools import wraps as _wraps
from enum import IntEnum as _IntEnum
_globals = globals()
@@ -42,6 +41,16 @@ def _enum_to_int(value):
return value
+# Similar to functools.wraps(), but only assign __doc__.
+# __module__ should be preserved,
+# __name__ and __qualname__ are already fine,
+# __annotations__ is not set.
+def _wraps(wrapped):
+ def decorator(wrapper):
+ wrapper.__doc__ = wrapped.__doc__
+ return wrapper
+ return decorator
+
@_wraps(_signal.signal)
def signal(signalnum, handler):
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
@@ -59,7 +68,6 @@ def getsignal(signalnum):
def pthread_sigmask(how, mask):
sigs_set = _signal.pthread_sigmask(how, mask)
return set(_int_to_enum(x, Signals) for x in sigs_set)
- pthread_sigmask.__doc__ = _signal.pthread_sigmask.__doc__
if 'sigpending' in _globals:
@@ -73,7 +81,6 @@ def sigpending():
def sigwait(sigset):
retsig = _signal.sigwait(sigset)
return _int_to_enum(retsig, Signals)
- sigwait.__doc__ = _signal.sigwait
if 'valid_signals' in _globals:
diff --git a/Lib/site.py b/Lib/site.py
index 939893eb5ee93b..5302037e0bf2c1 100644
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -74,6 +74,7 @@
import builtins
import _sitebuiltins
import io
+import stat
# Prefixes for site-packages; add additional prefixes like /usr/local here
PREFIXES = [sys.prefix, sys.exec_prefix]
@@ -168,6 +169,14 @@ def addpackage(sitedir, name, known_paths):
else:
reset = False
fullname = os.path.join(sitedir, name)
+ try:
+ st = os.lstat(fullname)
+ except OSError:
+ return
+ if ((getattr(st, 'st_flags', 0) & stat.UF_HIDDEN) or
+ (getattr(st, 'st_file_attributes', 0) & stat.FILE_ATTRIBUTE_HIDDEN)):
+ _trace(f"Skipping hidden .pth file: {fullname!r}")
+ return
_trace(f"Processing .pth file: {fullname!r}")
try:
# locale encoding is not ideal especially on Windows. But we have used
@@ -221,7 +230,8 @@ def addsitedir(sitedir, known_paths=None):
names = os.listdir(sitedir)
except OSError:
return
- names = [name for name in names if name.endswith(".pth")]
+ names = [name for name in names
+ if name.endswith(".pth") and not name.startswith(".")]
for name in sorted(names):
addpackage(sitedir, name, known_paths)
if reset:
diff --git a/Lib/smtpd.py b/Lib/smtpd.py
index 1e2adc87a2bf2a..963e0a7689c269 100755
--- a/Lib/smtpd.py
+++ b/Lib/smtpd.py
@@ -83,8 +83,6 @@
import getopt
import time
import socket
-import asyncore
-import asynchat
import collections
from warnings import warn
from email._header_value_parser import get_addr_spec, get_angle_addr
@@ -94,6 +92,20 @@
"MailmanProxy",
]
+warn(
+ 'The smtpd module is deprecated and unmaintained and will be removed '
+ 'in Python 3.12. Please see aiosmtpd '
+ '(https://aiosmtpd.readthedocs.io/) for the recommended replacement.',
+ DeprecationWarning,
+ stacklevel=2)
+
+
+# These are imported after the above warning so that users get the correct
+# deprecation warning.
+import asyncore
+import asynchat
+
+
program = sys.argv[0]
__version__ = 'Python SMTP proxy version 0.3'
diff --git a/Lib/smtplib.py b/Lib/smtplib.py
index bfba5c8f21be06..324a1c19f12afe 100755
--- a/Lib/smtplib.py
+++ b/Lib/smtplib.py
@@ -168,7 +168,7 @@ def quotedata(data):
"""Quote data for email.
Double leading '.', and change Unix newline '\\n', or Mac '\\r' into
- Internet CRLF end-of-line.
+ internet CRLF end-of-line.
"""
return re.sub(r'(?m)^\.', '..',
re.sub(r'(?:\r\n|\n|\r(?!\n))', CRLF, data))
@@ -367,10 +367,15 @@ def send(self, s):
def putcmd(self, cmd, args=""):
"""Send a command to the server."""
if args == "":
- str = '%s%s' % (cmd, CRLF)
+ s = cmd
else:
- str = '%s %s%s' % (cmd, args, CRLF)
- self.send(str)
+ s = f'{cmd} {args}'
+ if '\r' in s or '\n' in s:
+ s = s.replace('\n', '\\n').replace('\r', '\\r')
+ raise ValueError(
+ f'command and arguments contain prohibited newline characters: {s}'
+ )
+ self.send(f'{s}{CRLF}')
def getreply(self):
"""Get a reply from the server.
diff --git a/Lib/socket.py b/Lib/socket.py
old mode 100755
new mode 100644
index fc11eb783c3dd1..ecaf73cf307cfa
--- a/Lib/socket.py
+++ b/Lib/socket.py
@@ -589,16 +589,65 @@ def fromshare(info):
return socket(0, 0, 0, info)
__all__.append("fromshare")
-if hasattr(_socket, "socketpair"):
+# Origin: https://gist.github.com/4325783, by Geert Jansen. Public domain.
+# This is used if _socket doesn't natively provide socketpair. It's
+# always defined so that it can be patched in for testing purposes.
+def _fallback_socketpair(family=AF_INET, type=SOCK_STREAM, proto=0):
+ if family == AF_INET:
+ host = _LOCALHOST
+ elif family == AF_INET6:
+ host = _LOCALHOST_V6
+ else:
+ raise ValueError("Only AF_INET and AF_INET6 socket address families "
+ "are supported")
+ if type != SOCK_STREAM:
+ raise ValueError("Only SOCK_STREAM socket type is supported")
+ if proto != 0:
+ raise ValueError("Only protocol zero is supported")
+
+ # We create a connected TCP socket. Note the trick with
+ # setblocking(False) that prevents us from having to create a thread.
+ lsock = socket(family, type, proto)
+ try:
+ lsock.bind((host, 0))
+ lsock.listen()
+ # On IPv6, ignore flow_info and scope_id
+ addr, port = lsock.getsockname()[:2]
+ csock = socket(family, type, proto)
+ try:
+ csock.setblocking(False)
+ try:
+ csock.connect((addr, port))
+ except (BlockingIOError, InterruptedError):
+ pass
+ csock.setblocking(True)
+ ssock, _ = lsock.accept()
+ except:
+ csock.close()
+ raise
+ finally:
+ lsock.close()
- def socketpair(family=None, type=SOCK_STREAM, proto=0):
- """socketpair([family[, type[, proto]]]) -> (socket object, socket object)
+ # Authenticating avoids using a connection from something else
+ # able to connect to {host}:{port} instead of us.
+ # We expect only AF_INET and AF_INET6 families.
+ try:
+ if (
+ ssock.getsockname() != csock.getpeername()
+ or csock.getsockname() != ssock.getpeername()
+ ):
+ raise ConnectionError("Unexpected peer connection")
+ except:
+ # getsockname() and getpeername() can fail
+ # if either socket isn't connected.
+ ssock.close()
+ csock.close()
+ raise
- Create a pair of socket objects from the sockets returned by the platform
- socketpair() function.
- The arguments are the same as for socket() except the default family is
- AF_UNIX if defined on the platform; otherwise, the default is AF_INET.
- """
+ return (ssock, csock)
+
+if hasattr(_socket, "socketpair"):
+ def socketpair(family=None, type=SOCK_STREAM, proto=0):
if family is None:
try:
family = AF_UNIX
@@ -610,44 +659,7 @@ def socketpair(family=None, type=SOCK_STREAM, proto=0):
return a, b
else:
-
- # Origin: https://gist.github.com/4325783, by Geert Jansen. Public domain.
- def socketpair(family=AF_INET, type=SOCK_STREAM, proto=0):
- if family == AF_INET:
- host = _LOCALHOST
- elif family == AF_INET6:
- host = _LOCALHOST_V6
- else:
- raise ValueError("Only AF_INET and AF_INET6 socket address families "
- "are supported")
- if type != SOCK_STREAM:
- raise ValueError("Only SOCK_STREAM socket type is supported")
- if proto != 0:
- raise ValueError("Only protocol zero is supported")
-
- # We create a connected TCP socket. Note the trick with
- # setblocking(False) that prevents us from having to create a thread.
- lsock = socket(family, type, proto)
- try:
- lsock.bind((host, 0))
- lsock.listen()
- # On IPv6, ignore flow_info and scope_id
- addr, port = lsock.getsockname()[:2]
- csock = socket(family, type, proto)
- try:
- csock.setblocking(False)
- try:
- csock.connect((addr, port))
- except (BlockingIOError, InterruptedError):
- pass
- csock.setblocking(True)
- ssock, _ = lsock.accept()
- except:
- csock.close()
- raise
- finally:
- lsock.close()
- return (ssock, csock)
+ socketpair = _fallback_socketpair
__all__.append("socketpair")
socketpair.__doc__ = """socketpair([family[, type[, proto]]]) -> (socket object, socket object)
@@ -782,11 +794,12 @@ def getfqdn(name=''):
An empty argument is interpreted as meaning the local host.
First the hostname returned by gethostbyaddr() is checked, then
- possibly existing aliases. In case no FQDN is available, hostname
- from gethostname() is returned.
+ possibly existing aliases. In case no FQDN is available and `name`
+ was given, it is returned unchanged. If `name` was empty, '0.0.0.0' or '::',
+ hostname from gethostname() is returned.
"""
name = name.strip()
- if not name or name == '0.0.0.0':
+ if not name or name in ('0.0.0.0', '::'):
name = gethostname()
try:
hostname, aliases, ipaddrs = gethostbyaddr(name)
@@ -901,7 +914,7 @@ def create_server(address, *, family=AF_INET, backlog=None, reuse_port=False,
# address, effectively preventing this one from accepting
# connections. Also, it may set the process in a state where
# it'll no longer respond to any signals or graceful kills.
- # See: msdn2.microsoft.com/en-us/library/ms740621(VS.85).aspx
+ # See: https://learn.microsoft.com/windows/win32/winsock/using-so-reuseaddr-and-so-exclusiveaddruse
if os.name not in ('nt', 'cygwin') and \
hasattr(_socket, 'SO_REUSEADDR'):
try:
diff --git a/Lib/sqlite3/__init__.py b/Lib/sqlite3/__init__.py
index f001c0678e195f..5a2dbd360fb49f 100644
--- a/Lib/sqlite3/__init__.py
+++ b/Lib/sqlite3/__init__.py
@@ -20,6 +20,40 @@
# misrepresented as being the original software.
# 3. This notice may not be removed or altered from any source distribution.
+"""
+The sqlite3 extension module provides a DB-API 2.0 (PEP 249) compliant
+interface to the SQLite library, and requires SQLite 3.7.15 or newer.
+
+To use the module, start by creating a database Connection object:
+
+ import sqlite3
+ cx = sqlite3.connect("test.db") # test.db will be created or opened
+
+The special path name ":memory:" can be provided to connect to a transient
+in-memory database:
+
+ cx = sqlite3.connect(":memory:") # connect to a database in RAM
+
+Once a connection has been established, create a Cursor object and call
+its execute() method to perform SQL queries:
+
+ cu = cx.cursor()
+
+ # create a table
+ cu.execute("create table lang(name, first_appeared)")
+
+ # insert values into a table
+ cu.execute("insert into lang values (?, ?)", ("C", 1972))
+
+ # execute a query and iterate over the result
+ for row in cu.execute("select * from lang"):
+ print(row)
+
+ cx.close()
+
+The sqlite3 module is written by Gerhard Häring .
+"""
+
from sqlite3.dbapi2 import *
diff --git a/Lib/sqlite3/dump.py b/Lib/sqlite3/dump.py
index de9c368be3014e..07b9da10b920f9 100644
--- a/Lib/sqlite3/dump.py
+++ b/Lib/sqlite3/dump.py
@@ -28,9 +28,16 @@ def _iterdump(connection):
ORDER BY "name"
"""
schema_res = cu.execute(q)
+ sqlite_sequence = []
for table_name, type, sql in schema_res.fetchall():
if table_name == 'sqlite_sequence':
- yield('DELETE FROM "sqlite_sequence";')
+ rows = cu.execute('SELECT * FROM "sqlite_sequence";').fetchall()
+ sqlite_sequence = ['DELETE FROM "sqlite_sequence"']
+ sqlite_sequence += [
+ f'INSERT INTO "sqlite_sequence" VALUES(\'{row[0]}\',{row[1]})'
+ for row in rows
+ ]
+ continue
elif table_name == 'sqlite_stat1':
yield('ANALYZE "sqlite_master";')
elif table_name.startswith('sqlite_'):
@@ -67,4 +74,9 @@ def _iterdump(connection):
for name, type, sql in schema_res.fetchall():
yield('{0};'.format(sql))
+ # gh-79009: Yield statements concerning the sqlite_sequence table at the
+ # end of the transaction.
+ for row in sqlite_sequence:
+ yield('{0};'.format(row))
+
yield('COMMIT;')
diff --git a/Lib/sqlite3/test/dbapi.py b/Lib/sqlite3/test/dbapi.py
index 39c9bf5b61143d..a764c82b0d30fe 100644
--- a/Lib/sqlite3/test/dbapi.py
+++ b/Lib/sqlite3/test/dbapi.py
@@ -20,11 +20,13 @@
# misrepresented as being the original software.
# 3. This notice may not be removed or altered from any source distribution.
+import subprocess
import threading
import unittest
import sqlite3 as sqlite
import sys
+from test.support import check_disallow_instantiation, SHORT_TIMEOUT
from test.support.os_helper import TESTFN, unlink
@@ -92,6 +94,10 @@ def test_shared_cache_deprecated(self):
sqlite.enable_shared_cache(enable)
self.assertIn("dbapi.py", cm.filename)
+ def test_disallow_instantiation(self):
+ cx = sqlite.connect(":memory:")
+ check_disallow_instantiation(self, type(cx("select 1")))
+
class ConnectionTests(unittest.TestCase):
@@ -192,6 +198,26 @@ def test_open_uri(self):
cx.execute('insert into test(id) values(1)')
+class UninitialisedConnectionTests(unittest.TestCase):
+ def setUp(self):
+ self.cx = sqlite.Connection.__new__(sqlite.Connection)
+
+ def test_uninit_operations(self):
+ funcs = (
+ lambda: self.cx.isolation_level,
+ lambda: self.cx.total_changes,
+ lambda: self.cx.in_transaction,
+ lambda: self.cx.iterdump(),
+ lambda: self.cx.cursor(),
+ lambda: self.cx.close(),
+ )
+ for func in funcs:
+ with self.subTest(func=func):
+ self.assertRaisesRegex(sqlite.ProgrammingError,
+ "Base Connection.__init__ not called",
+ func)
+
+
class CursorTests(unittest.TestCase):
def setUp(self):
self.cx = sqlite.connect(":memory:")
@@ -355,6 +381,14 @@ def test_rowcount_executemany(self):
self.cu.executemany("insert into test(name) values (?)", [(1,), (2,), (3,)])
self.assertEqual(self.cu.rowcount, 3)
+ @unittest.skipIf(sqlite.sqlite_version_info < (3, 35, 0),
+ "Requires SQLite 3.35.0 or newer")
+ def test_rowcount_update_returning(self):
+ # gh-93421: rowcount is updated correctly for UPDATE...RETURNING queries
+ self.cu.execute("update test set name='bar' where name='foo' returning 1")
+ self.assertEqual(self.cu.fetchone()[0], 1)
+ self.assertEqual(self.cu.rowcount, 1)
+
def test_total_changes(self):
self.cu.execute("insert into test(name) values ('foo')")
self.cu.execute("insert into test(name) values ('foo')")
@@ -372,6 +406,9 @@ class MyIter:
def __init__(self):
self.value = 5
+ def __iter__(self):
+ return self
+
def __next__(self):
if self.value == 10:
raise StopIteration
@@ -425,7 +462,7 @@ def test_fetchone_no_statement(self):
self.assertEqual(row, None)
def test_array_size(self):
- # must default ot 1
+ # must default to 1
self.assertEqual(self.cu.arraysize, 1)
# now set to 2
@@ -933,6 +970,77 @@ def test_on_conflict_replace(self):
self.assertEqual(self.cu.fetchall(), [('Very different data!', 'foo')])
+class MultiprocessTests(unittest.TestCase):
+ CONNECTION_TIMEOUT = SHORT_TIMEOUT / 1000. # Defaults to 30 ms
+
+ def tearDown(self):
+ unlink(TESTFN)
+
+ def test_ctx_mgr_rollback_if_commit_failed(self):
+ # bpo-27334: ctx manager does not rollback if commit fails
+ SCRIPT = f"""if 1:
+ import sqlite3
+ def wait():
+ print("started")
+ assert "database is locked" in input()
+
+ cx = sqlite3.connect("{TESTFN}", timeout={self.CONNECTION_TIMEOUT})
+ cx.create_function("wait", 0, wait)
+ with cx:
+ cx.execute("create table t(t)")
+ try:
+ # execute two transactions; both will try to lock the db
+ cx.executescript('''
+ -- start a transaction and wait for parent
+ begin transaction;
+ select * from t;
+ select wait();
+ rollback;
+
+ -- start a new transaction; would fail if parent holds lock
+ begin transaction;
+ select * from t;
+ rollback;
+ ''')
+ finally:
+ cx.close()
+ """
+
+ # spawn child process
+ proc = subprocess.Popen(
+ [sys.executable, "-c", SCRIPT],
+ encoding="utf-8",
+ bufsize=0,
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ )
+ self.addCleanup(proc.communicate)
+
+ # wait for child process to start
+ self.assertEqual("started", proc.stdout.readline().strip())
+
+ cx = sqlite.connect(TESTFN, timeout=self.CONNECTION_TIMEOUT)
+ try: # context manager should correctly release the db lock
+ with cx:
+ cx.execute("insert into t values('test')")
+ except sqlite.OperationalError as exc:
+ proc.stdin.write(str(exc))
+ else:
+ proc.stdin.write("no error")
+ finally:
+ cx.close()
+
+ # terminate child process
+ self.assertIsNone(proc.returncode)
+ try:
+ proc.communicate(input="end", timeout=SHORT_TIMEOUT)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ proc.communicate()
+ raise
+ self.assertEqual(proc.returncode, 0)
+
+
def suite():
tests = [
ClosedConTests,
@@ -942,8 +1050,10 @@ def suite():
CursorTests,
ExtensionTests,
ModuleTests,
+ MultiprocessTests,
SqliteOnConflictTests,
ThreadTests,
+ UninitialisedConnectionTests,
]
return unittest.TestSuite(
[unittest.TestLoader().loadTestsFromTestCase(t) for t in tests]
diff --git a/Lib/sqlite3/test/dump.py b/Lib/sqlite3/test/dump.py
index 618a7fd31c1543..0e324ee1769e87 100644
--- a/Lib/sqlite3/test/dump.py
+++ b/Lib/sqlite3/test/dump.py
@@ -3,6 +3,7 @@
import unittest
import sqlite3 as sqlite
+
class DumpTests(unittest.TestCase):
def setUp(self):
self.cx = sqlite.connect(":memory:")
@@ -49,6 +50,51 @@ def test_table_dump(self):
[self.assertEqual(expected_sqls[i], actual_sqls[i])
for i in range(len(expected_sqls))]
+ def test_dump_autoincrement(self):
+ expected = [
+ 'CREATE TABLE "t1" (id integer primary key autoincrement);',
+ 'INSERT INTO "t1" VALUES(NULL);',
+ 'CREATE TABLE "t2" (id integer primary key autoincrement);',
+ ]
+ self.cu.executescript("".join(expected))
+
+ # the NULL value should now be automatically be set to 1
+ expected[1] = expected[1].replace("NULL", "1")
+ expected.insert(0, "BEGIN TRANSACTION;")
+ expected.extend([
+ 'DELETE FROM "sqlite_sequence";',
+ 'INSERT INTO "sqlite_sequence" VALUES(\'t1\',1);',
+ 'COMMIT;',
+ ])
+
+ actual = [stmt for stmt in self.cx.iterdump()]
+ self.assertEqual(expected, actual)
+
+ def test_dump_autoincrement_create_new_db(self):
+ self.cu.execute("BEGIN TRANSACTION")
+ self.cu.execute("CREATE TABLE t1 (id integer primary key autoincrement)")
+ self.cu.execute("CREATE TABLE t2 (id integer primary key autoincrement)")
+ self.cu.executemany("INSERT INTO t1 VALUES(?)", ((None,) for _ in range(9)))
+ self.cu.executemany("INSERT INTO t2 VALUES(?)", ((None,) for _ in range(4)))
+ self.cx.commit()
+
+ cx2 = sqlite.connect(":memory:")
+ query = "".join(self.cx.iterdump())
+ cx2.executescript(query)
+ cu2 = cx2.cursor()
+
+ dataset = (
+ ("t1", 9),
+ ("t2", 4),
+ )
+ for table, seq in dataset:
+ with self.subTest(table=table, seq=seq):
+ res = cu2.execute("""
+ SELECT "seq" FROM "sqlite_sequence" WHERE "name" == ?
+ """, (table,))
+ rows = res.fetchall()
+ self.assertEqual(rows[0][0], seq)
+
def test_unorderable_row(self):
# iterdump() should be able to cope with unorderable row types (issue #15545)
class UnorderableRow:
diff --git a/Lib/sqlite3/test/factory.py b/Lib/sqlite3/test/factory.py
index 876428497542f8..40a290f0c9816f 100644
--- a/Lib/sqlite3/test/factory.py
+++ b/Lib/sqlite3/test/factory.py
@@ -155,8 +155,14 @@ def test_sqlite_row_iter(self):
"""Checks if the row object is iterable"""
self.con.row_factory = sqlite.Row
row = self.con.execute("select 1 as a, 2 as b").fetchone()
- for col in row:
- pass
+
+ # Is iterable in correct order and produces valid results:
+ items = [col for col in row]
+ self.assertEqual(items, [1, 2])
+
+ # Is iterable the second time:
+ items = [col for col in row]
+ self.assertEqual(items, [1, 2])
def test_sqlite_row_as_tuple(self):
"""Checks if the row object can be converted to a tuple"""
diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py
index a219e8911f7063..97121eea1be5fb 100644
--- a/Lib/sqlite3/test/hooks.py
+++ b/Lib/sqlite3/test/hooks.py
@@ -20,6 +20,7 @@
# misrepresented as being the original software.
# 3. This notice may not be removed or altered from any source distribution.
+import contextlib
import unittest
import sqlite3 as sqlite
@@ -200,6 +201,16 @@ def progress():
self.assertEqual(action, 0, "progress handler was not cleared")
class TraceCallbackTests(unittest.TestCase):
+ @contextlib.contextmanager
+ def check_stmt_trace(self, cx, expected):
+ try:
+ traced = []
+ cx.set_trace_callback(lambda stmt: traced.append(stmt))
+ yield
+ finally:
+ self.assertEqual(traced, expected)
+ cx.set_trace_callback(None)
+
def test_trace_callback_used(self):
"""
Test that the trace callback is invoked once it is set.
@@ -237,7 +248,7 @@ def trace(statement):
traced_statements.append(statement)
con.set_trace_callback(trace)
con.execute("create table foo(x)")
- con.execute('insert into foo(x) values ("%s")' % unicode_value)
+ con.execute("insert into foo(x) values ('%s')" % unicode_value)
con.commit()
self.assertTrue(any(unicode_value in stmt for stmt in traced_statements),
"Unicode data %s garbled in trace callback: %s"
@@ -261,6 +272,21 @@ def trace(statement):
cur.execute(queries[1])
self.assertEqual(traced_statements, queries)
+ def test_trace_expanded_sql(self):
+ expected = [
+ "create table t(t)",
+ "BEGIN ",
+ "insert into t values(0)",
+ "insert into t values(1)",
+ "insert into t values(2)",
+ "COMMIT",
+ ]
+ cx = sqlite.connect(":memory:")
+ with self.check_stmt_trace(cx, expected):
+ with cx:
+ cx.execute("create table t(t)")
+ cx.executemany("insert into t values(?)", ((v,) for v in range(3)))
+
def suite():
tests = [
diff --git a/Lib/sqlite3/test/regression.py b/Lib/sqlite3/test/regression.py
index 417a53109c87c7..8937f9f69eb034 100644
--- a/Lib/sqlite3/test/regression.py
+++ b/Lib/sqlite3/test/regression.py
@@ -27,6 +27,9 @@
import functools
from test import support
+from unittest.mock import patch
+
+
class RegressionTests(unittest.TestCase):
def setUp(self):
self.con = sqlite.connect(":memory:")
@@ -125,11 +128,11 @@ def test_type_map_usage(self):
con = sqlite.connect(":memory:",detect_types=sqlite.PARSE_DECLTYPES)
con.execute("create table foo(bar timestamp)")
con.execute("insert into foo(bar) values (?)", (datetime.datetime.now(),))
- con.execute(SELECT)
+ con.execute(SELECT).close()
con.execute("drop table foo")
con.execute("create table foo(bar integer)")
con.execute("insert into foo(bar) values (5)")
- con.execute(SELECT)
+ con.execute(SELECT).close()
def test_bind_mutating_list(self):
# Issue41662: Crash when mutate a list of parameters during iteration.
@@ -415,9 +418,46 @@ def test_return_empty_bytestring(self):
self.assertEqual(val, b'')
+class RecursiveUseOfCursors(unittest.TestCase):
+ # GH-80254: sqlite3 should not segfault for recursive use of cursors.
+ msg = "Recursive use of cursors not allowed"
+
+ def setUp(self):
+ self.con = sqlite.connect(":memory:",
+ detect_types=sqlite.PARSE_COLNAMES)
+ self.cur = self.con.cursor()
+ self.cur.execute("create table test(x foo)")
+ self.cur.executemany("insert into test(x) values (?)",
+ [("foo",), ("bar",)])
+
+ def tearDown(self):
+ self.cur.close()
+ self.con.close()
+
+ def test_recursive_cursor_init(self):
+ conv = lambda x: self.cur.__init__(self.con)
+ with patch.dict(sqlite.converters, {"INIT": conv}):
+ with self.assertRaisesRegex(sqlite.ProgrammingError, self.msg):
+ self.cur.execute(f'select x as "x [INIT]", x from test')
+
+ def test_recursive_cursor_close(self):
+ conv = lambda x: self.cur.close()
+ with patch.dict(sqlite.converters, {"CLOSE": conv}):
+ with self.assertRaisesRegex(sqlite.ProgrammingError, self.msg):
+ self.cur.execute(f'select x as "x [CLOSE]", x from test')
+
+ def test_recursive_cursor_fetch(self):
+ conv = lambda x, l=[]: self.cur.fetchone() if l else l.append(None)
+ with patch.dict(sqlite.converters, {"ITER": conv}):
+ self.cur.execute(f'select x as "x [ITER]", x from test')
+ with self.assertRaisesRegex(sqlite.ProgrammingError, self.msg):
+ self.cur.fetchall()
+
+
def suite():
tests = [
- RegressionTests
+ RegressionTests,
+ RecursiveUseOfCursors,
]
return unittest.TestSuite(
[unittest.TestLoader().loadTestsFromTestCase(t) for t in tests]
diff --git a/Lib/sqlite3/test/transactions.py b/Lib/sqlite3/test/transactions.py
index 80284902a1a6e9..4ebc7fbc785a90 100644
--- a/Lib/sqlite3/test/transactions.py
+++ b/Lib/sqlite3/test/transactions.py
@@ -23,33 +23,31 @@
import os, unittest
import sqlite3 as sqlite
-def get_db_path():
- return "sqlite_testdb"
+from test.support import LOOPBACK_TIMEOUT
+from test.support.os_helper import TESTFN, unlink
+
+
+TIMEOUT = LOOPBACK_TIMEOUT / 10
+
class TransactionTests(unittest.TestCase):
def setUp(self):
- try:
- os.remove(get_db_path())
- except OSError:
- pass
-
- self.con1 = sqlite.connect(get_db_path(), timeout=0.1)
+ self.con1 = sqlite.connect(TESTFN, timeout=TIMEOUT)
self.cur1 = self.con1.cursor()
- self.con2 = sqlite.connect(get_db_path(), timeout=0.1)
+ self.con2 = sqlite.connect(TESTFN, timeout=TIMEOUT)
self.cur2 = self.con2.cursor()
def tearDown(self):
- self.cur1.close()
- self.con1.close()
+ try:
+ self.cur1.close()
+ self.con1.close()
- self.cur2.close()
- self.con2.close()
+ self.cur2.close()
+ self.con2.close()
- try:
- os.unlink(get_db_path())
- except OSError:
- pass
+ finally:
+ unlink(TESTFN)
def test_dml_does_not_auto_commit_before(self):
self.cur1.execute("create table test(i)")
diff --git a/Lib/sqlite3/test/userfunctions.py b/Lib/sqlite3/test/userfunctions.py
index 148d9f596a91c8..539fd4bcac6958 100644
--- a/Lib/sqlite3/test/userfunctions.py
+++ b/Lib/sqlite3/test/userfunctions.py
@@ -25,8 +25,13 @@
import unittest.mock
import sqlite3 as sqlite
+from test.support import gc_collect
+
+
def func_returntext():
return "foo"
+def func_returntextwithnull():
+ return "1\x002"
def func_returnunicode():
return "bar"
def func_returnint():
@@ -42,22 +47,6 @@ def func_returnlonglong():
def func_raiseexception():
5/0
-def func_isstring(v):
- return type(v) is str
-def func_isint(v):
- return type(v) is int
-def func_isfloat(v):
- return type(v) is float
-def func_isnone(v):
- return type(v) is type(None)
-def func_isblob(v):
- return isinstance(v, (bytes, memoryview))
-def func_islonglong(v):
- return isinstance(v, int) and v >= 1<<31
-
-def func(*args):
- return len(args)
-
class AggrNoStep:
def __init__(self):
pass
@@ -137,26 +126,34 @@ def step(self, val):
def finalize(self):
return self.val
+class AggrText:
+ def __init__(self):
+ self.txt = ""
+ def step(self, txt):
+ self.txt = self.txt + txt
+ def finalize(self):
+ return self.txt
+
+
class FunctionTests(unittest.TestCase):
def setUp(self):
self.con = sqlite.connect(":memory:")
self.con.create_function("returntext", 0, func_returntext)
+ self.con.create_function("returntextwithnull", 0, func_returntextwithnull)
self.con.create_function("returnunicode", 0, func_returnunicode)
self.con.create_function("returnint", 0, func_returnint)
self.con.create_function("returnfloat", 0, func_returnfloat)
self.con.create_function("returnnull", 0, func_returnnull)
self.con.create_function("returnblob", 0, func_returnblob)
self.con.create_function("returnlonglong", 0, func_returnlonglong)
+ self.con.create_function("returnnan", 0, lambda: float("nan"))
+ self.con.create_function("returntoolargeint", 0, lambda: 1 << 65)
self.con.create_function("raiseexception", 0, func_raiseexception)
- self.con.create_function("isstring", 1, func_isstring)
- self.con.create_function("isint", 1, func_isint)
- self.con.create_function("isfloat", 1, func_isfloat)
- self.con.create_function("isnone", 1, func_isnone)
- self.con.create_function("isblob", 1, func_isblob)
- self.con.create_function("islonglong", 1, func_islonglong)
- self.con.create_function("spam", -1, func)
+ self.con.create_function("isblob", 1, lambda x: isinstance(x, bytes))
+ self.con.create_function("isnone", 1, lambda x: x is None)
+ self.con.create_function("spam", -1, lambda *x: len(x))
self.con.execute("create table test(t text)")
def tearDown(self):
@@ -185,6 +182,12 @@ def test_func_return_text(self):
self.assertEqual(type(val), str)
self.assertEqual(val, "foo")
+ def test_func_return_text_with_null_char(self):
+ cur = self.con.cursor()
+ res = cur.execute("select returntextwithnull()").fetchone()[0]
+ self.assertEqual(type(res), str)
+ self.assertEqual(res, "1\x002")
+
def test_func_return_unicode(self):
cur = self.con.cursor()
cur.execute("select returnunicode()")
@@ -227,6 +230,16 @@ def test_func_return_long_long(self):
val = cur.fetchone()[0]
self.assertEqual(val, 1<<31)
+ def test_func_return_nan(self):
+ cur = self.con.cursor()
+ cur.execute("select returnnan()")
+ self.assertIsNone(cur.fetchone()[0])
+
+ def test_func_return_too_large_int(self):
+ cur = self.con.cursor()
+ with self.assertRaises(sqlite.OperationalError):
+ self.con.execute("select returntoolargeint()")
+
def test_func_exception(self):
cur = self.con.cursor()
with self.assertRaises(sqlite.OperationalError) as cm:
@@ -234,42 +247,6 @@ def test_func_exception(self):
cur.fetchone()
self.assertEqual(str(cm.exception), 'user-defined function raised exception')
- def test_param_string(self):
- cur = self.con.cursor()
- cur.execute("select isstring(?)", ("foo",))
- val = cur.fetchone()[0]
- self.assertEqual(val, 1)
-
- def test_param_int(self):
- cur = self.con.cursor()
- cur.execute("select isint(?)", (42,))
- val = cur.fetchone()[0]
- self.assertEqual(val, 1)
-
- def test_param_float(self):
- cur = self.con.cursor()
- cur.execute("select isfloat(?)", (3.14,))
- val = cur.fetchone()[0]
- self.assertEqual(val, 1)
-
- def test_param_none(self):
- cur = self.con.cursor()
- cur.execute("select isnone(?)", (None,))
- val = cur.fetchone()[0]
- self.assertEqual(val, 1)
-
- def test_param_blob(self):
- cur = self.con.cursor()
- cur.execute("select isblob(?)", (memoryview(b"blob"),))
- val = cur.fetchone()[0]
- self.assertEqual(val, 1)
-
- def test_param_long_long(self):
- cur = self.con.cursor()
- cur.execute("select islonglong(?)", (1<<42,))
- val = cur.fetchone()[0]
- self.assertEqual(val, 1)
-
def test_any_arguments(self):
cur = self.con.cursor()
cur.execute("select spam(?, ?)", (1, 2))
@@ -280,6 +257,52 @@ def test_empty_blob(self):
cur = self.con.execute("select isblob(x'')")
self.assertTrue(cur.fetchone()[0])
+ def test_nan_float(self):
+ cur = self.con.execute("select isnone(?)", (float("nan"),))
+ # SQLite has no concept of nan; it is converted to NULL
+ self.assertTrue(cur.fetchone()[0])
+
+ def test_too_large_int(self):
+ err = "Python int too large to convert to SQLite INTEGER"
+ self.assertRaisesRegex(OverflowError, err, self.con.execute,
+ "select spam(?)", (1 << 65,))
+
+ def test_non_contiguous_blob(self):
+ self.assertRaisesRegex(ValueError, "could not convert BLOB to buffer",
+ self.con.execute, "select spam(?)",
+ (memoryview(b"blob")[::2],))
+
+ def test_param_surrogates(self):
+ self.assertRaisesRegex(UnicodeEncodeError, "surrogates not allowed",
+ self.con.execute, "select spam(?)",
+ ("\ud803\ude6d",))
+
+ def test_func_params(self):
+ results = []
+ def append_result(arg):
+ results.append((arg, type(arg)))
+ self.con.create_function("test_params", 1, append_result)
+
+ dataset = [
+ (42, int),
+ (-1, int),
+ (1234567890123456789, int),
+ (4611686018427387905, int), # 63-bit int with non-zero low bits
+ (3.14, float),
+ (float('inf'), float),
+ ("text", str),
+ ("1\x002", str),
+ ("\u02e2q\u02e1\u2071\u1d57\u1d49", str),
+ (b"blob", bytes),
+ (bytearray(range(2)), bytes),
+ (memoryview(b"blob"), bytes),
+ (None, type(None)),
+ ]
+ for val, _ in dataset:
+ cur = self.con.execute("select test_params(?)", (val,))
+ cur.fetchone()
+ self.assertEqual(dataset, results)
+
# Regarding deterministic functions:
#
# Between 3.8.3 and 3.15.0, deterministic functions were only used to
@@ -320,6 +343,22 @@ def test_func_deterministic_keyword_only(self):
with self.assertRaises(TypeError):
self.con.create_function("deterministic", 0, int, True)
+ def test_function_destructor_via_gc(self):
+ # See bpo-44304: The destructor of the user function can
+ # crash if is called without the GIL from the gc functions
+ dest = sqlite.connect(':memory:')
+ def md5sum(t):
+ return
+
+ dest.create_function("md5", 1, md5sum)
+ x = dest("create table lang (name, first_appeared)")
+ del md5sum, dest
+
+ y = [x]
+ y.append(y)
+
+ del x,y
+ gc_collect()
class AggregateTests(unittest.TestCase):
def setUp(self):
@@ -345,6 +384,7 @@ def setUp(self):
self.con.create_aggregate("checkType", 2, AggrCheckType)
self.con.create_aggregate("checkTypes", -1, AggrCheckTypes)
self.con.create_aggregate("mysum", 1, AggrSum)
+ self.con.create_aggregate("aggtxt", 1, AggrText)
def tearDown(self):
#self.cur.close()
@@ -391,9 +431,9 @@ def test_aggr_exception_in_finalize(self):
def test_aggr_check_param_str(self):
cur = self.con.cursor()
- cur.execute("select checkType('str', ?)", ("foo",))
+ cur.execute("select checkTypes('str', ?, ?)", ("foo", str()))
val = cur.fetchone()[0]
- self.assertEqual(val, 1)
+ self.assertEqual(val, 2)
def test_aggr_check_param_int(self):
cur = self.con.cursor()
@@ -438,6 +478,15 @@ def test_aggr_no_match(self):
val = cur.fetchone()[0]
self.assertIsNone(val)
+ def test_aggr_text(self):
+ cur = self.con.cursor()
+ for txt in ["foo", "1\x002"]:
+ with self.subTest(txt=txt):
+ cur.execute("select aggtxt(?) from test", (txt,))
+ val = cur.fetchone()[0]
+ self.assertEqual(val, txt)
+
+
class AuthorizerTests(unittest.TestCase):
@staticmethod
def authorizer_cb(action, arg1, arg2, dbname, source):
diff --git a/Lib/sre_compile.py b/Lib/sre_compile.py
index c6398bfb83a576..aed752d11d2e5e 100644
--- a/Lib/sre_compile.py
+++ b/Lib/sre_compile.py
@@ -52,6 +52,22 @@
(0x3c2, 0x3c3), # ςσ
# GREEK SMALL LETTER PHI, GREEK PHI SYMBOL
(0x3c6, 0x3d5), # φϕ
+ # CYRILLIC SMALL LETTER VE, CYRILLIC SMALL LETTER ROUNDED VE
+ (0x432, 0x1c80), # вᲀ
+ # CYRILLIC SMALL LETTER DE, CYRILLIC SMALL LETTER LONG-LEGGED DE
+ (0x434, 0x1c81), # дᲁ
+ # CYRILLIC SMALL LETTER O, CYRILLIC SMALL LETTER NARROW O
+ (0x43e, 0x1c82), # оᲂ
+ # CYRILLIC SMALL LETTER ES, CYRILLIC SMALL LETTER WIDE ES
+ (0x441, 0x1c83), # сᲃ
+ # CYRILLIC SMALL LETTER TE, CYRILLIC SMALL LETTER TALL TE, CYRILLIC SMALL LETTER THREE-LEGGED TE
+ (0x442, 0x1c84, 0x1c85), # тᲄᲅ
+ # CYRILLIC SMALL LETTER HARD SIGN, CYRILLIC SMALL LETTER TALL HARD SIGN
+ (0x44a, 0x1c86), # ъᲆ
+ # CYRILLIC SMALL LETTER YAT, CYRILLIC SMALL LETTER TALL YAT
+ (0x463, 0x1c87), # ѣᲇ
+ # CYRILLIC SMALL LETTER UNBLENDED UK, CYRILLIC SMALL LETTER MONOGRAPH UK
+ (0x1c88, 0xa64b), # ᲈꙋ
# LATIN SMALL LETTER S WITH DOT ABOVE, LATIN SMALL LETTER LONG S WITH DOT ABOVE
(0x1e61, 0x1e9b), # ṡẛ
# LATIN SMALL LIGATURE LONG S T, LATIN SMALL LIGATURE ST
@@ -320,11 +336,19 @@ def _optimize_charset(charset, iscased=None, fixup=None, fixes=None):
charmap += b'\0' * 0xff00
continue
# Character set contains non-BMP character codes.
+ # For range, all BMP characters in the range are already
+ # proceeded.
if fixup:
hascased = True
- # There are only two ranges of cased non-BMP characters:
- # 10400-1044F (Deseret) and 118A0-118DF (Warang Citi),
- # and for both ranges RANGE_UNI_IGNORE works.
+ # For now, IN_UNI_IGNORE+LITERAL and
+ # IN_UNI_IGNORE+RANGE_UNI_IGNORE work for all non-BMP
+ # characters, because two characters (at least one of
+ # which is not in the BMP) match case-insensitively
+ # if and only if:
+ # 1) c1.lower() == c2.lower()
+ # 2) c1.lower() == c2 or c1.lower().upper() == c2
+ # Also, both c.lower() and c.lower().upper() are single
+ # characters for every non-BMP character.
if op is RANGE:
op = RANGE_UNI_IGNORE
tail.append((op, av))
diff --git a/Lib/sre_constants.py b/Lib/sre_constants.py
index 8e613cb3fa5dcb..db3ca51e8306ad 100644
--- a/Lib/sre_constants.py
+++ b/Lib/sre_constants.py
@@ -62,6 +62,8 @@ def __new__(cls, value, name):
def __repr__(self):
return self.name
+ __reduce__ = None
+
MAXREPEAT = _NamedIntConstant(MAXREPEAT, 'MAXREPEAT')
def _makecodes(names):
diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py
index 83119168e6376e..20a602501191f6 100644
--- a/Lib/sre_parse.py
+++ b/Lib/sre_parse.py
@@ -78,6 +78,7 @@ def __init__(self):
self.groupdict = {}
self.groupwidths = [None] # group 0
self.lookbehindgroups = None
+ self.grouprefpos = {}
@property
def groups(self):
return len(self.groupwidths)
@@ -330,7 +331,7 @@ def _class_escape(source, escape):
charname = source.getuntil('}', 'character name')
try:
c = ord(unicodedata.lookup(charname))
- except KeyError:
+ except (KeyError, TypeError):
raise source.error("undefined character name %r" % charname,
len(charname) + len(r'\N{}'))
return LITERAL, c
@@ -390,7 +391,7 @@ def _escape(source, escape, state):
charname = source.getuntil('}', 'character name')
try:
c = ord(unicodedata.lookup(charname))
- except KeyError:
+ except (KeyError, TypeError):
raise source.error("undefined character name %r" % charname,
len(charname) + len(r'\N{}'))
return LITERAL, c
@@ -786,6 +787,10 @@ def _parse(source, state, verbose, nested, first=False):
if condgroup >= MAXGROUPS:
msg = "invalid group reference %d" % condgroup
raise source.error(msg, len(condname) + 1)
+ if condgroup not in state.grouprefpos:
+ state.grouprefpos[condgroup] = (
+ source.tell() - len(condname) - 1
+ )
state.checklookbehindgroup(condgroup, source)
item_yes = _parse(source, state, verbose, nested + 1)
if source.match("|"):
@@ -807,9 +812,11 @@ def _parse(source, state, verbose, nested, first=False):
if not first or subpattern:
import warnings
warnings.warn(
- 'Flags not at the start of the expression %r%s' % (
+ 'Flags not at the start of the expression %r%s'
+ ' but at position %d' % (
source.string[:20], # truncate long regexes
' (truncated)' if len(source.string) > 20 else '',
+ start,
),
DeprecationWarning, stacklevel=nested + 6
)
@@ -961,6 +968,11 @@ def parse(str, flags=0, state=None):
assert source.next == ")"
raise source.error("unbalanced parenthesis")
+ for g in p.state.grouprefpos:
+ if g >= p.state.groups:
+ msg = "invalid group reference %d" % g
+ raise error(msg, str, p.state.grouprefpos[g])
+
if flags & SRE_FLAG_DEBUG:
p.dump()
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 2b131de04306ac..f386fa7831528f 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -18,9 +18,10 @@
seconds past the Epoch (the time values
returned from time.time())
- fetch_server_certificate (HOST, PORT) -- fetch the certificate provided
- by the server running on HOST at port PORT. No
- validation of the certificate is performed.
+ get_server_certificate (addr, ssl_version, ca_certs, timeout) -- Retrieve the
+ certificate from the server at the specified
+ address and return it as a PEM-encoded string
+
Integer constants:
@@ -94,7 +95,6 @@
import os
from collections import namedtuple
from enum import Enum as _Enum, IntEnum as _IntEnum, IntFlag as _IntFlag
-from enum import _simple_enum, _test_simple_enum
import _ssl # if we can't import it, let the error propagate
@@ -156,8 +156,7 @@
_SSLv2_IF_EXISTS = getattr(_SSLMethod, 'PROTOCOL_SSLv2', None)
-@_simple_enum(_IntEnum)
-class TLSVersion:
+class TLSVersion(_IntEnum):
MINIMUM_SUPPORTED = _ssl.PROTO_MINIMUM_SUPPORTED
SSLv3 = _ssl.PROTO_SSLv3
TLSv1 = _ssl.PROTO_TLSv1
@@ -167,8 +166,7 @@ class TLSVersion:
MAXIMUM_SUPPORTED = _ssl.PROTO_MAXIMUM_SUPPORTED
-@_simple_enum(_IntEnum)
-class _TLSContentType:
+class _TLSContentType(_IntEnum):
"""Content types (record layer)
See RFC 8446, section B.1
@@ -182,8 +180,7 @@ class _TLSContentType:
INNER_CONTENT_TYPE = 0x101
-@_simple_enum(_IntEnum)
-class _TLSAlertType:
+class _TLSAlertType(_IntEnum):
"""Alert types for TLSContentType.ALERT messages
See RFC 8466, section B.2
@@ -224,8 +221,7 @@ class _TLSAlertType:
NO_APPLICATION_PROTOCOL = 120
-@_simple_enum(_IntEnum)
-class _TLSMessageType:
+class _TLSMessageType(_IntEnum):
"""Message types (handshake protocol)
See RFC 8446, section B.3
@@ -387,7 +383,7 @@ def match_hostname(cert, hostname):
returns nothing.
"""
warnings.warn(
- "ssl module: match_hostname() is deprecated",
+ "ssl.match_hostname() is deprecated",
category=DeprecationWarning,
stacklevel=2
)
@@ -492,8 +488,7 @@ class SSLContext(_SSLContext):
def __new__(cls, protocol=None, *args, **kwargs):
if protocol is None:
warnings.warn(
- "ssl module: "
- "SSLContext() without protocol argument is deprecated.",
+ "ssl.SSLContext() without protocol argument is deprecated.",
category=DeprecationWarning,
stacklevel=2
)
@@ -536,7 +531,11 @@ def wrap_bio(self, incoming, outgoing, server_side=False,
)
def set_npn_protocols(self, npn_protocols):
- warnings.warn("NPN is deprecated, use ALPN instead", stacklevel=2)
+ warnings.warn(
+ "ssl NPN is deprecated, use ALPN instead",
+ DeprecationWarning,
+ stacklevel=2
+ )
protos = bytearray()
for protocol in npn_protocols:
b = bytes(protocol, 'ascii')
@@ -940,7 +939,9 @@ def selected_npn_protocol(self):
if a next protocol was not negotiated or if NPN is not supported by one
of the peers."""
warnings.warn(
- "ssl module: NPN is deprecated, use ALPN instead", stacklevel=2
+ "ssl NPN is deprecated, use ALPN instead",
+ DeprecationWarning,
+ stacklevel=2
)
def selected_alpn_protocol(self):
@@ -1032,7 +1033,7 @@ def _create(cls, sock, server_side=False, do_handshake_on_connect=True,
)
self = cls.__new__(cls, **kwargs)
super(SSLSocket, self).__init__(**kwargs)
- self.settimeout(sock.gettimeout())
+ sock_timeout = sock.gettimeout()
sock.detach()
self._context = context
@@ -1051,9 +1052,42 @@ def _create(cls, sock, server_side=False, do_handshake_on_connect=True,
if e.errno != errno.ENOTCONN:
raise
connected = False
+ blocking = self.getblocking()
+ self.setblocking(False)
+ try:
+ # We are not connected so this is not supposed to block, but
+ # testing revealed otherwise on macOS and Windows so we do
+ # the non-blocking dance regardless. Our raise when any data
+ # is found means consuming the data is harmless.
+ notconn_pre_handshake_data = self.recv(1)
+ except OSError as e:
+ # EINVAL occurs for recv(1) on non-connected on unix sockets.
+ if e.errno not in (errno.ENOTCONN, errno.EINVAL):
+ raise
+ notconn_pre_handshake_data = b''
+ self.setblocking(blocking)
+ if notconn_pre_handshake_data:
+ # This prevents pending data sent to the socket before it was
+ # closed from escaping to the caller who could otherwise
+ # presume it came through a successful TLS connection.
+ reason = "Closed before TLS handshake with data in recv buffer."
+ notconn_pre_handshake_data_error = SSLError(e.errno, reason)
+ # Add the SSLError attributes that _ssl.c always adds.
+ notconn_pre_handshake_data_error.reason = reason
+ notconn_pre_handshake_data_error.library = None
+ try:
+ self.close()
+ except OSError:
+ pass
+ try:
+ raise notconn_pre_handshake_data_error
+ finally:
+ # Explicitly break the reference cycle.
+ notconn_pre_handshake_data_error = None
else:
connected = True
+ self.settimeout(sock_timeout) # Must come after setblocking() calls.
self._connected = connected
if connected:
# create the SSL object
@@ -1157,7 +1191,9 @@ def getpeercert(self, binary_form=False):
def selected_npn_protocol(self):
self._checkClosed()
warnings.warn(
- "ssl module: NPN is deprecated, use ALPN instead", stacklevel=2
+ "ssl NPN is deprecated, use ALPN instead",
+ DeprecationWarning,
+ stacklevel=2
)
return None
@@ -1419,7 +1455,7 @@ def wrap_socket(sock, keyfile=None, certfile=None,
suppress_ragged_eofs=True,
ciphers=None):
warnings.warn(
- "ssl module: wrap_socket is deprecated, use SSLContext.wrap_socket()",
+ "ssl.wrap_socket() is deprecated, use SSLContext.wrap_socket()",
category=DeprecationWarning,
stacklevel=2
)
diff --git a/Lib/statistics.py b/Lib/statistics.py
index 673a162b3a79fc..52f17851591beb 100644
--- a/Lib/statistics.py
+++ b/Lib/statistics.py
@@ -94,7 +94,7 @@
>>> correlation(x, y) #doctest: +ELLIPSIS
0.31622776601...
>>> linear_regression(x, y) #doctest:
-LinearRegression(intercept=1.5, slope=0.1)
+LinearRegression(slope=0.1, intercept=1.5)
Exceptions
@@ -107,9 +107,12 @@
__all__ = [
'NormalDist',
'StatisticsError',
+ 'correlation',
+ 'covariance',
'fmean',
'geometric_mean',
'harmonic_mean',
+ 'linear_regression',
'mean',
'median',
'median_grouped',
@@ -122,9 +125,6 @@
'quantiles',
'stdev',
'variance',
- 'correlation',
- 'covariance',
- 'linear_regression',
]
import math
@@ -147,21 +147,17 @@ class StatisticsError(ValueError):
# === Private utilities ===
-def _sum(data, start=0):
- """_sum(data [, start]) -> (type, sum, count)
+def _sum(data):
+ """_sum(data) -> (type, sum, count)
Return a high-precision sum of the given numeric data as a fraction,
together with the type to be converted to and the count of items.
- If optional argument ``start`` is given, it is added to the total.
- If ``data`` is empty, ``start`` (defaulting to 0) is returned.
-
-
Examples
--------
- >>> _sum([3, 2.25, 4.5, -0.5, 1.0], 0.75)
- (, Fraction(11, 1), 5)
+ >>> _sum([3, 2.25, 4.5, -0.5, 0.25])
+ (, Fraction(19, 2), 5)
Some sources of round-off error will be avoided:
@@ -184,10 +180,9 @@ def _sum(data, start=0):
allowed.
"""
count = 0
- n, d = _exact_ratio(start)
- partials = {d: n}
+ partials = {}
partials_get = partials.get
- T = _coerce(int, type(start))
+ T = int
for typ, values in groupby(data, type):
T = _coerce(T, typ) # or raise TypeError
for n, d in map(_exact_ratio, values):
@@ -200,8 +195,7 @@ def _sum(data, start=0):
assert not _isfinite(total)
else:
# Sum all the partial sums using builtin sum.
- # FIXME is this faster if we sum them in order of the denominator?
- total = sum(Fraction(n, d) for d, n in sorted(partials.items()))
+ total = sum(Fraction(n, d) for d, n in partials.items())
return (T, total, count)
@@ -252,27 +246,19 @@ def _exact_ratio(x):
x is expected to be an int, Fraction, Decimal or float.
"""
try:
- # Optimise the common case of floats. We expect that the most often
- # used numeric type will be builtin floats, so try to make this as
- # fast as possible.
- if type(x) is float or type(x) is Decimal:
- return x.as_integer_ratio()
- try:
- # x may be an int, Fraction, or Integral ABC.
- return (x.numerator, x.denominator)
- except AttributeError:
- try:
- # x may be a float or Decimal subclass.
- return x.as_integer_ratio()
- except AttributeError:
- # Just give up?
- pass
+ return x.as_integer_ratio()
+ except AttributeError:
+ pass
except (OverflowError, ValueError):
# float NAN or INF.
assert not _isfinite(x)
return (x, None)
- msg = "can't convert type '{}' to numerator/denominator"
- raise TypeError(msg.format(type(x).__name__))
+ try:
+ # x may be an Integral ABC.
+ return (x.numerator, x.denominator)
+ except AttributeError:
+ msg = f"can't convert type '{type(x).__name__}' to numerator/denominator"
+ raise TypeError(msg)
def _convert(value, T):
@@ -388,7 +374,7 @@ def geometric_mean(data):
return exp(fmean(map(log, data)))
except ValueError:
raise StatisticsError('geometric mean requires a non-empty dataset '
- ' containing positive numbers') from None
+ 'containing positive numbers') from None
def harmonic_mean(data, weights=None):
@@ -719,14 +705,20 @@ def _ss(data, c=None):
if c is not None:
T, total, count = _sum((x-c)**2 for x in data)
return (T, total)
- c = mean(data)
- T, total, count = _sum((x-c)**2 for x in data)
- # The following sum should mathematically equal zero, but due to rounding
- # error may not.
- U, total2, count2 = _sum((x - c) for x in data)
- assert T == U and count == count2
- total -= total2 ** 2 / len(data)
- assert not total < 0, 'negative sum of square deviations: %f' % total
+ T, total, count = _sum(data)
+ mean_n, mean_d = (total / count).as_integer_ratio()
+ partials = Counter()
+ for n, d in map(_exact_ratio, data):
+ diff_n = n * mean_d - d * mean_n
+ diff_d = d * mean_d
+ partials[diff_d * diff_d] += diff_n * diff_n
+ if None in partials:
+ # The sum will be a NAN or INF. We can ignore all the finite
+ # partials, and just look at this special one.
+ total = partials[None]
+ assert not _isfinite(total)
+ else:
+ total = sum(Fraction(n, d) for d, n in partials.items())
return (T, total)
@@ -830,6 +822,9 @@ def stdev(data, xbar=None):
1.0810874155219827
"""
+ # Fixme: Despite the exact sum of squared deviations, some inaccuracy
+ # remain because there are two rounding steps. The first occurs in
+ # the _convert() step for variance(), the second occurs in math.sqrt().
var = variance(data, xbar)
try:
return var.sqrt()
@@ -846,6 +841,9 @@ def pstdev(data, mu=None):
0.986893273527251
"""
+ # Fixme: Despite the exact sum of squared deviations, some inaccuracy
+ # remain because there are two rounding steps. The first occurs in
+ # the _convert() step for pvariance(), the second occurs in math.sqrt().
var = pvariance(data, mu)
try:
return var.sqrt()
@@ -882,10 +880,10 @@ def covariance(x, y, /):
raise StatisticsError('covariance requires that both inputs have same number of data points')
if n < 2:
raise StatisticsError('covariance requires at least two data points')
- xbar = mean(x)
- ybar = mean(y)
- total = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y))
- return total / (n - 1)
+ xbar = fsum(x) / n
+ ybar = fsum(y) / n
+ sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y))
+ return sxy / (n - 1)
def correlation(x, y, /):
@@ -910,54 +908,60 @@ def correlation(x, y, /):
raise StatisticsError('correlation requires that both inputs have same number of data points')
if n < 2:
raise StatisticsError('correlation requires at least two data points')
- cov = covariance(x, y)
- stdx = stdev(x)
- stdy = stdev(y)
+ xbar = fsum(x) / n
+ ybar = fsum(y) / n
+ sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y))
+ sxx = fsum((xi - xbar) ** 2.0 for xi in x)
+ syy = fsum((yi - ybar) ** 2.0 for yi in y)
try:
- return cov / (stdx * stdy)
+ return sxy / sqrt(sxx * syy)
except ZeroDivisionError:
raise StatisticsError('at least one of the inputs is constant')
-LinearRegression = namedtuple('LinearRegression', ['intercept', 'slope'])
+LinearRegression = namedtuple('LinearRegression', ('slope', 'intercept'))
-def linear_regression(regressor, dependent_variable, /):
- """Intercept and slope for simple linear regression
+def linear_regression(x, y, /):
+ """Slope and intercept for simple linear regression.
- Return the intercept and slope of simple linear regression
+ Return the slope and intercept of simple linear regression
parameters estimated using ordinary least squares. Simple linear
- regression describes relationship between *regressor* and
- *dependent variable* in terms of linear function::
+ regression describes relationship between an independent variable
+ *x* and a dependent variable *y* in terms of linear function:
- dependent_variable = intercept + slope * regressor + noise
+ y = slope * x + intercept + noise
- where ``intercept`` and ``slope`` are the regression parameters that are
- estimated, and noise term is an unobserved random variable, for the
- variability of the data that was not explained by the linear regression
- (it is equal to the difference between prediction and the actual values
- of dependent variable).
+ where *slope* and *intercept* are the regression parameters that are
+ estimated, and noise represents the variability of the data that was
+ not explained by the linear regression (it is equal to the
+ difference between predicted and actual values of the dependent
+ variable).
The parameters are returned as a named tuple.
- >>> regressor = [1, 2, 3, 4, 5]
+ >>> x = [1, 2, 3, 4, 5]
>>> noise = NormalDist().samples(5, seed=42)
- >>> dependent_variable = [2 + 3 * regressor[i] + noise[i] for i in range(5)]
- >>> linear_regression(regressor, dependent_variable) #doctest: +ELLIPSIS
- LinearRegression(intercept=1.75684970486..., slope=3.09078914170...)
+ >>> y = [3 * x[i] + 2 + noise[i] for i in range(5)]
+ >>> linear_regression(x, y) #doctest: +ELLIPSIS
+ LinearRegression(slope=3.09078914170..., intercept=1.75684970486...)
"""
- n = len(regressor)
- if len(dependent_variable) != n:
+ n = len(x)
+ if len(y) != n:
raise StatisticsError('linear regression requires that both inputs have same number of data points')
if n < 2:
raise StatisticsError('linear regression requires at least two data points')
+ xbar = fsum(x) / n
+ ybar = fsum(y) / n
+ sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y))
+ sxx = fsum((xi - xbar) ** 2.0 for xi in x)
try:
- slope = covariance(regressor, dependent_variable) / variance(regressor)
+ slope = sxy / sxx # equivalent to: covariance(x, y) / variance(x)
except ZeroDivisionError:
- raise StatisticsError('regressor is constant')
- intercept = mean(dependent_variable) - slope * mean(regressor)
- return LinearRegression(intercept=intercept, slope=slope)
+ raise StatisticsError('x is constant')
+ intercept = ybar - slope * xbar
+ return LinearRegression(slope=slope, intercept=intercept)
## Normal Distribution #####################################################
@@ -1261,3 +1265,9 @@ def __hash__(self):
def __repr__(self):
return f'{type(self).__name__}(mu={self._mu!r}, sigma={self._sigma!r})'
+
+ def __getstate__(self):
+ return self._mu, self._sigma
+
+ def __setstate__(self, state):
+ self._mu, self._sigma = state
diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index 8203aded7d5529..f1e3d64dfe0419 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -411,7 +411,8 @@ def check_output(*popenargs, timeout=None, **kwargs):
if 'input' in kwargs and kwargs['input'] is None:
# Explicitly passing input=None was previously equivalent to passing an
# empty string. That is maintained here for backwards compatibility.
- if kwargs.get('universal_newlines') or kwargs.get('text'):
+ if kwargs.get('universal_newlines') or kwargs.get('text') or kwargs.get('encoding') \
+ or kwargs.get('errors'):
empty = ''
else:
empty = b''
@@ -463,7 +464,8 @@ def run(*popenargs,
The returned instance will have attributes args, returncode, stdout and
stderr. By default, stdout and stderr are not captured, and those attributes
- will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
+ will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
+ or pass capture_output=True to capture both.
If check is True and the exit code was non-zero, it raises a
CalledProcessError. The CalledProcessError object will have the return code
@@ -660,8 +662,9 @@ def _use_posix_spawn():
# os.posix_spawn() is not available
return False
- if sys.platform == 'darwin':
- # posix_spawn() is a syscall on macOS and properly reports errors
+ if sys.platform in ('darwin', 'sunos5'):
+ # posix_spawn() is a syscall on both macOS and Solaris,
+ # and properly reports errors
return True
# Check libc name and runtime libc version
@@ -690,7 +693,10 @@ def _use_posix_spawn():
return False
+# These are primarily fail-safe knobs for negatives. A True value does not
+# guarantee the given libc/syscall API will be used.
_USE_POSIX_SPAWN = _use_posix_spawn()
+_USE_VFORK = True
class Popen:
@@ -1421,7 +1427,23 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
if shell:
startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = _winapi.SW_HIDE
- comspec = os.environ.get("COMSPEC", "cmd.exe")
+ if not executable:
+ # gh-101283: without a fully-qualified path, before Windows
+ # checks the system directories, it first looks in the
+ # application directory, and also the current directory if
+ # NeedCurrentDirectoryForExePathW(ExeName) is true, so try
+ # to avoid executing unqualified "cmd.exe".
+ comspec = os.environ.get('ComSpec')
+ if not comspec:
+ system_root = os.environ.get('SystemRoot', '')
+ comspec = os.path.join(system_root, 'System32', 'cmd.exe')
+ if not os.path.isabs(comspec):
+ raise FileNotFoundError('shell not found: neither %ComSpec% nor %SystemRoot% is set')
+ if os.path.isabs(comspec):
+ executable = comspec
+ else:
+ comspec = executable
+
args = '{} /c "{}"'.format (comspec, args)
if cwd is not None:
diff --git a/Lib/symtable.py b/Lib/symtable.py
index 98db1e2557d37f..e11e5fffc4e1be 100644
--- a/Lib/symtable.py
+++ b/Lib/symtable.py
@@ -111,7 +111,7 @@ def has_children(self):
return bool(self._table.children)
def get_identifiers(self):
- """Return a list of names of symbols in the table.
+ """Return a view object containing the names of symbols in the table.
"""
return self._table.symbols.keys()
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index e8869af0b5cc94..daf9f000060a35 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -101,7 +101,7 @@ def joinuser(*args):
'stdlib': '{userbase}/{platlibdir}/python{py_version_short}',
'platstdlib': '{userbase}/{platlibdir}/python{py_version_short}',
'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
- 'platlib': '{userbase}/{platlibdir}/python{py_version_short}/site-packages',
+ 'platlib': '{userbase}/lib/python{py_version_short}/site-packages',
'include': '{userbase}/include/python{py_version_short}',
'scripts': '{userbase}/bin',
'data': '{userbase}',
@@ -111,7 +111,7 @@ def joinuser(*args):
'platstdlib': '{userbase}/lib/python',
'purelib': '{userbase}/lib/python/site-packages',
'platlib': '{userbase}/lib/python/site-packages',
- 'include': '{userbase}/include',
+ 'include': '{userbase}/include/python{py_version_short}',
'scripts': '{userbase}/bin',
'data': '{userbase}',
},
@@ -182,6 +182,18 @@ def is_python_build(check_home=False):
_PYTHON_BUILD = is_python_build(True)
+if _PYTHON_BUILD:
+ for scheme in ('posix_prefix', 'posix_home'):
+ # On POSIX-y platforms, Python will:
+ # - Build from .h files in 'headers' (which is only added to the
+ # scheme when building CPython)
+ # - Install .h files to 'include'
+ scheme = _INSTALL_SCHEMES[scheme]
+ scheme['headers'] = scheme['include']
+ scheme['include'] = '{srcdir}/Include'
+ scheme['platinclude'] = '{projectbase}/.'
+
+
def _subst_vars(s, local_vars):
try:
return s.format(**local_vars)
diff --git a/Lib/tabnanny.py b/Lib/tabnanny.py
index 7973f26f98b8b2..a47f5a96b89722 100755
--- a/Lib/tabnanny.py
+++ b/Lib/tabnanny.py
@@ -23,8 +23,6 @@
import os
import sys
import tokenize
-if not hasattr(tokenize, 'NL'):
- raise ValueError("tokenize.NL doesn't exist -- tokenize module too old")
__all__ = ["check", "NannyNag", "process_tokens"]
diff --git a/Lib/tarfile.py b/Lib/tarfile.py
index 18d415adf54418..3ab6811d63335b 100755
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -46,6 +46,7 @@
import struct
import copy
import re
+import warnings
try:
import pwd
@@ -71,6 +72,7 @@
"ENCODING", "USTAR_FORMAT", "GNU_FORMAT", "PAX_FORMAT",
"DEFAULT_FORMAT", "open"]
+
#---------------------------------------------------------
# tar constants
#---------------------------------------------------------
@@ -158,6 +160,8 @@
def stn(s, length, encoding, errors):
"""Convert a string to a null-terminated bytes object.
"""
+ if s is None:
+ raise ValueError("metadata cannot contain None")
s = s.encode(encoding, errors)
return s[:length] + (length - len(s)) * NUL
@@ -709,9 +713,137 @@ def __init__(self, tarfile, tarinfo):
super().__init__(fileobj)
#class ExFileObject
+
+#-----------------------------
+# extraction filters (PEP 706)
+#-----------------------------
+
+class FilterError(TarError):
+ pass
+
+class AbsolutePathError(FilterError):
+ def __init__(self, tarinfo):
+ self.tarinfo = tarinfo
+ super().__init__(f'member {tarinfo.name!r} has an absolute path')
+
+class OutsideDestinationError(FilterError):
+ def __init__(self, tarinfo, path):
+ self.tarinfo = tarinfo
+ self._path = path
+ super().__init__(f'{tarinfo.name!r} would be extracted to {path!r}, '
+ + 'which is outside the destination')
+
+class SpecialFileError(FilterError):
+ def __init__(self, tarinfo):
+ self.tarinfo = tarinfo
+ super().__init__(f'{tarinfo.name!r} is a special file')
+
+class AbsoluteLinkError(FilterError):
+ def __init__(self, tarinfo):
+ self.tarinfo = tarinfo
+ super().__init__(f'{tarinfo.name!r} is a link to an absolute path')
+
+class LinkOutsideDestinationError(FilterError):
+ def __init__(self, tarinfo, path):
+ self.tarinfo = tarinfo
+ self._path = path
+ super().__init__(f'{tarinfo.name!r} would link to {path!r}, '
+ + 'which is outside the destination')
+
+def _get_filtered_attrs(member, dest_path, for_data=True):
+ new_attrs = {}
+ name = member.name
+ dest_path = os.path.realpath(dest_path)
+ # Strip leading / (tar's directory separator) from filenames.
+ # Include os.sep (target OS directory separator) as well.
+ if name.startswith(('/', os.sep)):
+ name = new_attrs['name'] = member.path.lstrip('/' + os.sep)
+ if os.path.isabs(name):
+ # Path is absolute even after stripping.
+ # For example, 'C:/foo' on Windows.
+ raise AbsolutePathError(member)
+ # Ensure we stay in the destination
+ target_path = os.path.realpath(os.path.join(dest_path, name))
+ if os.path.commonpath([target_path, dest_path]) != dest_path:
+ raise OutsideDestinationError(member, target_path)
+ # Limit permissions (no high bits, and go-w)
+ mode = member.mode
+ if mode is not None:
+ # Strip high bits & group/other write bits
+ mode = mode & 0o755
+ if for_data:
+ # For data, handle permissions & file types
+ if member.isreg() or member.islnk():
+ if not mode & 0o100:
+ # Clear executable bits if not executable by user
+ mode &= ~0o111
+ # Ensure owner can read & write
+ mode |= 0o600
+ elif member.isdir() or member.issym():
+ # Ignore mode for directories & symlinks
+ mode = None
+ else:
+ # Reject special files
+ raise SpecialFileError(member)
+ if mode != member.mode:
+ new_attrs['mode'] = mode
+ if for_data:
+ # Ignore ownership for 'data'
+ if member.uid is not None:
+ new_attrs['uid'] = None
+ if member.gid is not None:
+ new_attrs['gid'] = None
+ if member.uname is not None:
+ new_attrs['uname'] = None
+ if member.gname is not None:
+ new_attrs['gname'] = None
+ # Check link destination for 'data'
+ if member.islnk() or member.issym():
+ if os.path.isabs(member.linkname):
+ raise AbsoluteLinkError(member)
+ if member.issym():
+ target_path = os.path.join(dest_path,
+ os.path.dirname(name),
+ member.linkname)
+ else:
+ target_path = os.path.join(dest_path,
+ member.linkname)
+ target_path = os.path.realpath(target_path)
+ if os.path.commonpath([target_path, dest_path]) != dest_path:
+ raise LinkOutsideDestinationError(member, target_path)
+ return new_attrs
+
+def fully_trusted_filter(member, dest_path):
+ return member
+
+def tar_filter(member, dest_path):
+ new_attrs = _get_filtered_attrs(member, dest_path, False)
+ if new_attrs:
+ return member.replace(**new_attrs, deep=False)
+ return member
+
+def data_filter(member, dest_path):
+ new_attrs = _get_filtered_attrs(member, dest_path, True)
+ if new_attrs:
+ return member.replace(**new_attrs, deep=False)
+ return member
+
+_NAMED_FILTERS = {
+ "fully_trusted": fully_trusted_filter,
+ "tar": tar_filter,
+ "data": data_filter,
+}
+
#------------------
# Exported Classes
#------------------
+
+# Sentinel for replace() defaults, meaning "don't change the attribute"
+_KEEP = object()
+
+# Header length is digits followed by a space.
+_header_length_prefix_re = re.compile(br"([0-9]{1,20}) ")
+
class TarInfo(object):
"""Informational class which holds the details about an
archive member given by a tar header block.
@@ -792,12 +924,44 @@ def linkpath(self, linkname):
def __repr__(self):
return "<%s %r at %#x>" % (self.__class__.__name__,self.name,id(self))
+ def replace(self, *,
+ name=_KEEP, mtime=_KEEP, mode=_KEEP, linkname=_KEEP,
+ uid=_KEEP, gid=_KEEP, uname=_KEEP, gname=_KEEP,
+ deep=True, _KEEP=_KEEP):
+ """Return a deep copy of self with the given attributes replaced.
+ """
+ if deep:
+ result = copy.deepcopy(self)
+ else:
+ result = copy.copy(self)
+ if name is not _KEEP:
+ result.name = name
+ if mtime is not _KEEP:
+ result.mtime = mtime
+ if mode is not _KEEP:
+ result.mode = mode
+ if linkname is not _KEEP:
+ result.linkname = linkname
+ if uid is not _KEEP:
+ result.uid = uid
+ if gid is not _KEEP:
+ result.gid = gid
+ if uname is not _KEEP:
+ result.uname = uname
+ if gname is not _KEEP:
+ result.gname = gname
+ return result
+
def get_info(self):
"""Return the TarInfo's attributes as a dictionary.
"""
+ if self.mode is None:
+ mode = None
+ else:
+ mode = self.mode & 0o7777
info = {
"name": self.name,
- "mode": self.mode & 0o7777,
+ "mode": mode,
"uid": self.uid,
"gid": self.gid,
"size": self.size,
@@ -820,6 +984,9 @@ def tobuf(self, format=DEFAULT_FORMAT, encoding=ENCODING, errors="surrogateescap
"""Return a tar header as a string of 512 byte blocks.
"""
info = self.get_info()
+ for name, value in info.items():
+ if value is None:
+ raise ValueError("%s may not be None" % name)
if format == USTAR_FORMAT:
return self.create_ustar_header(info, encoding, errors)
@@ -888,15 +1055,24 @@ def create_pax_header(self, info, encoding):
# Test number fields for values that exceed the field limit or values
# that like to be stored as float.
for name, digits in (("uid", 8), ("gid", 8), ("size", 12), ("mtime", 12)):
- if name in pax_headers:
- # The pax header has priority. Avoid overflow.
- info[name] = 0
- continue
+ needs_pax = False
val = info[name]
- if not 0 <= val < 8 ** (digits - 1) or isinstance(val, float):
- pax_headers[name] = str(val)
+ val_is_float = isinstance(val, float)
+ val_int = round(val) if val_is_float else val
+ if not 0 <= val_int < 8 ** (digits - 1):
+ # Avoid overflow.
info[name] = 0
+ needs_pax = True
+ elif val_is_float:
+ # Put rounded value in ustar header, and full
+ # precision value in pax header.
+ info[name] = val_int
+ needs_pax = True
+
+ # The existing pax header has priority.
+ if needs_pax and name not in pax_headers:
+ pax_headers[name] = str(val)
# Create a pax extended header if necessary.
if pax_headers:
@@ -941,6 +1117,12 @@ def _create_header(info, format, encoding, errors):
devmajor = stn("", 8, encoding, errors)
devminor = stn("", 8, encoding, errors)
+ # None values in metadata should cause ValueError.
+ # itn()/stn() do this for all fields except type.
+ filetype = info.get("type", REGTYPE)
+ if filetype is None:
+ raise ValueError("TarInfo.type must not be None")
+
parts = [
stn(info.get("name", ""), 100, encoding, errors),
itn(info.get("mode", 0) & 0o7777, 8, format),
@@ -949,7 +1131,7 @@ def _create_header(info, format, encoding, errors):
itn(info.get("size", 0), 12, format),
itn(info.get("mtime", 0), 12, format),
b" ", # checksum field
- info.get("type", REGTYPE),
+ filetype,
stn(info.get("linkname", ""), 100, encoding, errors),
info.get("magic", POSIX_MAGIC),
stn(info.get("uname", ""), 32, encoding, errors),
@@ -1154,6 +1336,11 @@ def _proc_builtin(self, tarfile):
# header information.
self._apply_pax_info(tarfile.pax_headers, tarfile.encoding, tarfile.errors)
+ # Remove redundant slashes from directories. This is to be consistent
+ # with frombuf().
+ if self.isdir():
+ self.name = self.name.rstrip("/")
+
return self
def _proc_gnulong(self, tarfile):
@@ -1176,6 +1363,11 @@ def _proc_gnulong(self, tarfile):
elif self.type == GNUTYPE_LONGLINK:
next.linkname = nts(buf, tarfile.encoding, tarfile.errors)
+ # Remove redundant slashes from directories. This is to be consistent
+ # with frombuf().
+ if next.isdir():
+ next.name = next.name.removesuffix("/")
+
return next
def _proc_sparse(self, tarfile):
@@ -1221,41 +1413,59 @@ def _proc_pax(self, tarfile):
else:
pax_headers = tarfile.pax_headers.copy()
- # Check if the pax header contains a hdrcharset field. This tells us
- # the encoding of the path, linkpath, uname and gname fields. Normally,
- # these fields are UTF-8 encoded but since POSIX.1-2008 tar
- # implementations are allowed to store them as raw binary strings if
- # the translation to UTF-8 fails.
- match = re.search(br"\d+ hdrcharset=([^\n]+)\n", buf)
- if match is not None:
- pax_headers["hdrcharset"] = match.group(1).decode("utf-8")
-
- # For the time being, we don't care about anything other than "BINARY".
- # The only other value that is currently allowed by the standard is
- # "ISO-IR 10646 2000 UTF-8" in other words UTF-8.
- hdrcharset = pax_headers.get("hdrcharset")
- if hdrcharset == "BINARY":
- encoding = tarfile.encoding
- else:
- encoding = "utf-8"
-
# Parse pax header information. A record looks like that:
# "%d %s=%s\n" % (length, keyword, value). length is the size
# of the complete record including the length field itself and
- # the newline. keyword and value are both UTF-8 encoded strings.
- regex = re.compile(br"(\d+) ([^=]+)=")
+ # the newline.
pos = 0
- while True:
- match = regex.match(buf, pos)
- if not match:
- break
+ encoding = None
+ raw_headers = []
+ while len(buf) > pos and buf[pos] != 0x00:
+ if not (match := _header_length_prefix_re.match(buf, pos)):
+ raise InvalidHeaderError("invalid header")
+ try:
+ length = int(match.group(1))
+ except ValueError:
+ raise InvalidHeaderError("invalid header")
+ # Headers must be at least 5 bytes, shortest being '5 x=\n'.
+ # Value is allowed to be empty.
+ if length < 5:
+ raise InvalidHeaderError("invalid header")
+ if pos + length > len(buf):
+ raise InvalidHeaderError("invalid header")
+
+ header_value_end_offset = match.start(1) + length - 1 # Last byte of the header
+ keyword_and_value = buf[match.end(1) + 1:header_value_end_offset]
+ raw_keyword, equals, raw_value = keyword_and_value.partition(b"=")
- length, keyword = match.groups()
- length = int(length)
- if length == 0:
+ # Check the framing of the header. The last character must be '\n' (0x0A)
+ if not raw_keyword or equals != b"=" or buf[header_value_end_offset] != 0x0A:
raise InvalidHeaderError("invalid header")
- value = buf[match.end(2) + 1:match.start(1) + length - 1]
+ raw_headers.append((length, raw_keyword, raw_value))
+
+ # Check if the pax header contains a hdrcharset field. This tells us
+ # the encoding of the path, linkpath, uname and gname fields. Normally,
+ # these fields are UTF-8 encoded but since POSIX.1-2008 tar
+ # implementations are allowed to store them as raw binary strings if
+ # the translation to UTF-8 fails. For the time being, we don't care about
+ # anything other than "BINARY". The only other value that is currently
+ # allowed by the standard is "ISO-IR 10646 2000 UTF-8" in other words UTF-8.
+ # Note that we only follow the initial 'hdrcharset' setting to preserve
+ # the initial behavior of the 'tarfile' module.
+ if raw_keyword == b"hdrcharset" and encoding is None:
+ if raw_value == b"BINARY":
+ encoding = tarfile.encoding
+ else: # This branch ensures only the first 'hdrcharset' header is used.
+ encoding = "utf-8"
+ pos += length
+
+ # If no explicit hdrcharset is set, we use UTF-8 as a default.
+ if encoding is None:
+ encoding = "utf-8"
+
+ # After parsing the raw headers we can decode them to text.
+ for length, raw_keyword, raw_value in raw_headers:
# Normally, we could just use "utf-8" as the encoding and "strict"
# as the error handler, but we better not take the risk. For
# example, GNU tar <= 1.23 is known to store filenames it cannot
@@ -1263,17 +1473,16 @@ def _proc_pax(self, tarfile):
# hdrcharset=BINARY header).
# We first try the strict standard encoding, and if that fails we
# fall back on the user's encoding and error handler.
- keyword = self._decode_pax_field(keyword, "utf-8", "utf-8",
+ keyword = self._decode_pax_field(raw_keyword, "utf-8", "utf-8",
tarfile.errors)
if keyword in PAX_NAME_FIELDS:
- value = self._decode_pax_field(value, encoding, tarfile.encoding,
+ value = self._decode_pax_field(raw_value, encoding, tarfile.encoding,
tarfile.errors)
else:
- value = self._decode_pax_field(value, "utf-8", "utf-8",
+ value = self._decode_pax_field(raw_value, "utf-8", "utf-8",
tarfile.errors)
pax_headers[keyword] = value
- pos += length
# Fetch the next header.
try:
@@ -1288,7 +1497,7 @@ def _proc_pax(self, tarfile):
elif "GNU.sparse.size" in pax_headers:
# GNU extended sparse format version 0.0.
- self._proc_gnusparse_00(next, pax_headers, buf)
+ self._proc_gnusparse_00(next, raw_headers)
elif pax_headers.get("GNU.sparse.major") == "1" and pax_headers.get("GNU.sparse.minor") == "0":
# GNU extended sparse format version 1.0.
@@ -1310,15 +1519,24 @@ def _proc_pax(self, tarfile):
return next
- def _proc_gnusparse_00(self, next, pax_headers, buf):
+ def _proc_gnusparse_00(self, next, raw_headers):
"""Process a GNU tar extended sparse header, version 0.0.
"""
offsets = []
- for match in re.finditer(br"\d+ GNU.sparse.offset=(\d+)\n", buf):
- offsets.append(int(match.group(1)))
numbytes = []
- for match in re.finditer(br"\d+ GNU.sparse.numbytes=(\d+)\n", buf):
- numbytes.append(int(match.group(1)))
+ for _, keyword, value in raw_headers:
+ if keyword == b"GNU.sparse.offset":
+ try:
+ offsets.append(int(value.decode()))
+ except ValueError:
+ raise InvalidHeaderError("invalid header")
+
+ elif keyword == b"GNU.sparse.numbytes":
+ try:
+ numbytes.append(int(value.decode()))
+ except ValueError:
+ raise InvalidHeaderError("invalid header")
+
next.sparse = list(zip(offsets, numbytes))
def _proc_gnusparse_01(self, next, pax_headers):
@@ -1449,6 +1667,8 @@ class TarFile(object):
fileobject = ExFileObject # The file-object for extractfile().
+ extraction_filter = None # The default filter for extraction.
+
def __init__(self, name=None, mode="r", fileobj=None, format=None,
tarinfo=None, dereference=None, ignore_zeros=None, encoding=None,
errors="surrogateescape", pax_headers=None, debug=None,
@@ -1789,7 +2009,7 @@ def getmember(self, name):
than once in the archive, its last occurrence is assumed to be the
most up-to-date version.
"""
- tarinfo = self._getmember(name)
+ tarinfo = self._getmember(name.rstrip('/'))
if tarinfo is None:
raise KeyError("filename %r not found" % name)
return tarinfo
@@ -1921,7 +2141,10 @@ def list(self, verbose=True, *, members=None):
members = self
for tarinfo in members:
if verbose:
- _safe_print(stat.filemode(tarinfo.mode))
+ if tarinfo.mode is None:
+ _safe_print("??????????")
+ else:
+ _safe_print(stat.filemode(tarinfo.mode))
_safe_print("%s/%s" % (tarinfo.uname or tarinfo.uid,
tarinfo.gname or tarinfo.gid))
if tarinfo.ischr() or tarinfo.isblk():
@@ -1929,8 +2152,11 @@ def list(self, verbose=True, *, members=None):
("%d,%d" % (tarinfo.devmajor, tarinfo.devminor)))
else:
_safe_print("%10d" % tarinfo.size)
- _safe_print("%d-%02d-%02d %02d:%02d:%02d" \
- % time.localtime(tarinfo.mtime)[:6])
+ if tarinfo.mtime is None:
+ _safe_print("????-??-?? ??:??:??")
+ else:
+ _safe_print("%d-%02d-%02d %02d:%02d:%02d" \
+ % time.localtime(tarinfo.mtime)[:6])
_safe_print(tarinfo.name + ("/" if tarinfo.isdir() else ""))
@@ -2017,32 +2243,58 @@ def addfile(self, tarinfo, fileobj=None):
self.members.append(tarinfo)
- def extractall(self, path=".", members=None, *, numeric_owner=False):
+ def _get_filter_function(self, filter):
+ if filter is None:
+ filter = self.extraction_filter
+ if filter is None:
+ return fully_trusted_filter
+ if isinstance(filter, str):
+ raise TypeError(
+ 'String names are not supported for '
+ + 'TarFile.extraction_filter. Use a function such as '
+ + 'tarfile.data_filter directly.')
+ return filter
+ if callable(filter):
+ return filter
+ try:
+ return _NAMED_FILTERS[filter]
+ except KeyError:
+ raise ValueError(f"filter {filter!r} not found") from None
+
+ def extractall(self, path=".", members=None, *, numeric_owner=False,
+ filter=None):
"""Extract all members from the archive to the current working
directory and set owner, modification time and permissions on
directories afterwards. `path' specifies a different directory
to extract to. `members' is optional and must be a subset of the
list returned by getmembers(). If `numeric_owner` is True, only
the numbers for user/group names are used and not the names.
+
+ The `filter` function will be called on each member just
+ before extraction.
+ It can return a changed TarInfo or None to skip the member.
+ String names of common filters are accepted.
"""
directories = []
+ filter_function = self._get_filter_function(filter)
if members is None:
members = self
- for tarinfo in members:
+ for member in members:
+ tarinfo = self._get_extract_tarinfo(member, filter_function, path)
+ if tarinfo is None:
+ continue
if tarinfo.isdir():
- # Extract directories with a safe mode.
+ # For directories, delay setting attributes until later,
+ # since permissions can interfere with extraction and
+ # extracting contents can reset mtime.
directories.append(tarinfo)
- tarinfo = copy.copy(tarinfo)
- tarinfo.mode = 0o700
- # Do not set_attrs directories, as we will do that further down
- self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
- numeric_owner=numeric_owner)
+ self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
+ numeric_owner=numeric_owner)
# Reverse sort directories.
- directories.sort(key=lambda a: a.name)
- directories.reverse()
+ directories.sort(key=lambda a: a.name, reverse=True)
# Set correct owner, mtime and filemode on directories.
for tarinfo in directories:
@@ -2052,12 +2304,10 @@ def extractall(self, path=".", members=None, *, numeric_owner=False):
self.utime(tarinfo, dirpath)
self.chmod(tarinfo, dirpath)
except ExtractError as e:
- if self.errorlevel > 1:
- raise
- else:
- self._dbg(1, "tarfile: %s" % e)
+ self._handle_nonfatal_error(e)
- def extract(self, member, path="", set_attrs=True, *, numeric_owner=False):
+ def extract(self, member, path="", set_attrs=True, *, numeric_owner=False,
+ filter=None):
"""Extract a member from the archive to the current working directory,
using its full name. Its file information is extracted as accurately
as possible. `member' may be a filename or a TarInfo object. You can
@@ -2065,35 +2315,70 @@ def extract(self, member, path="", set_attrs=True, *, numeric_owner=False):
mtime, mode) are set unless `set_attrs' is False. If `numeric_owner`
is True, only the numbers for user/group names are used and not
the names.
+
+ The `filter` function will be called before extraction.
+ It can return a changed TarInfo or None to skip the member.
+ String names of common filters are accepted.
"""
- self._check("r")
+ filter_function = self._get_filter_function(filter)
+ tarinfo = self._get_extract_tarinfo(member, filter_function, path)
+ if tarinfo is not None:
+ self._extract_one(tarinfo, path, set_attrs, numeric_owner)
+ def _get_extract_tarinfo(self, member, filter_function, path):
+ """Get filtered TarInfo (or None) from member, which might be a str"""
if isinstance(member, str):
tarinfo = self.getmember(member)
else:
tarinfo = member
+ unfiltered = tarinfo
+ try:
+ tarinfo = filter_function(tarinfo, path)
+ except (OSError, FilterError) as e:
+ self._handle_fatal_error(e)
+ except ExtractError as e:
+ self._handle_nonfatal_error(e)
+ if tarinfo is None:
+ self._dbg(2, "tarfile: Excluded %r" % unfiltered.name)
+ return None
# Prepare the link target for makelink().
if tarinfo.islnk():
+ tarinfo = copy.copy(tarinfo)
tarinfo._link_target = os.path.join(path, tarinfo.linkname)
+ return tarinfo
+
+ def _extract_one(self, tarinfo, path, set_attrs, numeric_owner):
+ """Extract from filtered tarinfo to disk"""
+ self._check("r")
try:
self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
set_attrs=set_attrs,
numeric_owner=numeric_owner)
except OSError as e:
- if self.errorlevel > 0:
- raise
- else:
- if e.filename is None:
- self._dbg(1, "tarfile: %s" % e.strerror)
- else:
- self._dbg(1, "tarfile: %s %r" % (e.strerror, e.filename))
+ self._handle_fatal_error(e)
except ExtractError as e:
- if self.errorlevel > 1:
- raise
+ self._handle_nonfatal_error(e)
+
+ def _handle_nonfatal_error(self, e):
+ """Handle non-fatal error (ExtractError) according to errorlevel"""
+ if self.errorlevel > 1:
+ raise
+ else:
+ self._dbg(1, "tarfile: %s" % e)
+
+ def _handle_fatal_error(self, e):
+ """Handle "fatal" error according to self.errorlevel"""
+ if self.errorlevel > 0:
+ raise
+ elif isinstance(e, OSError):
+ if e.filename is None:
+ self._dbg(1, "tarfile: %s" % e.strerror)
else:
- self._dbg(1, "tarfile: %s" % e)
+ self._dbg(1, "tarfile: %s %r" % (e.strerror, e.filename))
+ else:
+ self._dbg(1, "tarfile: %s %s" % (type(e).__name__, e))
def extractfile(self, member):
"""Extract a member from the archive as a file object. `member' may be
@@ -2180,9 +2465,13 @@ def makedir(self, tarinfo, targetpath):
"""Make a directory called targetpath.
"""
try:
- # Use a safe mode for the directory, the real mode is set
- # later in _extract_member().
- os.mkdir(targetpath, 0o700)
+ if tarinfo.mode is None:
+ # Use the system's default mode
+ os.mkdir(targetpath)
+ else:
+ # Use a safe mode for the directory, the real mode is set
+ # later in _extract_member().
+ os.mkdir(targetpath, 0o700)
except FileExistsError:
pass
@@ -2225,6 +2514,9 @@ def makedev(self, tarinfo, targetpath):
raise ExtractError("special devices not supported by system")
mode = tarinfo.mode
+ if mode is None:
+ # Use mknod's default
+ mode = 0o600
if tarinfo.isblk():
mode |= stat.S_IFBLK
else:
@@ -2246,7 +2538,6 @@ def makelink(self, tarinfo, targetpath):
os.unlink(targetpath)
os.symlink(tarinfo.linkname, targetpath)
else:
- # See extract().
if os.path.exists(tarinfo._link_target):
os.link(tarinfo._link_target, targetpath)
else:
@@ -2271,15 +2562,19 @@ def chown(self, tarinfo, targetpath, numeric_owner):
u = tarinfo.uid
if not numeric_owner:
try:
- if grp:
+ if grp and tarinfo.gname:
g = grp.getgrnam(tarinfo.gname)[2]
except KeyError:
pass
try:
- if pwd:
+ if pwd and tarinfo.uname:
u = pwd.getpwnam(tarinfo.uname)[2]
except KeyError:
pass
+ if g is None:
+ g = -1
+ if u is None:
+ u = -1
try:
if tarinfo.issym() and hasattr(os, "lchown"):
os.lchown(targetpath, u, g)
@@ -2291,6 +2586,8 @@ def chown(self, tarinfo, targetpath, numeric_owner):
def chmod(self, tarinfo, targetpath):
"""Set file permissions of targetpath according to tarinfo.
"""
+ if tarinfo.mode is None:
+ return
try:
os.chmod(targetpath, tarinfo.mode)
except OSError as e:
@@ -2299,10 +2596,13 @@ def chmod(self, tarinfo, targetpath):
def utime(self, tarinfo, targetpath):
"""Set modification time of targetpath according to tarinfo.
"""
+ mtime = tarinfo.mtime
+ if mtime is None:
+ return
if not hasattr(os, 'utime'):
return
try:
- os.utime(targetpath, (tarinfo.mtime, tarinfo.mtime))
+ os.utime(targetpath, (mtime, mtime))
except OSError as e:
raise ExtractError("could not change modification time") from e
@@ -2349,6 +2649,15 @@ def next(self):
raise ReadError(str(e)) from None
except SubsequentHeaderError as e:
raise ReadError(str(e)) from None
+ except Exception as e:
+ try:
+ import zlib
+ if isinstance(e, zlib.error):
+ raise ReadError(f'zlib error: {e}') from None
+ else:
+ raise e
+ except ImportError:
+ raise e
break
if tarinfo is not None:
@@ -2369,13 +2678,26 @@ def _getmember(self, name, tarinfo=None, normalize=False):
members = self.getmembers()
# Limit the member search list up to tarinfo.
+ skipping = False
if tarinfo is not None:
- members = members[:members.index(tarinfo)]
+ try:
+ index = members.index(tarinfo)
+ except ValueError:
+ # The given starting point might be a (modified) copy.
+ # We'll later skip members until we find an equivalent.
+ skipping = True
+ else:
+ # Happy fast path
+ members = members[:index]
if normalize:
name = os.path.normpath(name)
for member in reversed(members):
+ if skipping:
+ if tarinfo.offset == member.offset:
+ skipping = False
+ continue
if normalize:
member_name = os.path.normpath(member.name)
else:
@@ -2384,6 +2706,10 @@ def _getmember(self, name, tarinfo=None, normalize=False):
if name == member_name:
return member
+ if skipping:
+ # Starting point was not found
+ raise ValueError(tarinfo)
+
def _load(self):
"""Read through the entire archive file and look for readable
members.
@@ -2476,6 +2802,7 @@ def __exit__(self, type, value, traceback):
#--------------------
# exported functions
#--------------------
+
def is_tarfile(name):
"""Return True if name points to a tar archive that we
are able to handle, else return False.
@@ -2502,6 +2829,10 @@ def main():
parser = argparse.ArgumentParser(description=description)
parser.add_argument('-v', '--verbose', action='store_true', default=False,
help='Verbose output')
+ parser.add_argument('--filter', metavar='',
+ choices=_NAMED_FILTERS,
+ help='Filter for extraction')
+
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('-l', '--list', metavar='',
help='Show listing of a tarfile')
@@ -2513,8 +2844,12 @@ def main():
help='Create tarfile from sources')
group.add_argument('-t', '--test', metavar='',
help='Test if a tarfile is valid')
+
args = parser.parse_args()
+ if args.filter and args.extract is None:
+ parser.exit(1, '--filter is only valid for extraction\n')
+
if args.test is not None:
src = args.test
if is_tarfile(src):
@@ -2545,7 +2880,7 @@ def main():
if is_tarfile(src):
with TarFile.open(src, 'r:*') as tf:
- tf.extractall(path=curdir)
+ tf.extractall(path=curdir, filter=args.filter)
if args.verbose:
if curdir == '.':
msg = '{!r} file is extracted.'.format(src)
diff --git a/Lib/tempfile.py b/Lib/tempfile.py
index efcf7a7fb3bbc1..fd78998df9fd85 100644
--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -88,6 +88,10 @@ def _infer_return_type(*args):
for arg in args:
if arg is None:
continue
+
+ if isinstance(arg, _os.PathLike):
+ arg = _os.fspath(arg)
+
if isinstance(arg, bytes):
if return_type is str:
raise TypeError("Can't mix bytes and non-bytes in "
@@ -199,8 +203,7 @@ def _get_default_tempdir():
fd = _os.open(filename, _bin_openflags, 0o600)
try:
try:
- with _io.open(fd, 'wb', closefd=False) as fp:
- fp.write(b'blat')
+ _os.write(fd, b'blat')
finally:
_os.close(fd)
finally:
@@ -240,6 +243,7 @@ def _get_candidate_names():
def _mkstemp_inner(dir, pre, suf, flags, output_type):
"""Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""
+ dir = _os.path.abspath(dir)
names = _get_candidate_names()
if output_type is bytes:
names = map(_os.fsencode, names)
@@ -260,11 +264,27 @@ def _mkstemp_inner(dir, pre, suf, flags, output_type):
continue
else:
raise
- return (fd, _os.path.abspath(file))
+ return fd, file
raise FileExistsError(_errno.EEXIST,
"No usable temporary file name found")
+def _dont_follow_symlinks(func, path, *args):
+ # Pass follow_symlinks=False, unless not supported on this platform.
+ if func in _os.supports_follow_symlinks:
+ func(path, *args, follow_symlinks=False)
+ elif _os.name == 'nt' or not _os.path.islink(path):
+ func(path, *args)
+
+def _resetperms(path):
+ try:
+ chflags = _os.chflags
+ except AttributeError:
+ pass
+ else:
+ _dont_follow_symlinks(chflags, path, 0)
+ _dont_follow_symlinks(_os.chmod, path, 0o700)
+
# User visible interfaces.
@@ -546,15 +566,26 @@ def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None,
if "b" not in mode:
encoding = _io.text_encoding(encoding)
- (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
+ name = None
+ def opener(*args):
+ nonlocal name
+ fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
+ return fd
try:
- file = _io.open(fd, mode, buffering=buffering,
- newline=newline, encoding=encoding, errors=errors)
-
- return _TemporaryFileWrapper(file, name, delete)
- except BaseException:
- _os.unlink(name)
- _os.close(fd)
+ file = _io.open(dir, mode, buffering=buffering,
+ newline=newline, encoding=encoding, errors=errors,
+ opener=opener)
+ try:
+ raw = getattr(file, 'buffer', file)
+ raw = getattr(raw, 'raw', raw)
+ raw.name = name
+ return _TemporaryFileWrapper(file, name, delete)
+ except:
+ file.close()
+ raise
+ except:
+ if name is not None and not (_os.name == 'nt' and delete):
+ _os.unlink(name)
raise
if _os.name != 'posix' or _sys.platform == 'cygwin':
@@ -593,9 +624,20 @@ def TemporaryFile(mode='w+b', buffering=-1, encoding=None,
flags = _bin_openflags
if _O_TMPFILE_WORKS:
- try:
+ fd = None
+ def opener(*args):
+ nonlocal fd
flags2 = (flags | _os.O_TMPFILE) & ~_os.O_CREAT
fd = _os.open(dir, flags2, 0o600)
+ return fd
+ try:
+ file = _io.open(dir, mode, buffering=buffering,
+ newline=newline, encoding=encoding,
+ errors=errors, opener=opener)
+ raw = getattr(file, 'buffer', file)
+ raw = getattr(raw, 'raw', raw)
+ raw.name = fd
+ return file
except IsADirectoryError:
# Linux kernel older than 3.11 ignores the O_TMPFILE flag:
# O_TMPFILE is read as O_DIRECTORY. Trying to open a directory
@@ -612,24 +654,25 @@ def TemporaryFile(mode='w+b', buffering=-1, encoding=None,
# fails with NotADirectoryError, because O_TMPFILE is read as
# O_DIRECTORY.
pass
- else:
- try:
- return _io.open(fd, mode, buffering=buffering,
- newline=newline, encoding=encoding,
- errors=errors)
- except:
- _os.close(fd)
- raise
# Fallback to _mkstemp_inner().
- (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
- try:
- _os.unlink(name)
- return _io.open(fd, mode, buffering=buffering,
- newline=newline, encoding=encoding, errors=errors)
- except:
- _os.close(fd)
- raise
+ fd = None
+ def opener(*args):
+ nonlocal fd
+ fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
+ try:
+ _os.unlink(name)
+ except BaseException as e:
+ _os.close(fd)
+ raise
+ return fd
+ file = _io.open(dir, mode, buffering=buffering,
+ newline=newline, encoding=encoding, errors=errors,
+ opener=opener)
+ raw = getattr(file, 'buffer', file)
+ raw = getattr(raw, 'raw', raw)
+ raw.name = fd
+ return file
class SpooledTemporaryFile:
"""Temporary file wrapper, specialized to switch from BytesIO
@@ -800,17 +843,10 @@ def __init__(self, suffix=None, prefix=None, dir=None,
def _rmtree(cls, name, ignore_errors=False):
def onerror(func, path, exc_info):
if issubclass(exc_info[0], PermissionError):
- def resetperms(path):
- try:
- _os.chflags(path, 0)
- except AttributeError:
- pass
- _os.chmod(path, 0o700)
-
try:
if path != name:
- resetperms(_os.path.dirname(path))
- resetperms(path)
+ _resetperms(_os.path.dirname(path))
+ _resetperms(path)
try:
_os.unlink(path)
diff --git a/Lib/test/_test_atexit.py b/Lib/test/_test_atexit.py
index a31658531113ba..55d28083349175 100644
--- a/Lib/test/_test_atexit.py
+++ b/Lib/test/_test_atexit.py
@@ -116,6 +116,21 @@ def test_bound_methods(self):
atexit._run_exitfuncs()
self.assertEqual(l, [5])
+ def test_atexit_with_unregistered_function(self):
+ # See bpo-46025 for more info
+ def func():
+ atexit.unregister(func)
+ 1/0
+ atexit.register(func)
+ try:
+ with support.catch_unraisable_exception() as cm:
+ atexit._run_exitfuncs()
+ self.assertEqual(cm.unraisable.object, func)
+ self.assertEqual(cm.unraisable.exc_type, ZeroDivisionError)
+ self.assertEqual(type(cm.unraisable.exc_value), ZeroDivisionError)
+ finally:
+ atexit.unregister(func)
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index ead92cfa2abfea..57eada634c4459 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -5,6 +5,7 @@
import unittest
import unittest.mock
import queue as pyqueue
+import textwrap
import time
import io
import itertools
@@ -73,6 +74,12 @@
msvcrt = None
+if support.check_sanitizer(address=True):
+ # bpo-45200: Skip multiprocessing tests if Python is built with ASAN to
+ # work around a libasan race condition: dead lock in pthread_create().
+ raise unittest.SkipTest("libasan has a pthread_create() dead lock")
+
+
def latin(s):
return s.encode('latin')
@@ -611,6 +618,7 @@ def test_lose_target_ref(self):
del c
p.start()
p.join()
+ gc.collect() # For PyPy or other GCs.
self.assertIs(wr(), None)
self.assertEqual(q.get(), 5)
close_queue(q)
@@ -2286,6 +2294,16 @@ def test_dict_proxy_nested(self):
self.assertIsInstance(outer[0], list) # Not a ListProxy
self.assertEqual(outer[-1][-1]['feed'], 3)
+ def test_nested_queue(self):
+ a = self.list() # Test queue inside list
+ a.append(self.Queue())
+ a[0].put(123)
+ self.assertEqual(a[0].get(), 123)
+ b = self.dict() # Test queue inside dict
+ b[0] = self.Queue()
+ b[0].put(456)
+ self.assertEqual(b[0].get(), 456)
+
def test_namespace(self):
n = self.Namespace()
n.name = 'Bob'
@@ -2657,6 +2675,7 @@ def test_release_task_refs(self):
self.pool.map(identity, objs)
del objs
+ gc.collect() # For PyPy or other GCs.
time.sleep(DELTA) # let threaded cleanup code run
self.assertEqual(set(wr() for wr in refs), {None})
# With a process pool, copies of the objects are returned, check
@@ -2786,6 +2805,11 @@ def test_pool_worker_lifetime_early_close(self):
for (j, res) in enumerate(results):
self.assertEqual(res.get(), sqr(j))
+ def test_pool_maxtasksperchild_invalid(self):
+ for value in [0, -1, 0.5, "12"]:
+ with self.assertRaises(ValueError):
+ multiprocessing.Pool(3, maxtasksperchild=value)
+
def test_worker_finalization_via_atexit_handler_of_multiprocessing(self):
# tests cases against bpo-38744 and bpo-39360
cmd = '''if 1:
@@ -3761,12 +3785,19 @@ def _attach_existing_shmem_then_write(shmem_name_or_obj, binary_data):
local_sms.buf[:len(binary_data)] = binary_data
local_sms.close()
+ def _new_shm_name(self, prefix):
+ # Add a PID to the name of a POSIX shared memory object to allow
+ # running multiprocessing tests (test_multiprocessing_fork,
+ # test_multiprocessing_spawn, etc) in parallel.
+ return prefix + str(os.getpid())
+
def test_shared_memory_basics(self):
- sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512)
+ name_tsmb = self._new_shm_name('test01_tsmb')
+ sms = shared_memory.SharedMemory(name_tsmb, create=True, size=512)
self.addCleanup(sms.unlink)
# Verify attributes are readable.
- self.assertEqual(sms.name, 'test01_tsmb')
+ self.assertEqual(sms.name, name_tsmb)
self.assertGreaterEqual(sms.size, 512)
self.assertGreaterEqual(len(sms.buf), sms.size)
@@ -3774,25 +3805,17 @@ def test_shared_memory_basics(self):
self.assertIn(sms.name, str(sms))
self.assertIn(str(sms.size), str(sms))
- # Test pickling
- sms.buf[0:6] = b'pickle'
- pickled_sms = pickle.dumps(sms)
- sms2 = pickle.loads(pickled_sms)
- self.assertEqual(sms.name, sms2.name)
- self.assertEqual(sms.size, sms2.size)
- self.assertEqual(bytes(sms.buf[0:6]), bytes(sms2.buf[0:6]), b'pickle')
-
# Modify contents of shared memory segment through memoryview.
sms.buf[0] = 42
self.assertEqual(sms.buf[0], 42)
# Attach to existing shared memory segment.
- also_sms = shared_memory.SharedMemory('test01_tsmb')
+ also_sms = shared_memory.SharedMemory(name_tsmb)
self.assertEqual(also_sms.buf[0], 42)
also_sms.close()
# Attach to existing shared memory segment but specify a new size.
- same_sms = shared_memory.SharedMemory('test01_tsmb', size=20*sms.size)
+ same_sms = shared_memory.SharedMemory(name_tsmb, size=20*sms.size)
self.assertLess(same_sms.size, 20*sms.size) # Size was ignored.
same_sms.close()
@@ -3810,7 +3833,7 @@ def test_shared_memory_basics(self):
'multiprocessing.shared_memory._make_filename') as mock_make_filename:
NAME_PREFIX = shared_memory._SHM_NAME_PREFIX
- names = ['test01_fn', 'test02_fn']
+ names = [self._new_shm_name('test01_fn'), self._new_shm_name('test02_fn')]
# Prepend NAME_PREFIX which can be '/psm_' or 'wnsm_', necessary
# because some POSIX compliant systems require name to start with /
names = [NAME_PREFIX + name for name in names]
@@ -3832,17 +3855,17 @@ def test_shared_memory_basics(self):
# manages unlinking on its own and unlink() does nothing).
# True release of shared memory segment does not necessarily
# happen until process exits, depending on the OS platform.
+ name_dblunlink = self._new_shm_name('test01_dblunlink')
+ sms_uno = shared_memory.SharedMemory(
+ name_dblunlink,
+ create=True,
+ size=5000
+ )
with self.assertRaises(FileNotFoundError):
- sms_uno = shared_memory.SharedMemory(
- 'test01_dblunlink',
- create=True,
- size=5000
- )
-
try:
self.assertGreaterEqual(sms_uno.size, 5000)
- sms_duo = shared_memory.SharedMemory('test01_dblunlink')
+ sms_duo = shared_memory.SharedMemory(name_dblunlink)
sms_duo.unlink() # First shm_unlink() call.
sms_duo.close()
sms_uno.close()
@@ -3854,7 +3877,7 @@ def test_shared_memory_basics(self):
# Attempting to create a new shared memory segment with a
# name that is already in use triggers an exception.
there_can_only_be_one_sms = shared_memory.SharedMemory(
- 'test01_tsmb',
+ name_tsmb,
create=True,
size=512
)
@@ -3868,7 +3891,7 @@ def test_shared_memory_basics(self):
# case of MacOS/darwin, requesting a smaller size is disallowed.
class OptionalAttachSharedMemory(shared_memory.SharedMemory):
_flags = os.O_CREAT | os.O_RDWR
- ok_if_exists_sms = OptionalAttachSharedMemory('test01_tsmb')
+ ok_if_exists_sms = OptionalAttachSharedMemory(name_tsmb)
self.assertEqual(ok_if_exists_sms.size, sms.size)
ok_if_exists_sms.close()
@@ -3880,6 +3903,29 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory):
sms.close()
+ def test_shared_memory_recreate(self):
+ # Test if shared memory segment is created properly,
+ # when _make_filename returns an existing shared memory segment name
+ with unittest.mock.patch(
+ 'multiprocessing.shared_memory._make_filename') as mock_make_filename:
+
+ NAME_PREFIX = shared_memory._SHM_NAME_PREFIX
+ names = [self._new_shm_name('test03_fn'), self._new_shm_name('test04_fn')]
+ # Prepend NAME_PREFIX which can be '/psm_' or 'wnsm_', necessary
+ # because some POSIX compliant systems require name to start with /
+ names = [NAME_PREFIX + name for name in names]
+
+ mock_make_filename.side_effect = names
+ shm1 = shared_memory.SharedMemory(create=True, size=1)
+ self.addCleanup(shm1.unlink)
+ self.assertEqual(shm1._name, names[0])
+
+ mock_make_filename.side_effect = names
+ shm2 = shared_memory.SharedMemory(create=True, size=1)
+ self.addCleanup(shm2.unlink)
+ self.assertEqual(shm2._name, names[1])
+
+ def test_invalid_shared_memory_cration(self):
# Test creating a shared memory segment with negative size
with self.assertRaises(ValueError):
sms_invalid = shared_memory.SharedMemory(create=True, size=-1)
@@ -3892,6 +3938,47 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory):
with self.assertRaises(ValueError):
sms_invalid = shared_memory.SharedMemory(create=True)
+ def test_shared_memory_pickle_unpickle(self):
+ for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+ with self.subTest(proto=proto):
+ sms = shared_memory.SharedMemory(create=True, size=512)
+ self.addCleanup(sms.unlink)
+ sms.buf[0:6] = b'pickle'
+
+ # Test pickling
+ pickled_sms = pickle.dumps(sms, protocol=proto)
+
+ # Test unpickling
+ sms2 = pickle.loads(pickled_sms)
+ self.assertIsInstance(sms2, shared_memory.SharedMemory)
+ self.assertEqual(sms.name, sms2.name)
+ self.assertEqual(bytes(sms.buf[0:6]), b'pickle')
+ self.assertEqual(bytes(sms2.buf[0:6]), b'pickle')
+
+ # Test that unpickled version is still the same SharedMemory
+ sms.buf[0:6] = b'newval'
+ self.assertEqual(bytes(sms.buf[0:6]), b'newval')
+ self.assertEqual(bytes(sms2.buf[0:6]), b'newval')
+
+ sms2.buf[0:6] = b'oldval'
+ self.assertEqual(bytes(sms.buf[0:6]), b'oldval')
+ self.assertEqual(bytes(sms2.buf[0:6]), b'oldval')
+
+ def test_shared_memory_pickle_unpickle_dead_object(self):
+ for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+ with self.subTest(proto=proto):
+ sms = shared_memory.SharedMemory(create=True, size=512)
+ sms.buf[0:6] = b'pickle'
+ pickled_sms = pickle.dumps(sms, protocol=proto)
+
+ # Now, we are going to kill the original object.
+ # So, unpickled one won't be able to attach to it.
+ sms.close()
+ sms.unlink()
+
+ with self.assertRaises(FileNotFoundError):
+ pickle.loads(pickled_sms)
+
def test_shared_memory_across_processes(self):
# bpo-40135: don't define shared memory block's name in case of
# the failure when we run multiprocessing tests in parallel.
@@ -4073,10 +4160,11 @@ def test_shared_memory_ShareableList_basics(self):
self.assertEqual(sl.count(b'adios'), 0)
# Exercise creating a duplicate.
- sl_copy = shared_memory.ShareableList(sl, name='test03_duplicate')
+ name_duplicate = self._new_shm_name('test03_duplicate')
+ sl_copy = shared_memory.ShareableList(sl, name=name_duplicate)
try:
self.assertNotEqual(sl.shm.name, sl_copy.shm.name)
- self.assertEqual('test03_duplicate', sl_copy.shm.name)
+ self.assertEqual(name_duplicate, sl_copy.shm.name)
self.assertEqual(list(sl), list(sl_copy))
self.assertEqual(sl.format, sl_copy.format)
sl_copy[-1] = 77
@@ -4108,29 +4196,45 @@ def test_shared_memory_ShareableList_basics(self):
empty_sl.shm.unlink()
def test_shared_memory_ShareableList_pickling(self):
- sl = shared_memory.ShareableList(range(10))
- self.addCleanup(sl.shm.unlink)
-
- serialized_sl = pickle.dumps(sl)
- deserialized_sl = pickle.loads(serialized_sl)
- self.assertTrue(
- isinstance(deserialized_sl, shared_memory.ShareableList)
- )
- self.assertTrue(deserialized_sl[-1], 9)
- self.assertFalse(sl is deserialized_sl)
- deserialized_sl[4] = "changed"
- self.assertEqual(sl[4], "changed")
-
- # Verify data is not being put into the pickled representation.
- name = 'a' * len(sl.shm.name)
- larger_sl = shared_memory.ShareableList(range(400))
- self.addCleanup(larger_sl.shm.unlink)
- serialized_larger_sl = pickle.dumps(larger_sl)
- self.assertTrue(len(serialized_sl) == len(serialized_larger_sl))
- larger_sl.shm.close()
-
- deserialized_sl.shm.close()
- sl.shm.close()
+ for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+ with self.subTest(proto=proto):
+ sl = shared_memory.ShareableList(range(10))
+ self.addCleanup(sl.shm.unlink)
+
+ serialized_sl = pickle.dumps(sl, protocol=proto)
+ deserialized_sl = pickle.loads(serialized_sl)
+ self.assertIsInstance(
+ deserialized_sl, shared_memory.ShareableList)
+ self.assertEqual(deserialized_sl[-1], 9)
+ self.assertIsNot(sl, deserialized_sl)
+
+ deserialized_sl[4] = "changed"
+ self.assertEqual(sl[4], "changed")
+ sl[3] = "newvalue"
+ self.assertEqual(deserialized_sl[3], "newvalue")
+
+ larger_sl = shared_memory.ShareableList(range(400))
+ self.addCleanup(larger_sl.shm.unlink)
+ serialized_larger_sl = pickle.dumps(larger_sl, protocol=proto)
+ self.assertEqual(len(serialized_sl), len(serialized_larger_sl))
+ larger_sl.shm.close()
+
+ deserialized_sl.shm.close()
+ sl.shm.close()
+
+ def test_shared_memory_ShareableList_pickling_dead_object(self):
+ for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+ with self.subTest(proto=proto):
+ sl = shared_memory.ShareableList(range(10))
+ serialized_sl = pickle.dumps(sl, protocol=proto)
+
+ # Now, we are going to kill the original object.
+ # So, unpickled one won't be able to attach to it.
+ sl.shm.close()
+ sl.shm.unlink()
+
+ with self.assertRaises(FileNotFoundError):
+ pickle.loads(serialized_sl)
def test_shared_memory_cleaned_after_process_termination(self):
cmd = '''if 1:
@@ -4167,6 +4271,13 @@ def test_shared_memory_cleaned_after_process_termination(self):
" a process was abruptly terminated.")
if os.name == 'posix':
+ # Without this line it was raising warnings like:
+ # UserWarning: resource_tracker:
+ # There appear to be 1 leaked shared_memory
+ # objects to clean up at shutdown
+ # See: https://bugs.python.org/issue45209
+ resource_tracker.unregister(f"/{name}", "shared_memory")
+
# A warning was emitted by the subprocess' own
# resource_tracker (on Windows, shared memory segments
# are released automatically by the OS).
@@ -4176,7 +4287,7 @@ def test_shared_memory_cleaned_after_process_termination(self):
"shared_memory objects to clean up at shutdown", err)
#
-#
+# Test to verify that `Finalize` works.
#
class _TestFinalize(BaseTestCase):
@@ -4188,6 +4299,7 @@ def setUp(self):
util._finalizer_registry.clear()
def tearDown(self):
+ gc.collect() # For PyPy or other GCs.
self.assertFalse(util._finalizer_registry)
util._finalizer_registry.update(self.registry_backup)
@@ -4199,12 +4311,14 @@ class Foo(object):
a = Foo()
util.Finalize(a, conn.send, args=('a',))
del a # triggers callback for a
+ gc.collect() # For PyPy or other GCs.
b = Foo()
close_b = util.Finalize(b, conn.send, args=('b',))
close_b() # triggers callback for b
close_b() # does nothing because callback has already been called
del b # does nothing because callback has already been called
+ gc.collect() # For PyPy or other GCs.
c = Foo()
util.Finalize(c, conn.send, args=('c',))
@@ -5263,6 +5377,14 @@ def test_resource_tracker_reused(self):
self.assertTrue(is_resource_tracker_reused)
+ def test_too_long_name_resource(self):
+ # gh-96819: Resource names that will make the length of a write to a pipe
+ # greater than PIPE_BUF are not allowed
+ rtype = "shared_memory"
+ too_long_name_resource = "a" * (512 - len(rtype))
+ with self.assertRaises(ValueError):
+ resource_tracker.register(too_long_name_resource, rtype)
+
class TestSimpleQueue(unittest.TestCase):
@@ -5591,6 +5713,35 @@ def test_namespace(self):
self.run_worker(self._test_namespace, o)
+class TestNamedResource(unittest.TestCase):
+ def test_global_named_resource_spawn(self):
+ #
+ # gh-90549: Check that global named resources in main module
+ # will not leak by a subprocess, in spawn context.
+ #
+ testfn = os_helper.TESTFN
+ self.addCleanup(os_helper.unlink, testfn)
+ with open(testfn, 'w', encoding='utf-8') as f:
+ f.write(textwrap.dedent('''\
+ import multiprocessing as mp
+
+ ctx = mp.get_context('spawn')
+
+ global_resource = ctx.Semaphore()
+
+ def submain(): pass
+
+ if __name__ == '__main__':
+ p = ctx.Process(target=submain)
+ p.start()
+ p.join()
+ '''))
+ rc, out, err = test.support.script_helper.assert_python_ok(testfn)
+ # on error, err = 'UserWarning: resource_tracker: There appear to
+ # be 1 leaked semaphore objects to clean up at shutdown'
+ self.assertEqual(err, b'')
+
+
class MiscTestCase(unittest.TestCase):
def test__all__(self):
# Just make sure names in not_exported are excluded
@@ -5821,3 +5972,15 @@ def tearDownModule():
remote_globs['setUpModule'] = setUpModule
remote_globs['tearDownModule'] = tearDownModule
+
+
+@unittest.skipIf(not hasattr(_multiprocessing, 'SemLock'), 'SemLock not available')
+@unittest.skipIf(sys.platform != "linux", "Linux only")
+class SemLockTests(unittest.TestCase):
+
+ def test_semlock_subclass(self):
+ class SemLock(_multiprocessing.SemLock):
+ pass
+ name = f'test_semlock_subclass-{os.getpid()}'
+ s = SemLock(1, 0, 10, name, False)
+ _multiprocessing.sem_unlink(name)
diff --git a/Lib/test/_typed_dict_helper.py b/Lib/test/_typed_dict_helper.py
new file mode 100644
index 00000000000000..d333db193183eb
--- /dev/null
+++ b/Lib/test/_typed_dict_helper.py
@@ -0,0 +1,18 @@
+"""Used to test `get_type_hints()` on a cross-module inherited `TypedDict` class
+
+This script uses future annotations to postpone a type that won't be available
+on the module inheriting from to `Foo`. The subclass in the other module should
+look something like this:
+
+ class Bar(_typed_dict_helper.Foo, total=False):
+ b: int
+"""
+
+from __future__ import annotations
+
+from typing import Optional, TypedDict
+
+OptionalIntType = Optional[int]
+
+class Foo(TypedDict):
+ a: OptionalIntType
diff --git a/Lib/test/ann_module.py b/Lib/test/ann_module.py
index 0567d6de1b6478..5081e6b58345a9 100644
--- a/Lib/test/ann_module.py
+++ b/Lib/test/ann_module.py
@@ -58,3 +58,5 @@ def dec(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
+
+u: int | float
diff --git a/Lib/test/ann_module5.py b/Lib/test/ann_module5.py
new file mode 100644
index 00000000000000..837041e121f652
--- /dev/null
+++ b/Lib/test/ann_module5.py
@@ -0,0 +1,10 @@
+# Used by test_typing to verify that Final wrapped in ForwardRef works.
+
+from __future__ import annotations
+
+from typing import Final
+
+name: Final[str] = "final"
+
+class MyClass:
+ value: Final = 3000
diff --git a/Lib/test/ann_module6.py b/Lib/test/ann_module6.py
new file mode 100644
index 00000000000000..679175669bc3ac
--- /dev/null
+++ b/Lib/test/ann_module6.py
@@ -0,0 +1,7 @@
+# Tests that top-level ClassVar is not allowed
+
+from __future__ import annotations
+
+from typing import ClassVar
+
+wrong: ClassVar[int] = 1
diff --git a/Lib/test/ann_module7.py b/Lib/test/ann_module7.py
new file mode 100644
index 00000000000000..8f890cd28025be
--- /dev/null
+++ b/Lib/test/ann_module7.py
@@ -0,0 +1,11 @@
+# Tests class have ``__text_signature__``
+
+from __future__ import annotations
+
+DEFAULT_BUFFER_SIZE = 8192
+
+class BufferedReader(object):
+ """BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE)\n--\n\n
+ Create a new buffered reader using the given readable raw IO object.
+ """
+ pass
diff --git a/Lib/test/audit-tests.py b/Lib/test/audit-tests.py
index 7a7de637c38823..481aedd6b4b2c7 100644
--- a/Lib/test/audit-tests.py
+++ b/Lib/test/audit-tests.py
@@ -6,6 +6,7 @@
"""
import contextlib
+import os
import sys
@@ -106,6 +107,32 @@ def test_block_add_hook_baseexception():
pass
+def test_marshal():
+ import marshal
+ o = ("a", "b", "c", 1, 2, 3)
+ payload = marshal.dumps(o)
+
+ with TestHook() as hook:
+ assertEqual(o, marshal.loads(marshal.dumps(o)))
+
+ try:
+ with open("test-marshal.bin", "wb") as f:
+ marshal.dump(o, f)
+ with open("test-marshal.bin", "rb") as f:
+ assertEqual(o, marshal.load(f))
+ finally:
+ os.unlink("test-marshal.bin")
+
+ actual = [(a[0], a[1]) for e, a in hook.seen if e == "marshal.dumps"]
+ assertSequenceEqual(actual, [(o, marshal.version)] * 2)
+
+ actual = [a[0] for e, a in hook.seen if e == "marshal.loads"]
+ assertSequenceEqual(actual, [payload])
+
+ actual = [e for e, a in hook.seen if e == "marshal.load"]
+ assertSequenceEqual(actual, ["marshal.load"])
+
+
def test_pickle():
import pickle
@@ -381,6 +408,38 @@ def hook(event, *args):
raise RuntimeError("Expected sqlite3.load_extension to fail")
+def test_syslog():
+ import syslog
+
+ def hook(event, args):
+ if event.startswith("syslog."):
+ print(event, *args)
+
+ sys.addaudithook(hook)
+ syslog.openlog('python')
+ syslog.syslog('test')
+ syslog.setlogmask(syslog.LOG_DEBUG)
+ syslog.closelog()
+ # implicit open
+ syslog.syslog('test2')
+ # open with default ident
+ syslog.openlog(logoption=syslog.LOG_NDELAY, facility=syslog.LOG_LOCAL0)
+ sys.argv = None
+ syslog.openlog()
+ syslog.closelog()
+
+
+def test_not_in_gc():
+ import gc
+
+ hook = lambda *a: None
+ sys.addaudithook(hook)
+
+ for o in gc.get_objects():
+ if isinstance(o, list):
+ assert hook not in o
+
+
if __name__ == "__main__":
from test.support import suppress_msvcrt_asserts
diff --git a/Lib/test/clinic.test b/Lib/test/clinic.test
index 07e13829d5db97..af111dbee2b8c2 100644
--- a/Lib/test/clinic.test
+++ b/Lib/test/clinic.test
@@ -1740,29 +1740,18 @@ test_str_converter_encoding(PyObject *module, PyObject *const *args, Py_ssize_t
goto exit;
}
return_value = test_str_converter_encoding_impl(module, a, b, c, d, d_length, e, e_length);
+ /* Post parse cleanup for a */
+ PyMem_FREE(a);
+ /* Post parse cleanup for b */
+ PyMem_FREE(b);
+ /* Post parse cleanup for c */
+ PyMem_FREE(c);
+ /* Post parse cleanup for d */
+ PyMem_FREE(d);
+ /* Post parse cleanup for e */
+ PyMem_FREE(e);
exit:
- /* Cleanup for a */
- if (a) {
- PyMem_FREE(a);
- }
- /* Cleanup for b */
- if (b) {
- PyMem_FREE(b);
- }
- /* Cleanup for c */
- if (c) {
- PyMem_FREE(c);
- }
- /* Cleanup for d */
- if (d) {
- PyMem_FREE(d);
- }
- /* Cleanup for e */
- if (e) {
- PyMem_FREE(e);
- }
-
return return_value;
}
@@ -1770,7 +1759,7 @@ static PyObject *
test_str_converter_encoding_impl(PyObject *module, char *a, char *b, char *c,
char *d, Py_ssize_clean_t d_length, char *e,
Py_ssize_clean_t e_length)
-/*[clinic end generated code: output=f579dd9e795a364e input=eb4c38e1f898f402]*/
+/*[clinic end generated code: output=2d7e8b6203db31aa input=eb4c38e1f898f402]*/
/*[clinic input]
@@ -1805,12 +1794,12 @@ static PyObject *
test_Py_UNICODE_converter(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
{
PyObject *return_value = NULL;
- const Py_UNICODE *a;
- const Py_UNICODE *b;
- const Py_UNICODE *c;
- const Py_UNICODE *d;
+ const Py_UNICODE *a = NULL;
+ const Py_UNICODE *b = NULL;
+ const Py_UNICODE *c = NULL;
+ const Py_UNICODE *d = NULL;
Py_ssize_clean_t d_length;
- const Py_UNICODE *e;
+ const Py_UNICODE *e = NULL;
Py_ssize_clean_t e_length;
if (!_PyArg_ParseStack(args, nargs, "O&O&O&u#Z#:test_Py_UNICODE_converter",
@@ -1843,7 +1832,7 @@ test_Py_UNICODE_converter_impl(PyObject *module, const Py_UNICODE *a,
Py_ssize_clean_t d_length,
const Py_UNICODE *e,
Py_ssize_clean_t e_length)
-/*[clinic end generated code: output=ef45e982fedf0b3d input=064a3b68ad7f04b0]*/
+/*[clinic end generated code: output=e45007c49e77ad4a input=064a3b68ad7f04b0]*/
/*[clinic input]
@@ -3308,3 +3297,44 @@ test_preprocessor_guarded_else(PyObject *module, PyObject *Py_UNUSED(ignored))
#define TEST_PREPROCESSOR_GUARDED_ELSE_METHODDEF
#endif /* !defined(TEST_PREPROCESSOR_GUARDED_ELSE_METHODDEF) */
/*[clinic end generated code: output=3804bb18d454038c input=3fc80c9989d2f2e1]*/
+
+/*[clinic input]
+test_paramname_module
+
+ module as mod: object
+[clinic start generated code]*/
+
+PyDoc_STRVAR(test_paramname_module__doc__,
+"test_paramname_module($module, /, module)\n"
+"--\n"
+"\n");
+
+#define TEST_PARAMNAME_MODULE_METHODDEF \
+ {"test_paramname_module", (PyCFunction)(void(*)(void))test_paramname_module, METH_FASTCALL|METH_KEYWORDS, test_paramname_module__doc__},
+
+static PyObject *
+test_paramname_module_impl(PyObject *module, PyObject *mod);
+
+static PyObject *
+test_paramname_module(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
+{
+ PyObject *return_value = NULL;
+ static const char * const _keywords[] = {"module", NULL};
+ static _PyArg_Parser _parser = {NULL, _keywords, "test_paramname_module", 0};
+ PyObject *argsbuf[1];
+ PyObject *mod;
+
+ args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf);
+ if (!args) {
+ goto exit;
+ }
+ mod = args[0];
+ return_value = test_paramname_module_impl(module, mod);
+
+exit:
+ return return_value;
+}
+
+static PyObject *
+test_paramname_module_impl(PyObject *module, PyObject *mod)
+/*[clinic end generated code: output=2a6769d34d1b2be0 input=afefe259667f13ba]*/
diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py
index 316cf0b8da7b48..f474a756b33a90 100644
--- a/Lib/test/datetimetester.py
+++ b/Lib/test/datetimetester.py
@@ -1,6 +1,6 @@
"""Test date/time type.
-See http://www.zope.org/Members/fdrake/DateTimeWiki/TestCases
+See https://www.zope.dev/Members/fdrake/DateTimeWiki/TestCases
"""
import io
import itertools
@@ -344,7 +344,7 @@ def test_comparison(self):
with self.assertRaises(TypeError): timezone(ZERO) < timezone(ZERO)
self.assertIn(timezone(ZERO), {timezone(ZERO)})
self.assertTrue(timezone(ZERO) != None)
- self.assertFalse(timezone(ZERO) == None)
+ self.assertFalse(timezone(ZERO) == None)
tz = timezone(ZERO)
self.assertTrue(tz == ALWAYS_EQ)
@@ -1861,8 +1861,6 @@ def test_fromisoformat_fails(self):
def test_fromisoformat_fails_typeerror(self):
# Test that fromisoformat fails when passed the wrong type
- import io
-
bad_types = [b'2009-03-01', None, io.StringIO('2009-03-01')]
for bad_type in bad_types:
with self.assertRaises(TypeError):
@@ -2469,45 +2467,101 @@ def test_microsecond_rounding(self):
self.assertEqual(t.microsecond, 7812)
def test_timestamp_limits(self):
- # minimum timestamp
- min_dt = self.theclass.min.replace(tzinfo=timezone.utc)
+ with self.subTest("minimum UTC"):
+ min_dt = self.theclass.min.replace(tzinfo=timezone.utc)
+ min_ts = min_dt.timestamp()
+
+ # This test assumes that datetime.min == 0000-01-01T00:00:00.00
+ # If that assumption changes, this value can change as well
+ self.assertEqual(min_ts, -62135596800)
+
+ with self.subTest("maximum UTC"):
+ # Zero out microseconds to avoid rounding issues
+ max_dt = self.theclass.max.replace(tzinfo=timezone.utc,
+ microsecond=0)
+ max_ts = max_dt.timestamp()
+
+ # This test assumes that datetime.max == 9999-12-31T23:59:59.999999
+ # If that assumption changes, this value can change as well
+ self.assertEqual(max_ts, 253402300799.0)
+
+ def test_fromtimestamp_limits(self):
+ try:
+ self.theclass.fromtimestamp(-2**32 - 1)
+ except (OSError, OverflowError):
+ self.skipTest("Test not valid on this platform")
+
+ # XXX: Replace these with datetime.{min,max}.timestamp() when we solve
+ # the issue with gh-91012
+ min_dt = self.theclass.min + timedelta(days=1)
min_ts = min_dt.timestamp()
+
+ max_dt = self.theclass.max.replace(microsecond=0)
+ max_ts = ((self.theclass.max - timedelta(hours=23)).timestamp() +
+ timedelta(hours=22, minutes=59, seconds=59).total_seconds())
+
+ for (test_name, ts, expected) in [
+ ("minimum", min_ts, min_dt),
+ ("maximum", max_ts, max_dt),
+ ]:
+ with self.subTest(test_name, ts=ts, expected=expected):
+ actual = self.theclass.fromtimestamp(ts)
+
+ self.assertEqual(actual, expected)
+
+ # Test error conditions
+ test_cases = [
+ ("Too small by a little", min_ts - timedelta(days=1, hours=12).total_seconds()),
+ ("Too small by a lot", min_ts - timedelta(days=400).total_seconds()),
+ ("Too big by a little", max_ts + timedelta(days=1).total_seconds()),
+ ("Too big by a lot", max_ts + timedelta(days=400).total_seconds()),
+ ]
+
+ for test_name, ts in test_cases:
+ with self.subTest(test_name, ts=ts):
+ with self.assertRaises((ValueError, OverflowError)):
+ # converting a Python int to C time_t can raise a
+ # OverflowError, especially on 32-bit platforms.
+ self.theclass.fromtimestamp(ts)
+
+ def test_utcfromtimestamp_limits(self):
try:
- # date 0001-01-01 00:00:00+00:00: timestamp=-62135596800
- self.assertEqual(self.theclass.fromtimestamp(min_ts, tz=timezone.utc),
- min_dt)
- except (OverflowError, OSError) as exc:
- # the date 0001-01-01 doesn't fit into 32-bit time_t,
- # or platform doesn't support such very old date
- self.skipTest(str(exc))
-
- # maximum timestamp: set seconds to zero to avoid rounding issues
- max_dt = self.theclass.max.replace(tzinfo=timezone.utc,
- second=0, microsecond=0)
+ self.theclass.utcfromtimestamp(-2**32 - 1)
+ except (OSError, OverflowError):
+ self.skipTest("Test not valid on this platform")
+
+ min_dt = self.theclass.min.replace(tzinfo=timezone.utc)
+ min_ts = min_dt.timestamp()
+
+ max_dt = self.theclass.max.replace(microsecond=0, tzinfo=timezone.utc)
max_ts = max_dt.timestamp()
- # date 9999-12-31 23:59:00+00:00: timestamp 253402300740
- self.assertEqual(self.theclass.fromtimestamp(max_ts, tz=timezone.utc),
- max_dt)
-
- # number of seconds greater than 1 year: make sure that the new date
- # is not valid in datetime.datetime limits
- delta = 3600 * 24 * 400
-
- # too small
- ts = min_ts - delta
- # converting a Python int to C time_t can raise a OverflowError,
- # especially on 32-bit platforms.
- with self.assertRaises((ValueError, OverflowError)):
- self.theclass.fromtimestamp(ts)
- with self.assertRaises((ValueError, OverflowError)):
- self.theclass.utcfromtimestamp(ts)
-
- # too big
- ts = max_dt.timestamp() + delta
- with self.assertRaises((ValueError, OverflowError)):
- self.theclass.fromtimestamp(ts)
- with self.assertRaises((ValueError, OverflowError)):
- self.theclass.utcfromtimestamp(ts)
+
+ for (test_name, ts, expected) in [
+ ("minimum", min_ts, min_dt.replace(tzinfo=None)),
+ ("maximum", max_ts, max_dt.replace(tzinfo=None)),
+ ]:
+ with self.subTest(test_name, ts=ts, expected=expected):
+ try:
+ actual = self.theclass.utcfromtimestamp(ts)
+ except (OSError, OverflowError) as exc:
+ self.skipTest(str(exc))
+
+ self.assertEqual(actual, expected)
+
+ # Test error conditions
+ test_cases = [
+ ("Too small by a little", min_ts - 1),
+ ("Too small by a lot", min_ts - timedelta(days=400).total_seconds()),
+ ("Too big by a little", max_ts + 1),
+ ("Too big by a lot", max_ts + timedelta(days=400).total_seconds()),
+ ]
+
+ for test_name, ts in test_cases:
+ with self.subTest(test_name, ts=ts):
+ with self.assertRaises((ValueError, OverflowError)):
+ # converting a Python int to C time_t can raise a
+ # OverflowError, especially on 32-bit platforms.
+ self.theclass.utcfromtimestamp(ts)
def test_insane_fromtimestamp(self):
# It's possible that some platform maps time_t to double,
@@ -3985,8 +4039,6 @@ def test_fromisoformat_fails(self):
def test_fromisoformat_fails_typeerror(self):
# Test the fromisoformat fails when passed the wrong type
- import io
-
bad_types = [b'12:30:45', None, io.StringIO('12:30:45')]
for bad_type in bad_types:
@@ -4064,7 +4116,7 @@ def test_even_more_compare(self):
self.assertEqual(t1, t1)
self.assertEqual(t2, t2)
- # Equal afer adjustment.
+ # Equal after adjustment.
t1 = self.theclass(1, 12, 31, 23, 59, tzinfo=FixedOffset(1, ""))
t2 = self.theclass(2, 1, 1, 3, 13, tzinfo=FixedOffset(3*60+13+2, ""))
self.assertEqual(t1, t2)
@@ -4903,7 +4955,7 @@ def test_easy(self):
# OTOH, these fail! Don't enable them. The difficulty is that
# the edge case tests assume that every hour is representable in
# the "utc" class. This is always true for a fixed-offset tzinfo
- # class (lke utc_real and utc_fake), but not for Eastern or Central.
+ # class (like utc_real and utc_fake), but not for Eastern or Central.
# For these adjacent DST-aware time zones, the range of time offsets
# tested ends up creating hours in the one that aren't representable
# in the other. For the same reason, we would see failures in the
diff --git a/Lib/test/decimaltestdata/abs.decTest b/Lib/test/decimaltestdata/abs.decTest
index 01f73d7766648f..569b8fcd84ab62 100644
--- a/Lib/test/decimaltestdata/abs.decTest
+++ b/Lib/test/decimaltestdata/abs.decTest
@@ -20,7 +20,7 @@
version: 2.59
-- This set of tests primarily tests the existence of the operator.
--- Additon, subtraction, rounding, and more overflows are tested
+-- Addition, subtraction, rounding, and more overflows are tested
-- elsewhere.
precision: 9
diff --git a/Lib/test/decimaltestdata/extra.decTest b/Lib/test/decimaltestdata/extra.decTest
index b630d8e3f9d45e..31291202a35e79 100644
--- a/Lib/test/decimaltestdata/extra.decTest
+++ b/Lib/test/decimaltestdata/extra.decTest
@@ -156,7 +156,7 @@ extr1302 fma -Inf 0E-456 sNaN148 -> NaN Invalid_operation
-- max/min/max_mag/min_mag bug in 2.5.2/2.6/3.0: max(NaN, finite) gave
-- incorrect answers when the finite number required rounding; similarly
--- for the other thre functions
+-- for the other three functions
maxexponent: 999
minexponent: -999
precision: 6
diff --git a/Lib/test/doctest_lineno.py b/Lib/test/doctest_lineno.py
new file mode 100644
index 00000000000000..be198513a1502c
--- /dev/null
+++ b/Lib/test/doctest_lineno.py
@@ -0,0 +1,50 @@
+# This module is used in `test_doctest`.
+# It must not have a docstring.
+
+def func_with_docstring():
+ """Some unrelated info."""
+
+
+def func_without_docstring():
+ pass
+
+
+def func_with_doctest():
+ """
+ This function really contains a test case.
+
+ >>> func_with_doctest.__name__
+ 'func_with_doctest'
+ """
+ return 3
+
+
+class ClassWithDocstring:
+ """Some unrelated class information."""
+
+
+class ClassWithoutDocstring:
+ pass
+
+
+class ClassWithDoctest:
+ """This class really has a test case in it.
+
+ >>> ClassWithDoctest.__name__
+ 'ClassWithDoctest'
+ """
+
+
+class MethodWrapper:
+ def method_with_docstring(self):
+ """Method with a docstring."""
+
+ def method_without_docstring(self):
+ pass
+
+ def method_with_doctest(self):
+ """
+ This has a doctest!
+ >>> MethodWrapper.method_with_doctest.__name__
+ 'method_with_doctest'
+ """
diff --git a/Lib/test/imghdrdata/python.gif b/Lib/test/imghdrdata/python.gif
index 96fd9fef76b108..efa0be3861d79f 100644
Binary files a/Lib/test/imghdrdata/python.gif and b/Lib/test/imghdrdata/python.gif differ
diff --git a/Lib/test/inspect_fodder2.py b/Lib/test/inspect_fodder2.py
index e7d4b53ebefcc6..2dc49817087c44 100644
--- a/Lib/test/inspect_fodder2.py
+++ b/Lib/test/inspect_fodder2.py
@@ -259,3 +259,17 @@ def all_markers_with_args_and_kwargs(a, b, /, c, d, *args, e, f, **kwargs):
#line 259
def all_markers_with_defaults(a, b=1, /, c=2, d=3, *, e=4, f=5):
pass
+
+# line 263
+def deco_factory(**kwargs):
+ def deco(f):
+ @wraps(f)
+ def wrapper(*a, **kwd):
+ kwd.update(kwargs)
+ return f(*a, **kwd)
+ return wrapper
+ return deco
+
+@deco_factory(foo=(1 + 2), bar=lambda: 1)
+def complex_decorated(foo=0, bar=lambda: 0):
+ return foo + bar()
diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.py
index a4bac79c8af680..29f4ede523b007 100644
--- a/Lib/test/libregrtest/cmdline.py
+++ b/Lib/test/libregrtest/cmdline.py
@@ -140,6 +140,39 @@
# default (see bpo-30822).
RESOURCE_NAMES = ALL_RESOURCES + ('extralargefile', 'tzdata')
+
+class Namespace(argparse.Namespace):
+ def __init__(self, **kwargs) -> None:
+ self.testdir = None
+ self.verbose = 0
+ self.quiet = False
+ self.exclude = False
+ self.single = False
+ self.randomize = False
+ self.fromfile = None
+ self.findleaks = 1
+ self.fail_env_changed = False
+ self.use_resources = None
+ self.trace = False
+ self.coverdir = 'coverage'
+ self.runleaks = False
+ self.huntrleaks = False
+ self.verbose2 = False
+ self.verbose3 = False
+ self.print_slow = False
+ self.random_seed = None
+ self.use_mp = None
+ self.forever = False
+ self.header = False
+ self.failfast = False
+ self.match_tests = None
+ self.ignore_tests = None
+ self.pgo = False
+ self.pgo_extended = False
+
+ super().__init__(**kwargs)
+
+
class _ArgParser(argparse.ArgumentParser):
def error(self, message):
@@ -320,13 +353,7 @@ def resources_list(string):
def _parse_args(args, **kwargs):
# Defaults
- ns = argparse.Namespace(testdir=None, verbose=0, quiet=False,
- exclude=False, single=False, randomize=False, fromfile=None,
- findleaks=1, use_resources=None, trace=False, coverdir='coverage',
- runleaks=False, huntrleaks=False, verbose2=False, print_slow=False,
- random_seed=None, use_mp=None, verbose3=False, forever=False,
- header=False, failfast=False, match_tests=None, ignore_tests=None,
- pgo=False)
+ ns = Namespace()
for k, v in kwargs.items():
if not hasattr(ns, k):
raise TypeError('%r is an invalid keyword argument '
diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index 1df927da403d6e..400a3db5de4d27 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -11,10 +11,10 @@
import unittest
from test.libregrtest.cmdline import _parse_args
from test.libregrtest.runtest import (
- findtests, runtest, get_abs_module,
- STDTESTS, NOTTESTS, PASSED, FAILED, ENV_CHANGED, SKIPPED, RESOURCE_DENIED,
- INTERRUPTED, CHILD_ERROR, TEST_DID_NOT_RUN, TIMEOUT,
- PROGRESS_MIN_TIME, format_test_result, is_failed)
+ findtests, runtest, get_abs_module, is_failed,
+ STDTESTS, NOTTESTS, PROGRESS_MIN_TIME,
+ Passed, Failed, EnvChanged, Skipped, ResourceDenied, Interrupted,
+ ChildError, DidNotRun)
from test.libregrtest.setup import setup_tests
from test.libregrtest.pgo import setup_pgo_tests
from test.libregrtest.utils import removepy, count, format_duration, printlist
@@ -66,6 +66,7 @@ def __init__(self):
self.resource_denieds = []
self.environment_changed = []
self.run_no_tests = []
+ self.need_rerun = []
self.rerun = []
self.first_result = None
self.interrupted = False
@@ -99,34 +100,32 @@ def get_executed(self):
| set(self.run_no_tests))
def accumulate_result(self, result, rerun=False):
- test_name = result.test_name
- ok = result.result
+ test_name = result.name
- if ok not in (CHILD_ERROR, INTERRUPTED) and not rerun:
- self.test_times.append((result.test_time, test_name))
+ if not isinstance(result, (ChildError, Interrupted)) and not rerun:
+ self.test_times.append((result.duration_sec, test_name))
- if ok == PASSED:
+ if isinstance(result, Passed):
self.good.append(test_name)
- elif ok in (FAILED, CHILD_ERROR):
- if not rerun:
- self.bad.append(test_name)
- elif ok == ENV_CHANGED:
- self.environment_changed.append(test_name)
- elif ok == SKIPPED:
- self.skipped.append(test_name)
- elif ok == RESOURCE_DENIED:
+ elif isinstance(result, ResourceDenied):
self.skipped.append(test_name)
self.resource_denieds.append(test_name)
- elif ok == TEST_DID_NOT_RUN:
+ elif isinstance(result, Skipped):
+ self.skipped.append(test_name)
+ elif isinstance(result, EnvChanged):
+ self.environment_changed.append(test_name)
+ elif isinstance(result, Failed):
+ if not rerun:
+ self.bad.append(test_name)
+ self.need_rerun.append(result)
+ elif isinstance(result, DidNotRun):
self.run_no_tests.append(test_name)
- elif ok == INTERRUPTED:
+ elif isinstance(result, Interrupted):
self.interrupted = True
- elif ok == TIMEOUT:
- self.bad.append(test_name)
else:
- raise ValueError("invalid test result: %r" % ok)
+ raise ValueError("invalid test result: %r" % result)
- if rerun and ok not in {FAILED, CHILD_ERROR, INTERRUPTED}:
+ if rerun and not isinstance(result, (Failed, Interrupted)):
self.bad.remove(test_name)
xml_data = result.xml_data
@@ -314,15 +313,33 @@ def rerun_failed_tests(self):
self.log()
self.log("Re-running failed tests in verbose mode")
- self.rerun = self.bad[:]
- for test_name in self.rerun:
- self.log(f"Re-running {test_name} in verbose mode")
+ rerun_list = list(self.need_rerun)
+ self.need_rerun.clear()
+ for result in rerun_list:
+ test_name = result.name
+ self.rerun.append(test_name)
+
+ errors = result.errors or []
+ failures = result.failures or []
+ error_names = [test_full_name.split(" ")[0] for (test_full_name, *_) in errors]
+ failure_names = [test_full_name.split(" ")[0] for (test_full_name, *_) in failures]
self.ns.verbose = True
+ orig_match_tests = self.ns.match_tests
+ if errors or failures:
+ if self.ns.match_tests is None:
+ self.ns.match_tests = []
+ self.ns.match_tests.extend(error_names)
+ self.ns.match_tests.extend(failure_names)
+ matching = "matching: " + ", ".join(self.ns.match_tests)
+ self.log(f"Re-running {test_name} in verbose mode ({matching})")
+ else:
+ self.log(f"Re-running {test_name} in verbose mode")
result = runtest(self.ns, test_name)
+ self.ns.match_tests = orig_match_tests
self.accumulate_result(result, rerun=True)
- if result.result == INTERRUPTED:
+ if isinstance(result, Interrupted):
break
if self.bad:
@@ -423,14 +440,14 @@ def run_tests_sequential(self):
result = runtest(self.ns, test_name)
self.accumulate_result(result)
- if result.result == INTERRUPTED:
+ if isinstance(result, Interrupted):
break
- previous_test = format_test_result(result)
+ previous_test = str(result)
test_time = time.monotonic() - start_time
if test_time >= PROGRESS_MIN_TIME:
previous_test = "%s in %s" % (previous_test, format_duration(test_time))
- elif result.result == PASSED:
+ elif isinstance(result, Passed):
# be quiet: say nothing if the test passed shortly
previous_test = None
@@ -516,7 +533,24 @@ def run_tests(self):
if self.ns.use_mp:
from test.libregrtest.runtest_mp import run_tests_multiprocess
- run_tests_multiprocess(self)
+ # If we're on windows and this is the parent runner (not a worker),
+ # track the load average.
+ if sys.platform == 'win32' and self.worker_test_name is None:
+ from test.libregrtest.win_utils import WindowsLoadTracker
+
+ try:
+ self.win_load_tracker = WindowsLoadTracker()
+ except PermissionError as error:
+ # Standard accounts may not have access to the performance
+ # counters.
+ print(f'Failed to create WindowsLoadTracker: {error}')
+
+ try:
+ run_tests_multiprocess(self)
+ finally:
+ if self.win_load_tracker is not None:
+ self.win_load_tracker.close()
+ self.win_load_tracker = None
else:
self.run_tests_sequential()
@@ -678,28 +712,11 @@ def _main(self, tests, kwargs):
self.list_cases()
sys.exit(0)
- # If we're on windows and this is the parent runner (not a worker),
- # track the load average.
- if sys.platform == 'win32' and self.worker_test_name is None:
- from test.libregrtest.win_utils import WindowsLoadTracker
-
- try:
- self.win_load_tracker = WindowsLoadTracker()
- except FileNotFoundError as error:
- # Windows IoT Core and Windows Nano Server do not provide
- # typeperf.exe for x64, x86 or ARM
- print(f'Failed to create WindowsLoadTracker: {error}')
+ self.run_tests()
+ self.display_result()
- try:
- self.run_tests()
- self.display_result()
-
- if self.ns.verbose2 and self.bad:
- self.rerun_failed_tests()
- finally:
- if self.win_load_tracker is not None:
- self.win_load_tracker.close()
- self.win_load_tracker = None
+ if self.ns.verbose2 and self.bad:
+ self.rerun_failed_tests()
self.finalize()
diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py
index 7c7086a806b1cd..b56a84856d14fd 100644
--- a/Lib/test/libregrtest/refleak.py
+++ b/Lib/test/libregrtest/refleak.py
@@ -112,7 +112,7 @@ def get_pooled_int(value):
# These checkers return False on success, True on failure
def check_rc_deltas(deltas):
- # Checker for reference counters and memomry blocks.
+ # Checker for reference counters and memory blocks.
#
# bpo-30776: Try to ignore false positives:
#
diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py
index 927c4706629705..489ab986cb4e5e 100644
--- a/Lib/test/libregrtest/runtest.py
+++ b/Lib/test/libregrtest/runtest.py
@@ -1,4 +1,3 @@
-import collections
import faulthandler
import functools
import gc
@@ -12,33 +11,109 @@
from test import support
from test.support import os_helper
-from test.libregrtest.utils import clear_caches
+from test.libregrtest.cmdline import Namespace
from test.libregrtest.save_env import saved_test_environment
-from test.libregrtest.utils import format_duration, print_warning
-
-
-# Test result constants.
-PASSED = 1
-FAILED = 0
-ENV_CHANGED = -1
-SKIPPED = -2
-RESOURCE_DENIED = -3
-INTERRUPTED = -4
-CHILD_ERROR = -5 # error in a child process
-TEST_DID_NOT_RUN = -6
-TIMEOUT = -7
-
-_FORMAT_TEST_RESULT = {
- PASSED: '%s passed',
- FAILED: '%s failed',
- ENV_CHANGED: '%s failed (env changed)',
- SKIPPED: '%s skipped',
- RESOURCE_DENIED: '%s skipped (resource denied)',
- INTERRUPTED: '%s interrupted',
- CHILD_ERROR: '%s crashed',
- TEST_DID_NOT_RUN: '%s run no tests',
- TIMEOUT: '%s timed out',
-}
+from test.libregrtest.utils import clear_caches, format_duration, print_warning
+
+
+class TestResult:
+ def __init__(
+ self,
+ name: str,
+ duration_sec: float = 0.0,
+ xml_data: list[str] | None = None,
+ ) -> None:
+ self.name = name
+ self.duration_sec = duration_sec
+ self.xml_data = xml_data
+
+ def __str__(self) -> str:
+ return f"{self.name} finished"
+
+
+class Passed(TestResult):
+ def __str__(self) -> str:
+ return f"{self.name} passed"
+
+
+class Failed(TestResult):
+ def __init__(
+ self,
+ name: str,
+ duration_sec: float = 0.0,
+ xml_data: list[str] | None = None,
+ errors: list[tuple[str, str]] | None = None,
+ failures: list[tuple[str, str]] | None = None,
+ ) -> None:
+ super().__init__(name, duration_sec=duration_sec, xml_data=xml_data)
+ self.errors = errors
+ self.failures = failures
+
+ def __str__(self) -> str:
+ if self.errors and self.failures:
+ le = len(self.errors)
+ lf = len(self.failures)
+ error_s = "error" + ("s" if le > 1 else "")
+ failure_s = "failure" + ("s" if lf > 1 else "")
+ return f"{self.name} failed ({le} {error_s}, {lf} {failure_s})"
+
+ if self.errors:
+ le = len(self.errors)
+ error_s = "error" + ("s" if le > 1 else "")
+ return f"{self.name} failed ({le} {error_s})"
+
+ if self.failures:
+ lf = len(self.failures)
+ failure_s = "failure" + ("s" if lf > 1 else "")
+ return f"{self.name} failed ({lf} {failure_s})"
+
+ return f"{self.name} failed"
+
+
+class UncaughtException(Failed):
+ def __str__(self) -> str:
+ return f"{self.name} failed (uncaught exception)"
+
+
+class EnvChanged(Failed):
+ def __str__(self) -> str:
+ return f"{self.name} failed (env changed)"
+
+
+class RefLeak(Failed):
+ def __str__(self) -> str:
+ return f"{self.name} failed (reference leak)"
+
+
+class Skipped(TestResult):
+ def __str__(self) -> str:
+ return f"{self.name} skipped"
+
+
+class ResourceDenied(Skipped):
+ def __str__(self) -> str:
+ return f"{self.name} skipped (resource denied)"
+
+
+class Interrupted(TestResult):
+ def __str__(self) -> str:
+ return f"{self.name} interrupted"
+
+
+class ChildError(Failed):
+ def __str__(self) -> str:
+ return f"{self.name} crashed"
+
+
+class DidNotRun(TestResult):
+ def __str__(self) -> str:
+ return f"{self.name} ran no tests"
+
+
+class Timeout(Failed):
+ def __str__(self) -> str:
+ return f"{self.name} timed out ({format_duration(self.duration_sec)})"
+
# Minimum duration of a test to display its duration or to mention that
# the test is running in background
@@ -67,21 +142,10 @@
FOUND_GARBAGE = []
-def is_failed(result, ns):
- ok = result.result
- if ok in (PASSED, RESOURCE_DENIED, SKIPPED, TEST_DID_NOT_RUN):
- return False
- if ok == ENV_CHANGED:
+def is_failed(result: TestResult, ns: Namespace) -> bool:
+ if isinstance(result, EnvChanged):
return ns.fail_env_changed
- return True
-
-
-def format_test_result(result):
- fmt = _FORMAT_TEST_RESULT.get(result.result, "%s")
- text = fmt % result.test_name
- if result.result == TIMEOUT:
- text = '%s (%s)' % (text, format_duration(result.test_time))
- return text
+ return isinstance(result, Failed)
def findtestdir(path=None):
@@ -101,7 +165,7 @@ def findtests(testdir=None, stdtests=STDTESTS, nottests=NOTTESTS):
return stdtests + sorted(tests)
-def get_abs_module(ns, test_name):
+def get_abs_module(ns: Namespace, test_name: str) -> str:
if test_name.startswith('test.') or ns.testdir:
return test_name
else:
@@ -109,10 +173,7 @@ def get_abs_module(ns, test_name):
return 'test.' + test_name
-TestResult = collections.namedtuple('TestResult',
- 'test_name result test_time xml_data')
-
-def _runtest(ns, test_name):
+def _runtest(ns: Namespace, test_name: str) -> TestResult:
# Handle faulthandler timeout, capture stdout+stderr, XML serialization
# and measure time.
@@ -140,7 +201,7 @@ def _runtest(ns, test_name):
sys.stderr = stream
result = _runtest_inner(ns, test_name,
display_failure=False)
- if result != PASSED:
+ if not isinstance(result, Passed):
output = stream.getvalue()
orig_stderr.write(output)
orig_stderr.flush()
@@ -156,36 +217,26 @@ def _runtest(ns, test_name):
if xml_list:
import xml.etree.ElementTree as ET
- xml_data = [ET.tostring(x).decode('us-ascii') for x in xml_list]
- else:
- xml_data = None
-
- test_time = time.perf_counter() - start_time
+ result.xml_data = [
+ ET.tostring(x).decode('us-ascii')
+ for x in xml_list
+ ]
- return TestResult(test_name, result, test_time, xml_data)
+ result.duration_sec = time.perf_counter() - start_time
+ return result
finally:
if use_timeout:
faulthandler.cancel_dump_traceback_later()
support.junit_xml_list = None
-def runtest(ns, test_name):
+def runtest(ns: Namespace, test_name: str) -> TestResult:
"""Run a single test.
ns -- regrtest namespace of options
test_name -- the name of the test
- Returns the tuple (result, test_time, xml_data), where result is one
- of the constants:
-
- INTERRUPTED KeyboardInterrupt
- RESOURCE_DENIED test skipped because resource denied
- SKIPPED test skipped for some other reason
- ENV_CHANGED test failed because it changed the execution environment
- FAILED test failed
- PASSED test passed
- EMPTY_TEST_SUITE test ran no subtests.
- TIMEOUT test timed out.
+ Returns a TestResult sub-class depending on the kind of result received.
If ns.xmlpath is not None, xml_data is a list containing each
generated testsuite element.
@@ -197,7 +248,7 @@ def runtest(ns, test_name):
msg = traceback.format_exc()
print(f"test {test_name} crashed -- {msg}",
file=sys.stderr, flush=True)
- return TestResult(test_name, FAILED, 0.0, None)
+ return Failed(test_name)
def _test_module(the_module):
@@ -210,11 +261,11 @@ def _test_module(the_module):
support.run_unittest(tests)
-def save_env(ns, test_name):
+def save_env(ns: Namespace, test_name: str):
return saved_test_environment(test_name, ns.verbose, ns.quiet, pgo=ns.pgo)
-def _runtest_inner2(ns, test_name):
+def _runtest_inner2(ns: Namespace, test_name: str) -> bool:
# Load the test function, run the test function, handle huntrleaks
# and findleaks to detect leaks
@@ -265,7 +316,9 @@ def _runtest_inner2(ns, test_name):
return refleak
-def _runtest_inner(ns, test_name, display_failure=True):
+def _runtest_inner(
+ ns: Namespace, test_name: str, display_failure: bool = True
+) -> TestResult:
# Detect environment changes, handle exceptions.
# Reset the environment_altered flag to detect if a test altered
@@ -283,37 +336,43 @@ def _runtest_inner(ns, test_name, display_failure=True):
except support.ResourceDenied as msg:
if not ns.quiet and not ns.pgo:
print(f"{test_name} skipped -- {msg}", flush=True)
- return RESOURCE_DENIED
+ return ResourceDenied(test_name)
except unittest.SkipTest as msg:
if not ns.quiet and not ns.pgo:
print(f"{test_name} skipped -- {msg}", flush=True)
- return SKIPPED
+ return Skipped(test_name)
+ except support.TestFailedWithDetails as exc:
+ msg = f"test {test_name} failed"
+ if display_failure:
+ msg = f"{msg} -- {exc}"
+ print(msg, file=sys.stderr, flush=True)
+ return Failed(test_name, errors=exc.errors, failures=exc.failures)
except support.TestFailed as exc:
msg = f"test {test_name} failed"
if display_failure:
msg = f"{msg} -- {exc}"
print(msg, file=sys.stderr, flush=True)
- return FAILED
+ return Failed(test_name)
except support.TestDidNotRun:
- return TEST_DID_NOT_RUN
+ return DidNotRun(test_name)
except KeyboardInterrupt:
print()
- return INTERRUPTED
+ return Interrupted(test_name)
except:
if not ns.pgo:
msg = traceback.format_exc()
print(f"test {test_name} crashed -- {msg}",
file=sys.stderr, flush=True)
- return FAILED
+ return UncaughtException(test_name)
if refleak:
- return FAILED
+ return RefLeak(test_name)
if support.environment_altered:
- return ENV_CHANGED
- return PASSED
+ return EnvChanged(test_name)
+ return Passed(test_name)
-def cleanup_test_droppings(test_name, verbose):
+def cleanup_test_droppings(test_name: str, verbose: int) -> None:
# First kill any dangling references to open files etc.
# This can also issue some ResourceWarnings which would otherwise get
# triggered during the following test run, and possibly produce failures.
diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 3d503af23b6a76..d0d63ef8afe91a 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -9,13 +9,15 @@
import threading
import time
import traceback
-import types
+from typing import NamedTuple, NoReturn, Literal, Any
+
from test import support
from test.support import os_helper
+from test.libregrtest.cmdline import Namespace
+from test.libregrtest.main import Regrtest
from test.libregrtest.runtest import (
- runtest, INTERRUPTED, CHILD_ERROR, PROGRESS_MIN_TIME,
- format_test_result, TestResult, is_failed, TIMEOUT)
+ runtest, is_failed, TestResult, Interrupted, Timeout, ChildError, PROGRESS_MIN_TIME)
from test.libregrtest.setup import setup_tests
from test.libregrtest.utils import format_duration, print_warning
@@ -36,21 +38,21 @@
USE_PROCESS_GROUP = (hasattr(os, "setsid") and hasattr(os, "killpg"))
-def must_stop(result, ns):
- if result.result == INTERRUPTED:
+def must_stop(result: TestResult, ns: Namespace) -> bool:
+ if isinstance(result, Interrupted):
return True
if ns.failfast and is_failed(result, ns):
return True
return False
-def parse_worker_args(worker_args):
+def parse_worker_args(worker_args) -> tuple[Namespace, str]:
ns_dict, test_name = json.loads(worker_args)
- ns = types.SimpleNamespace(**ns_dict)
+ ns = Namespace(**ns_dict)
return (ns, test_name)
-def run_test_in_subprocess(testname, ns):
+def run_test_in_subprocess(testname: str, ns: Namespace) -> subprocess.Popen:
ns_dict = vars(ns)
worker_args = (ns_dict, testname)
worker_args = json.dumps(worker_args)
@@ -75,15 +77,15 @@ def run_test_in_subprocess(testname, ns):
**kw)
-def run_tests_worker(ns, test_name):
+def run_tests_worker(ns: Namespace, test_name: str) -> NoReturn:
setup_tests(ns)
result = runtest(ns, test_name)
print() # Force a newline (just in case)
- # Serialize TestResult as list in JSON
- print(json.dumps(list(result)), flush=True)
+ # Serialize TestResult as dict in JSON
+ print(json.dumps(result, cls=EncodeTestResult), flush=True)
sys.exit(0)
@@ -110,15 +112,23 @@ def stop(self):
self.tests_iter = None
-MultiprocessResult = collections.namedtuple('MultiprocessResult',
- 'result stdout stderr error_msg')
+class MultiprocessResult(NamedTuple):
+ result: TestResult
+ stdout: str
+ stderr: str
+ error_msg: str
+
+
+ExcStr = str
+QueueOutput = tuple[Literal[False], MultiprocessResult] | tuple[Literal[True], ExcStr]
+
class ExitThread(Exception):
pass
class TestWorkerProcess(threading.Thread):
- def __init__(self, worker_id, runner):
+ def __init__(self, worker_id: int, runner: "MultiprocessTestRunner") -> None:
super().__init__()
self.worker_id = worker_id
self.pending = runner.pending
@@ -132,7 +142,7 @@ def __init__(self, worker_id, runner):
self._killed = False
self._stopped = False
- def __repr__(self):
+ def __repr__(self) -> str:
info = [f'TestWorkerProcess #{self.worker_id}']
if self.is_alive():
info.append("running")
@@ -148,7 +158,7 @@ def __repr__(self):
f'time={format_duration(dt)}'))
return '<%s>' % ' '.join(info)
- def _kill(self):
+ def _kill(self) -> None:
popen = self._popen
if popen is None:
return
@@ -176,18 +186,22 @@ def _kill(self):
except OSError as exc:
print_warning(f"Failed to kill {what}: {exc!r}")
- def stop(self):
+ def stop(self) -> None:
# Method called from a different thread to stop this thread
self._stopped = True
self._kill()
- def mp_result_error(self, test_name, error_type, stdout='', stderr='',
- err_msg=None):
- test_time = time.monotonic() - self.start_time
- result = TestResult(test_name, error_type, test_time, None)
- return MultiprocessResult(result, stdout, stderr, err_msg)
-
- def _run_process(self, test_name):
+ def mp_result_error(
+ self,
+ test_result: TestResult,
+ stdout: str = '',
+ stderr: str = '',
+ err_msg=None
+ ) -> MultiprocessResult:
+ test_result.duration_sec = time.monotonic() - self.start_time
+ return MultiprocessResult(test_result, stdout, stderr, err_msg)
+
+ def _run_process(self, test_name: str) -> tuple[int, str, str]:
self.start_time = time.monotonic()
self.current_test_name = test_name
@@ -246,11 +260,11 @@ def _run_process(self, test_name):
self._popen = None
self.current_test_name = None
- def _runtest(self, test_name):
+ def _runtest(self, test_name: str) -> MultiprocessResult:
retcode, stdout, stderr = self._run_process(test_name)
if retcode is None:
- return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
+ return self.mp_result_error(Timeout(test_name), stdout, stderr)
err_msg = None
if retcode != 0:
@@ -263,18 +277,17 @@ def _runtest(self, test_name):
else:
try:
# deserialize run_tests_worker() output
- result = json.loads(result)
- result = TestResult(*result)
+ result = json.loads(result, object_hook=decode_test_result)
except Exception as exc:
err_msg = "Failed to parse worker JSON: %s" % exc
if err_msg is not None:
- return self.mp_result_error(test_name, CHILD_ERROR,
+ return self.mp_result_error(ChildError(test_name),
stdout, stderr, err_msg)
return MultiprocessResult(result, stdout, stderr, err_msg)
- def run(self):
+ def run(self) -> None:
while not self._stopped:
try:
try:
@@ -293,7 +306,7 @@ def run(self):
self.output.put((True, traceback.format_exc()))
break
- def _wait_completed(self):
+ def _wait_completed(self) -> None:
popen = self._popen
# stdout and stderr must be closed to ensure that communicate()
@@ -308,7 +321,7 @@ def _wait_completed(self):
f"(timeout={format_duration(JOIN_TIMEOUT)}): "
f"{exc!r}")
- def wait_stopped(self, start_time):
+ def wait_stopped(self, start_time: float) -> None:
# bpo-38207: MultiprocessTestRunner.stop_workers() called self.stop()
# which killed the process. Sometimes, killing the process from the
# main thread does not interrupt popen.communicate() in
@@ -332,7 +345,7 @@ def wait_stopped(self, start_time):
break
-def get_running(workers):
+def get_running(workers: list[TestWorkerProcess]) -> list[TestWorkerProcess]:
running = []
for worker in workers:
current_test_name = worker.current_test_name
@@ -346,11 +359,11 @@ def get_running(workers):
class MultiprocessTestRunner:
- def __init__(self, regrtest):
+ def __init__(self, regrtest: Regrtest) -> None:
self.regrtest = regrtest
self.log = self.regrtest.log
self.ns = regrtest.ns
- self.output = queue.Queue()
+ self.output: queue.Queue[QueueOutput] = queue.Queue()
self.pending = MultiprocessIterator(self.regrtest.tests)
if self.ns.timeout is not None:
# Rely on faulthandler to kill a worker process. This timouet is
@@ -362,7 +375,7 @@ def __init__(self, regrtest):
self.worker_timeout = None
self.workers = None
- def start_workers(self):
+ def start_workers(self) -> None:
self.workers = [TestWorkerProcess(index, self)
for index in range(1, self.ns.use_mp + 1)]
msg = f"Run tests in parallel using {len(self.workers)} child processes"
@@ -374,24 +387,20 @@ def start_workers(self):
for worker in self.workers:
worker.start()
- def stop_workers(self):
+ def stop_workers(self) -> None:
start_time = time.monotonic()
for worker in self.workers:
worker.stop()
for worker in self.workers:
worker.wait_stopped(start_time)
- def _get_result(self):
- if not any(worker.is_alive() for worker in self.workers):
- # all worker threads are done: consume pending results
- try:
- return self.output.get(timeout=0)
- except queue.Empty:
- return None
-
+ def _get_result(self) -> QueueOutput | None:
use_faulthandler = (self.ns.timeout is not None)
timeout = PROGRESS_UPDATE
- while True:
+
+ # bpo-46205: check the status of workers every iteration to avoid
+ # waiting forever on an empty queue.
+ while any(worker.is_alive() for worker in self.workers):
if use_faulthandler:
faulthandler.dump_traceback_later(MAIN_PROCESS_TIMEOUT,
exit=True)
@@ -407,21 +416,28 @@ def _get_result(self):
if running and not self.ns.pgo:
self.log('running: %s' % ', '.join(running))
- def display_result(self, mp_result):
+ # all worker threads are done: consume pending results
+ try:
+ return self.output.get(timeout=0)
+ except queue.Empty:
+ return None
+
+ def display_result(self, mp_result: MultiprocessResult) -> None:
result = mp_result.result
- text = format_test_result(result)
+ text = str(result)
if mp_result.error_msg is not None:
# CHILD_ERROR
text += ' (%s)' % mp_result.error_msg
- elif (result.test_time >= PROGRESS_MIN_TIME and not self.ns.pgo):
- text += ' (%s)' % format_duration(result.test_time)
+ elif (result.duration_sec >= PROGRESS_MIN_TIME and not self.ns.pgo):
+ text += ' (%s)' % format_duration(result.duration_sec)
running = get_running(self.workers)
if running and not self.ns.pgo:
text += ' -- running: %s' % ', '.join(running)
self.regrtest.display_progress(self.test_index, text)
- def _process_result(self, item):
+ def _process_result(self, item: QueueOutput) -> bool:
+ """Returns True if test runner must stop."""
if item[0]:
# Thread got an exception
format_exc = item[1]
@@ -443,7 +459,7 @@ def _process_result(self, item):
return False
- def run_tests(self):
+ def run_tests(self) -> None:
self.start_workers()
self.test_index = 0
@@ -469,5 +485,41 @@ def run_tests(self):
self.stop_workers()
-def run_tests_multiprocess(regrtest):
+def run_tests_multiprocess(regrtest: Regrtest) -> None:
MultiprocessTestRunner(regrtest).run_tests()
+
+
+class EncodeTestResult(json.JSONEncoder):
+ """Encode a TestResult (sub)class object into a JSON dict."""
+
+ def default(self, o: Any) -> dict[str, Any]:
+ if isinstance(o, TestResult):
+ result = vars(o)
+ result["__test_result__"] = o.__class__.__name__
+ return result
+
+ return super().default(o)
+
+
+def decode_test_result(d: dict[str, Any]) -> TestResult | dict[str, Any]:
+ """Decode a TestResult (sub)class object from a JSON dict."""
+
+ if "__test_result__" not in d:
+ return d
+
+ cls_name = d.pop("__test_result__")
+ for cls in get_all_test_result_classes():
+ if cls.__name__ == cls_name:
+ return cls(**d)
+
+
+def get_all_test_result_classes() -> set[type[TestResult]]:
+ prev_count = 0
+ classes = {TestResult}
+ while len(classes) > prev_count:
+ prev_count = len(classes)
+ to_add = []
+ for cls in classes:
+ to_add.extend(cls.__subclasses__())
+ classes.update(to_add)
+ return classes
diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py
index 83ce2f73f4e5a6..964c25e39700f4 100644
--- a/Lib/test/libregrtest/setup.py
+++ b/Lib/test/libregrtest/setup.py
@@ -5,6 +5,7 @@
import sys
import unittest
from test import support
+from test.support.os_helper import TESTFN_UNDECODABLE, FS_NONASCII
try:
import gc
except ImportError:
@@ -14,6 +15,9 @@
setup_threading_excepthook)
+UNICODE_GUARD_ENV = "PYTHONREGRTEST_UNICODE_GUARD"
+
+
def setup_tests(ns):
try:
stderr_fd = sys.__stderr__.fileno()
@@ -36,6 +40,7 @@ def setup_tests(ns):
for signum in signals:
faulthandler.register(signum, chain=True, file=stderr_fd)
+ _adjust_resource_limits()
replace_stdout()
support.record_original_stdout(sys.stdout)
@@ -99,6 +104,13 @@ def _test_audit_hook(name, args):
from test.support.testresult import RegressionTestResult
RegressionTestResult.USE_XML = True
+ # Ensure there's a non-ASCII character in env vars at all times to force
+ # tests consider this case. See BPO-44647 for details.
+ if TESTFN_UNDECODABLE and os.supports_bytes_environ:
+ os.environb.setdefault(UNICODE_GUARD_ENV.encode(), TESTFN_UNDECODABLE)
+ elif FS_NONASCII:
+ os.environ.setdefault(UNICODE_GUARD_ENV, FS_NONASCII)
+
def replace_stdout():
"""Set stdout encoder error handler to backslashreplace (as stderr error
@@ -124,3 +136,25 @@ def restore_stdout():
sys.stdout.close()
sys.stdout = stdout
atexit.register(restore_stdout)
+
+
+def _adjust_resource_limits():
+ """Adjust the system resource limits (ulimit) if needed."""
+ try:
+ import resource
+ from resource import RLIMIT_NOFILE, RLIM_INFINITY
+ except ImportError:
+ return
+ fd_limit, max_fds = resource.getrlimit(RLIMIT_NOFILE)
+ # On macOS the default fd limit is sometimes too low (256) for our
+ # test suite to succeed. Raise it to something more reasonable.
+ # 1024 is a common Linux default.
+ desired_fds = 1024
+ if fd_limit < desired_fds and fd_limit < max_fds:
+ new_fd_limit = min(desired_fds, max_fds)
+ try:
+ resource.setrlimit(RLIMIT_NOFILE, (new_fd_limit, max_fds))
+ print(f"Raised RLIMIT_NOFILE: {fd_limit} -> {new_fd_limit}")
+ except (ValueError, OSError) as err:
+ print(f"Unable to raise RLIMIT_NOFILE from {fd_limit} to "
+ f"{new_fd_limit}: {err}.")
diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index a1cc2201147dbc..5736cdfd3c7292 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -1,16 +1,11 @@
+import _overlapped
+import _thread
import _winapi
import math
-import msvcrt
-import os
-import subprocess
-import uuid
+import struct
import winreg
-from test.support import os_helper
-from test.libregrtest.utils import print_warning
-# Max size of asynchronous reads
-BUFSIZE = 8192
# Seconds per measurement
SAMPLING_INTERVAL = 1
# Exponential damping factor to compute exponentially weighted moving average
@@ -19,174 +14,111 @@
# Initialize the load using the arithmetic mean of the first NVALUE values
# of the Processor Queue Length
NVALUE = 5
-# Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
-# of typeperf are registered
-COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
- r"\Perflib\CurrentLanguage")
class WindowsLoadTracker():
"""
- This class asynchronously interacts with the `typeperf` command to read
- the system load on Windows. Multiprocessing and threads can't be used
- here because they interfere with the test suite's cases for those
- modules.
+ This class asynchronously reads the performance counters to calculate
+ the system load on Windows. A "raw" thread is used here to prevent
+ interference with the test suite's cases for the threading module.
"""
def __init__(self):
+ # Pre-flight test for access to the performance data;
+ # `PermissionError` will be raised if not allowed
+ winreg.QueryInfoKey(winreg.HKEY_PERFORMANCE_DATA)
+
self._values = []
self._load = None
- self._buffer = ''
- self._popen = None
- self.start()
-
- def start(self):
- # Create a named pipe which allows for asynchronous IO in Windows
- pipe_name = r'\\.\pipe\typeperf_output_' + str(uuid.uuid4())
-
- open_mode = _winapi.PIPE_ACCESS_INBOUND
- open_mode |= _winapi.FILE_FLAG_FIRST_PIPE_INSTANCE
- open_mode |= _winapi.FILE_FLAG_OVERLAPPED
-
- # This is the read end of the pipe, where we will be grabbing output
- self.pipe = _winapi.CreateNamedPipe(
- pipe_name, open_mode, _winapi.PIPE_WAIT,
- 1, BUFSIZE, BUFSIZE, _winapi.NMPWAIT_WAIT_FOREVER, _winapi.NULL
- )
- # The write end of the pipe which is passed to the created process
- pipe_write_end = _winapi.CreateFile(
- pipe_name, _winapi.GENERIC_WRITE, 0, _winapi.NULL,
- _winapi.OPEN_EXISTING, 0, _winapi.NULL
- )
- # Open up the handle as a python file object so we can pass it to
- # subprocess
- command_stdout = msvcrt.open_osfhandle(pipe_write_end, 0)
-
- # Connect to the read end of the pipe in overlap/async mode
- overlap = _winapi.ConnectNamedPipe(self.pipe, overlapped=True)
- overlap.GetOverlappedResult(True)
-
- # Spawn off the load monitor
- counter_name = self._get_counter_name()
- command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)]
- self._popen = subprocess.Popen(' '.join(command),
- stdout=command_stdout,
- cwd=os_helper.SAVEDCWD)
-
- # Close our copy of the write end of the pipe
- os.close(command_stdout)
-
- def _get_counter_name(self):
- # accessing the registry to get the counter localization name
- with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, COUNTER_REGISTRY_KEY) as perfkey:
- counters = winreg.QueryValueEx(perfkey, 'Counter')[0]
-
- # Convert [key1, value1, key2, value2, ...] list
- # to {key1: value1, key2: value2, ...} dict
- counters = iter(counters)
- counters_dict = dict(zip(counters, counters))
-
- # System counter has key '2' and Processor Queue Length has key '44'
- system = counters_dict['2']
- process_queue_length = counters_dict['44']
- return f'"\\{system}\\{process_queue_length}"'
-
- def close(self, kill=True):
- if self._popen is None:
+ self._running = _overlapped.CreateEvent(None, True, False, None)
+ self._stopped = _overlapped.CreateEvent(None, True, False, None)
+
+ _thread.start_new_thread(self._update_load, (), {})
+
+ def _update_load(self,
+ # localize module access to prevent shutdown errors
+ _wait=_winapi.WaitForSingleObject,
+ _signal=_overlapped.SetEvent):
+ # run until signaled to stop
+ while _wait(self._running, 1000):
+ self._calculate_load()
+ # notify stopped
+ _signal(self._stopped)
+
+ def _calculate_load(self,
+ # localize module access to prevent shutdown errors
+ _query=winreg.QueryValueEx,
+ _hkey=winreg.HKEY_PERFORMANCE_DATA,
+ _unpack=struct.unpack_from):
+ # get the 'System' object
+ data, _ = _query(_hkey, '2')
+ # PERF_DATA_BLOCK {
+ # WCHAR Signature[4] 8 +
+ # DWOWD LittleEndian 4 +
+ # DWORD Version 4 +
+ # DWORD Revision 4 +
+ # DWORD TotalByteLength 4 +
+ # DWORD HeaderLength = 24 byte offset
+ # ...
+ # }
+ obj_start, = _unpack('L', data, 24)
+ # PERF_OBJECT_TYPE {
+ # DWORD TotalByteLength
+ # DWORD DefinitionLength
+ # DWORD HeaderLength
+ # ...
+ # }
+ data_start, defn_start = _unpack('4xLL', data, obj_start)
+ data_base = obj_start + data_start
+ defn_base = obj_start + defn_start
+ # find the 'Processor Queue Length' counter (index=44)
+ while defn_base < data_base:
+ # PERF_COUNTER_DEFINITION {
+ # DWORD ByteLength
+ # DWORD CounterNameTitleIndex
+ # ... [7 DWORDs/28 bytes]
+ # DWORD CounterOffset
+ # }
+ size, idx, offset = _unpack('LL28xL', data, defn_base)
+ defn_base += size
+ if idx == 44:
+ counter_offset = data_base + offset
+ # the counter is known to be PERF_COUNTER_RAWCOUNT (DWORD)
+ processor_queue_length, = _unpack('L', data, counter_offset)
+ break
+ else:
return
- self._load = None
-
- if kill:
- self._popen.kill()
- self._popen.wait()
- self._popen = None
-
- def __del__(self):
- self.close()
-
- def _parse_line(self, line):
- # typeperf outputs in a CSV format like this:
- # "07/19/2018 01:32:26.605","3.000000"
- # (date, process queue length)
- tokens = line.split(',')
- if len(tokens) != 2:
- raise ValueError
-
- value = tokens[1]
- if not value.startswith('"') or not value.endswith('"'):
- raise ValueError
- value = value[1:-1]
- return float(value)
-
- def _read_lines(self):
- overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
- bytes_read, res = overlapped.GetOverlappedResult(False)
- if res != 0:
- return ()
-
- output = overlapped.getbuffer()
- output = output.decode('oem', 'replace')
- output = self._buffer + output
- lines = output.splitlines(True)
-
- # bpo-36670: typeperf only writes a newline *before* writing a value,
- # not after. Sometimes, the written line in incomplete (ex: only
- # timestamp, without the process queue length). Only pass the last line
- # to the parser if it's a valid value, otherwise store it in
- # self._buffer.
- try:
- self._parse_line(lines[-1])
- except ValueError:
- self._buffer = lines.pop(-1)
+ # We use an exponentially weighted moving average, imitating the
+ # load calculation on Unix systems.
+ # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation
+ # https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
+ if self._load is not None:
+ self._load = (self._load * LOAD_FACTOR_1
+ + processor_queue_length * (1.0 - LOAD_FACTOR_1))
+ elif len(self._values) < NVALUE:
+ self._values.append(processor_queue_length)
else:
- self._buffer = ''
+ self._load = sum(self._values) / len(self._values)
- return lines
+ def close(self, kill=True):
+ self.__del__()
+ return
+
+ def __del__(self,
+ # localize module access to prevent shutdown errors
+ _wait=_winapi.WaitForSingleObject,
+ _close=_winapi.CloseHandle,
+ _signal=_overlapped.SetEvent):
+ if self._running is not None:
+ # tell the update thread to quit
+ _signal(self._running)
+ # wait for the update thread to signal done
+ _wait(self._stopped, -1)
+ # cleanup events
+ _close(self._running)
+ _close(self._stopped)
+ self._running = self._stopped = None
def getloadavg(self):
- if self._popen is None:
- return None
-
- returncode = self._popen.poll()
- if returncode is not None:
- self.close(kill=False)
- return None
-
- try:
- lines = self._read_lines()
- except BrokenPipeError:
- self.close()
- return None
-
- for line in lines:
- line = line.rstrip()
-
- # Ignore the initial header:
- # "(PDH-CSV 4.0)","\\\\WIN\\System\\Processor Queue Length"
- if 'PDH-CSV' in line:
- continue
-
- # Ignore blank lines
- if not line:
- continue
-
- try:
- processor_queue_length = self._parse_line(line)
- except ValueError:
- print_warning("Failed to parse typeperf output: %a" % line)
- continue
-
- # We use an exponentially weighted moving average, imitating the
- # load calculation on Unix systems.
- # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation
- # https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
- if self._load is not None:
- self._load = (self._load * LOAD_FACTOR_1
- + processor_queue_length * (1.0 - LOAD_FACTOR_1))
- elif len(self._values) < NVALUE:
- self._values.append(processor_queue_length)
- else:
- self._load = sum(self._values) / len(self._values)
-
return self._load
diff --git a/Lib/test/lock_tests.py b/Lib/test/lock_tests.py
index d69bcc9496843f..dffb7d4418dfe6 100644
--- a/Lib/test/lock_tests.py
+++ b/Lib/test/lock_tests.py
@@ -3,6 +3,7 @@
"""
import os
+import gc
import sys
import time
from _thread import start_new_thread, TIMEOUT_MAX
@@ -221,6 +222,7 @@ def test_weakref_deleted(self):
lock = self.locktype()
ref = weakref.ref(lock)
del lock
+ gc.collect() # For PyPy or other GCs.
self.assertIsNone(ref())
diff --git a/Lib/test/make_ssl_certs.py b/Lib/test/make_ssl_certs.py
index c62896f861dab6..94a35a64ab1abe 100644
--- a/Lib/test/make_ssl_certs.py
+++ b/Lib/test/make_ssl_certs.py
@@ -307,6 +307,6 @@ def print_cert(path):
f.write(cert)
unmake_ca()
- print("update Lib/test/test_ssl.py and Lib/test/test_asyncio/util.py")
+ print("update Lib/test/test_ssl.py and Lib/test/test_asyncio/utils.py")
print_cert('keycert.pem')
print_cert('keycert3.pem')
diff --git a/Lib/test/mime.types b/Lib/test/mime.types
index 68724430c8a3fd..eb39a17b6bf4b2 100644
--- a/Lib/test/mime.types
+++ b/Lib/test/mime.types
@@ -1,11 +1,11 @@
# This is a comment. I love comments. -*- indent-tabs-mode: t -*-
-# This file controls what Internet media types are sent to the client for
+# This file controls what internet media types are sent to the client for
# given file extension(s). Sending the correct media type to the client
# is important so they know how to handle the content of the file.
# Extra types can either be added here or by using an AddType directive
-# in your config files. For more information about Internet media types,
-# please read RFC 2045, 2046, 2047, 2048, and 2077. The Internet media type
+# in your config files. For more information about internet media types,
+# please read RFC 2045, 2046, 2047, 2048, and 2077. The internet media type
# registry is at .
# IANA types
diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py
index 8e01d311360e35..72d6f52c90de7b 100644
--- a/Lib/test/pickletester.py
+++ b/Lib/test/pickletester.py
@@ -1,3 +1,4 @@
+import builtins
import collections
import copyreg
import dbm
@@ -11,6 +12,7 @@
import struct
import sys
import threading
+import types
import unittest
import weakref
from textwrap import dedent
@@ -828,7 +830,7 @@ def create_data():
return x
-class AbstractUnpickleTests(unittest.TestCase):
+class AbstractUnpickleTests:
# Subclass must define self.loads.
_testdata = create_data()
@@ -1441,7 +1443,7 @@ def t():
-class AbstractPickleTests(unittest.TestCase):
+class AbstractPickleTests:
# Subclass must define self.dumps, self.loads.
optimized = False
@@ -1979,6 +1981,33 @@ def test_singleton_types(self):
u = self.loads(s)
self.assertIs(type(singleton), u)
+ def test_builtin_types(self):
+ for t in builtins.__dict__.values():
+ if isinstance(t, type) and not issubclass(t, BaseException):
+ for proto in protocols:
+ s = self.dumps(t, proto)
+ self.assertIs(self.loads(s), t)
+
+ def test_builtin_exceptions(self):
+ for t in builtins.__dict__.values():
+ if isinstance(t, type) and issubclass(t, BaseException):
+ for proto in protocols:
+ s = self.dumps(t, proto)
+ u = self.loads(s)
+ if proto <= 2 and issubclass(t, OSError) and t is not BlockingIOError:
+ self.assertIs(u, OSError)
+ elif proto <= 2 and issubclass(t, ImportError):
+ self.assertIs(u, ImportError)
+ else:
+ self.assertIs(u, t)
+
+ def test_builtin_functions(self):
+ for t in builtins.__dict__.values():
+ if isinstance(t, types.BuiltinFunctionType):
+ for proto in protocols:
+ s = self.dumps(t, proto)
+ self.assertIs(self.loads(s), t)
+
# Tests for protocol 2
def test_proto(self):
@@ -2383,7 +2412,8 @@ def test_bad_getattr(self):
# Issue #3514: crash when there is an infinite loop in __getattr__
x = BadGetattr()
for proto in protocols:
- self.assertRaises(RuntimeError, self.dumps, x, proto)
+ with support.infinite_recursion():
+ self.assertRaises(RuntimeError, self.dumps, x, proto)
def test_reduce_bad_iterator(self):
# Issue4176: crash when 4th and 5th items of __reduce__()
@@ -2772,6 +2802,15 @@ def pie(self):
unpickled = self.loads(self.dumps(method, proto))
self.assertEqual(method(obj), unpickled(obj))
+ descriptors = (
+ PyMethodsTest.__dict__['cheese'], # static method descriptor
+ PyMethodsTest.__dict__['wine'], # class method descriptor
+ )
+ for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+ for descr in descriptors:
+ with self.subTest(proto=proto, descr=descr):
+ self.assertRaises(TypeError, self.dumps, descr, proto)
+
def test_c_methods(self):
global Subclass
class Subclass(tuple):
@@ -2807,6 +2846,15 @@ class Nested(str):
unpickled = self.loads(self.dumps(method, proto))
self.assertEqual(method(*args), unpickled(*args))
+ descriptors = (
+ bytearray.__dict__['maketrans'], # built-in static method descriptor
+ dict.__dict__['fromkeys'], # built-in class method descriptor
+ )
+ for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+ for descr in descriptors:
+ with self.subTest(proto=proto, descr=descr):
+ self.assertRaises(TypeError, self.dumps, descr, proto)
+
def test_compat_pickle(self):
tests = [
(range(1, 7), '__builtin__', 'xrange'),
@@ -3031,8 +3079,69 @@ def check_array(arr):
# 2-D, non-contiguous
check_array(arr[::2])
+ def test_evil_class_mutating_dict(self):
+ # https://github.com/python/cpython/issues/92930
+ from random import getrandbits
+
+ global Bad
+ class Bad:
+ def __eq__(self, other):
+ return ENABLED
+ def __hash__(self):
+ return 42
+ def __reduce__(self):
+ if getrandbits(6) == 0:
+ collection.clear()
+ return (Bad, ())
+
+ for proto in protocols:
+ for _ in range(20):
+ ENABLED = False
+ collection = {Bad(): Bad() for _ in range(20)}
+ for bad in collection:
+ bad.bad = bad
+ bad.collection = collection
+ ENABLED = True
+ try:
+ data = self.dumps(collection, proto)
+ self.loads(data)
+ except RuntimeError as e:
+ expected = "changed size during iteration"
+ self.assertIn(expected, str(e))
+
+ def test_evil_pickler_mutating_collection(self):
+ # https://github.com/python/cpython/issues/92930
+ if not hasattr(self, "pickler"):
+ raise self.skipTest(f"{type(self)} has no associated pickler type")
+
+ global Clearer
+ class Clearer:
+ pass
+
+ def check(collection):
+ class EvilPickler(self.pickler):
+ def persistent_id(self, obj):
+ if isinstance(obj, Clearer):
+ collection.clear()
+ return None
+ pickler = EvilPickler(io.BytesIO(), proto)
+ try:
+ pickler.dump(collection)
+ except RuntimeError as e:
+ expected = "changed size during iteration"
+ self.assertIn(expected, str(e))
+
+ for proto in protocols:
+ check([Clearer()])
+ check([Clearer(), Clearer()])
+ check({Clearer()})
+ check({Clearer(), Clearer()})
+ check({Clearer(): 1})
+ check({Clearer(): 1, Clearer(): 2})
+ check({1: Clearer(), 2: Clearer()})
+
-class BigmemPickleTests(unittest.TestCase):
+class BigmemPickleTests:
# Binary protocols can serialize longs of up to 2 GiB-1
@@ -3305,7 +3414,7 @@ def __getattr__(self, key):
self.foo
-class AbstractPickleModuleTests(unittest.TestCase):
+class AbstractPickleModuleTests:
def test_dump_closed_file(self):
f = open(TESTFN, "wb")
@@ -3412,7 +3521,7 @@ def loads(data, **kwargs):
self.check_dumps_loads_oob_buffers(dumps, loads)
-class AbstractPersistentPicklerTests(unittest.TestCase):
+class AbstractPersistentPicklerTests:
# This class defines persistent_id() and persistent_load()
# functions that should be used by the pickler. All even integers
@@ -3452,7 +3561,7 @@ def test_persistence(self):
self.assertEqual(self.load_false_count, 1)
-class AbstractIdentityPersistentPicklerTests(unittest.TestCase):
+class AbstractIdentityPersistentPicklerTests:
def persistent_id(self, obj):
return obj
@@ -3481,7 +3590,7 @@ def test_protocol0_is_ascii_only(self):
self.assertRaises(pickle.UnpicklingError, self.loads, pickled)
-class AbstractPicklerUnpicklerObjectTests(unittest.TestCase):
+class AbstractPicklerUnpicklerObjectTests:
pickler_class = None
unpickler_class = None
@@ -3695,7 +3804,7 @@ def reducer_override(self, obj):
return NotImplemented
-class AbstractHookTests(unittest.TestCase):
+class AbstractHookTests:
def test_pickler_hook(self):
# test the ability of a custom, user-defined CPickler subclass to
# override the default reducing routines of any type using the method
@@ -3723,7 +3832,7 @@ class MyClass:
self.assertEqual(new_f, 5)
self.assertEqual(some_str, 'some str')
- # math.log does not have its usual reducer overriden, so the
+ # math.log does not have its usual reducer overridden, so the
# custom reduction callback should silently direct the pickler
# to the default pickling by attribute, by returning
# NotImplemented
@@ -3740,7 +3849,7 @@ class MyClass:
def test_reducer_override_no_reference_cycle(self):
# bpo-39492: reducer_override used to induce a spurious reference cycle
# inside the Pickler object, that could prevent all serialized objects
- # from being garbage-collected without explicity invoking gc.collect.
+ # from being garbage-collected without explicitly invoking gc.collect.
for proto in range(0, pickle.HIGHEST_PROTOCOL + 1):
with self.subTest(proto=proto):
@@ -3761,7 +3870,7 @@ def f():
self.assertIsNone(wr())
-class AbstractDispatchTableTests(unittest.TestCase):
+class AbstractDispatchTableTests:
def test_default_dispatch_table(self):
# No dispatch_table attribute by default
diff --git a/Lib/test/pydoc_mod.py b/Lib/test/pydoc_mod.py
index 9c1fff5c2f2c14..f9bc4b89d3d0a4 100644
--- a/Lib/test/pydoc_mod.py
+++ b/Lib/test/pydoc_mod.py
@@ -1,5 +1,8 @@
"""This is a test module for test_pydoc"""
+import types
+import typing
+
__author__ = "Benjamin Peterson"
__credits__ = "Nobody"
__version__ = "1.2.3.4"
@@ -24,6 +27,8 @@ def get_answer(self):
def is_it_true(self):
""" Return self.get_answer() """
return self.get_answer()
+ def __class_getitem__(self, item):
+ return types.GenericAlias(self, item)
def doc_func():
"""
@@ -35,3 +40,10 @@ def doc_func():
def nodoc_func():
pass
+
+
+list_alias1 = typing.List[int]
+list_alias2 = list[int]
+c_alias = C[int]
+type_union1 = typing.Union[int, str]
+type_union2 = int | str
diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index 278dfe7f7da7ac..eef34f08121811 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -9,6 +9,9 @@
import warnings
+MS_WINDOWS = (sys.platform == 'win32')
+
+
def normalize_text(text):
if text is None:
return None
@@ -96,6 +99,7 @@ def collect_sys(info_add):
'maxunicode',
'path',
'platform',
+ 'platlibdir',
'prefix',
'thread_info',
'version',
@@ -124,13 +128,21 @@ def collect_sys(info_add):
encoding = '%s/%s' % (encoding, errors)
info_add('sys.%s.encoding' % name, encoding)
- # Were we compiled --with-pydebug or with #define Py_DEBUG?
+ # Were we compiled --with-pydebug?
Py_DEBUG = hasattr(sys, 'gettotalrefcount')
if Py_DEBUG:
text = 'Yes (sys.gettotalrefcount() present)'
else:
text = 'No (sys.gettotalrefcount() missing)'
- info_add('Py_DEBUG', text)
+ info_add('build.Py_DEBUG', text)
+
+ # Were we compiled --with-trace-refs?
+ Py_TRACE_REFS = hasattr(sys, 'getobjects')
+ if Py_TRACE_REFS:
+ text = 'Yes (sys.getobjects() present)'
+ else:
+ text = 'No (sys.getobjects() missing)'
+ info_add('build.Py_TRACE_REFS', text)
def collect_platform(info_add):
@@ -443,6 +455,11 @@ def collect_datetime(info_add):
def collect_sysconfig(info_add):
+ # On Windows, sysconfig is not reliable to get macros used
+ # to build Python
+ if MS_WINDOWS:
+ return
+
import sysconfig
for name in (
@@ -476,6 +493,28 @@ def collect_sysconfig(info_add):
value = normalize_text(value)
info_add('sysconfig[%s]' % name, value)
+ PY_CFLAGS = sysconfig.get_config_var('PY_CFLAGS')
+ NDEBUG = (PY_CFLAGS and '-DNDEBUG' in PY_CFLAGS)
+ if NDEBUG:
+ text = 'ignore assertions (macro defined)'
+ else:
+ text= 'build assertions (macro not defined)'
+ info_add('build.NDEBUG',text)
+
+ for name in (
+ 'WITH_DOC_STRINGS',
+ 'WITH_DTRACE',
+ 'WITH_FREELISTS',
+ 'WITH_PYMALLOC',
+ 'WITH_VALGRIND',
+ ):
+ value = sysconfig.get_config_var(name)
+ if value:
+ text = 'Yes'
+ else:
+ text = 'No'
+ info_add(f'build.{name}', text)
+
def collect_ssl(info_add):
import os
@@ -531,7 +570,10 @@ def format_attr(attr, value):
def collect_socket(info_add):
- import socket
+ try:
+ import socket
+ except ImportError:
+ return
hostname = socket.gethostname()
info_add('socket.hostname', hostname)
@@ -584,7 +626,6 @@ def collect_testcapi(info_add):
return
call_func(info_add, 'pymem.allocator', _testcapi, 'pymem_getallocatorsname')
- copy_attr(info_add, 'pymem.with_pymalloc', _testcapi, 'WITH_PYMALLOC')
def collect_resource(info_add):
@@ -626,6 +667,13 @@ def collect_test_support(info_add):
call_func(info_add, 'test_support._is_gui_available', support, '_is_gui_available')
call_func(info_add, 'test_support.python_is_optimized', support, 'python_is_optimized')
+ info_add('test_support.check_sanitizer(address=True)',
+ support.check_sanitizer(address=True))
+ info_add('test_support.check_sanitizer(memory=True)',
+ support.check_sanitizer(memory=True))
+ info_add('test_support.check_sanitizer(ub=True)',
+ support.check_sanitizer(ub=True))
+
def collect_cc(info_add):
import subprocess
@@ -719,6 +767,48 @@ def collect_windows(info_add):
except (ImportError, AttributeError):
pass
+ import subprocess
+ try:
+ # When wmic.exe output is redirected to a pipe,
+ # it uses the OEM code page
+ proc = subprocess.Popen(["wmic", "os", "get", "Caption,Version", "/value"],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ encoding="oem",
+ text=True)
+ output, stderr = proc.communicate()
+ if proc.returncode:
+ output = ""
+ except OSError:
+ pass
+ else:
+ for line in output.splitlines():
+ line = line.strip()
+ if line.startswith('Caption='):
+ line = line.removeprefix('Caption=').strip()
+ if line:
+ info_add('windows.version_caption', line)
+ elif line.startswith('Version='):
+ line = line.removeprefix('Version=').strip()
+ if line:
+ info_add('windows.version', line)
+
+ try:
+ proc = subprocess.Popen(["ver"], shell=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True)
+ output = proc.communicate()[0]
+ if proc.returncode:
+ output = ""
+ except OSError:
+ return
+ else:
+ output = output.strip()
+ line = output.splitlines()[0]
+ if line:
+ info_add('windows.ver', line)
+
def collect_fips(info_add):
try:
diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
index 80f3a04fcc6180..b7cf1e28581ed9 100644
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -13,6 +13,7 @@
import time
import types
import unittest
+import warnings
from .testresult import get_test_runner
@@ -40,6 +41,7 @@
"requires_IEEE_754", "requires_zlib",
"anticipate_failure", "load_package_tests", "detect_api_mismatch",
"check__all__", "skip_if_buggy_ucrt_strfptime",
+ "check_disallow_instantiation", "check_sanitizer", "skip_if_sanitizer",
# sys
"is_jython", "is_android", "check_impl_detail", "unix_shell",
"setswitchinterval",
@@ -73,8 +75,8 @@
elif sys.platform == 'vxworks':
LOOPBACK_TIMEOUT = 10
-# Timeout in seconds for network requests going to the Internet. The timeout is
-# short enough to prevent a test to wait for too long if the Internet request
+# Timeout in seconds for network requests going to the internet. The timeout is
+# short enough to prevent a test to wait for too long if the internet request
# is blocked for whatever reason.
#
# Usually, a timeout using INTERNET_TIMEOUT should not mark a test as failed,
@@ -104,6 +106,17 @@ class Error(Exception):
class TestFailed(Error):
"""Test failed."""
+class TestFailedWithDetails(TestFailed):
+ """Test failed."""
+ def __init__(self, msg, errors, failures):
+ self.msg = msg
+ self.errors = errors
+ self.failures = failures
+ super().__init__(msg, errors, failures)
+
+ def __str__(self):
+ return self.msg
+
class TestDidNotRun(Error):
"""Test did not run any subtests."""
@@ -354,6 +367,41 @@ def wrapper(*args, **kw):
return decorator
+def check_sanitizer(*, address=False, memory=False, ub=False):
+ """Returns True if Python is compiled with sanitizer support"""
+ if not (address or memory or ub):
+ raise ValueError('At least one of address, memory, or ub must be True')
+
+
+ _cflags = sysconfig.get_config_var('CFLAGS') or ''
+ _config_args = sysconfig.get_config_var('CONFIG_ARGS') or ''
+ memory_sanitizer = (
+ '-fsanitize=memory' in _cflags or
+ '--with-memory-sanitizer' in _config_args
+ )
+ address_sanitizer = (
+ '-fsanitize=address' in _cflags or
+ '--with-memory-sanitizer' in _config_args
+ )
+ ub_sanitizer = (
+ '-fsanitize=undefined' in _cflags or
+ '--with-undefined-behavior-sanitizer' in _config_args
+ )
+ return (
+ (memory and memory_sanitizer) or
+ (address and address_sanitizer) or
+ (ub and ub_sanitizer)
+ )
+
+
+def skip_if_sanitizer(reason=None, *, address=False, memory=False, ub=False):
+ """Decorator raising SkipTest if running with a sanitizer active."""
+ if not reason:
+ reason = 'not working with sanitizers active'
+ skip = check_sanitizer(address=address, memory=memory, ub=ub)
+ return unittest.skipIf(skip, reason)
+
+
def system_must_validate_cert(f):
"""Skip the test on TLS certificate validation failures."""
@functools.wraps(f)
@@ -377,8 +425,8 @@ def dec(*args, **kwargs):
# A constant likely larger than the underlying OS socket buffer size, to make
# writes blocking.
# The socket buffer sizes can usually be tuned system-wide (e.g. through sysctl
-# on Linux), or on a per-socket basis (SO_SNDBUF/SO_RCVBUF). See issue #18643
-# for a discussion of this number).
+# on Linux), or on a per-socket basis (SO_SNDBUF/SO_RCVBUF). See issue #18643
+# for a discussion of this number.
SOCK_MAX_SIZE = 16 * 1024 * 1024 + 1
# decorator for skipping tests on non-IEEE 754 platforms
@@ -447,6 +495,24 @@ def requires_lzma(reason='requires lzma'):
TEST_DATA_DIR = os.path.join(TEST_HOME_DIR, "data")
+def darwin_malloc_err_warning(test_name):
+ """Assure user that loud errors generated by macOS libc's malloc are
+ expected."""
+ if sys.platform != 'darwin':
+ return
+
+ import shutil
+ msg = ' NOTICE '
+ detail = (f'{test_name} may generate "malloc can\'t allocate region"\n'
+ 'warnings on macOS systems. This behavior is known. Do not\n'
+ 'report a bug unless tests are also failing. See bpo-40928.')
+
+ padding, _ = shutil.get_terminal_size()
+ print(msg.center(padding, '-'))
+ print(detail)
+ print('-' * padding)
+
+
def findfile(filename, subdir=None):
"""Try to find a file on sys.path or in the test directory. If it is not
found the argument passed to the function is returned (this does not
@@ -656,9 +722,8 @@ def check_sizeof(test, o, size):
# Decorator for running a function in a different locale, correctly resetting
# it afterwards.
+@contextlib.contextmanager
def run_with_locale(catstr, *locales):
- def decorator(func):
- def inner(*args, **kwds):
try:
import locale
category = getattr(locale, catstr)
@@ -677,16 +742,11 @@ def inner(*args, **kwds):
except:
pass
- # now run the function, resetting the locale on exceptions
try:
- return func(*args, **kwds)
+ yield
finally:
if locale and orig_locale:
locale.setlocale(category, orig_locale)
- inner.__name__ = func.__name__
- inner.__doc__ = func.__doc__
- return inner
- return decorator
#=======================================================================
# Decorator for running a function in a specific timezone, correctly
@@ -971,7 +1031,9 @@ def _run_suite(suite):
else:
err = "multiple errors occurred"
if not verbose: err += "; run in verbose mode for details"
- raise TestFailed(err)
+ errors = [(str(tc), exc_str) for tc, exc_str in result.errors]
+ failures = [(str(tc), exc_str) for tc, exc_str in result.failures]
+ raise TestFailedWithDetails(err, errors, failures)
# By default, don't filter tests
@@ -1319,9 +1381,6 @@ def __init__(self, link=None):
self._platform_specific()
- def _platform_specific(self):
- pass
-
if sys.platform == "win32":
def _platform_specific(self):
import glob
@@ -1349,6 +1408,9 @@ def _platform_specific(self):
self._env["PYTHONHOME"] = os.path.dirname(self.real)
if sysconfig.is_python_build(True):
self._env["PYTHONPATH"] = os.path.dirname(os.__file__)
+ else:
+ def _platform_specific(self):
+ pass
def __enter__(self):
os.symlink(self.real, self.link)
@@ -1619,6 +1681,16 @@ def cleanup():
setattr(object_to_patch, attr_name, new_value)
+@contextlib.contextmanager
+def patch_list(orig):
+ """Like unittest.mock.patch.dict, but for lists."""
+ try:
+ saved = orig[:]
+ yield
+ finally:
+ orig[:] = saved
+
+
def run_in_subinterp(code):
"""
Run code in a subinterpreter. Raise unittest.SkipTest if the tracemalloc
@@ -1914,7 +1986,7 @@ def wait_process(pid, *, exitcode, timeout=None):
Raise an AssertionError if the process exit code is not equal to exitcode.
- If the process runs longer than timeout seconds (SHORT_TIMEOUT by default),
+ If the process runs longer than timeout seconds (LONG_TIMEOUT by default),
kill the process (if signal.SIGKILL is available) and raise an
AssertionError. The timeout feature is not available on Windows.
"""
@@ -1922,7 +1994,7 @@ def wait_process(pid, *, exitcode, timeout=None):
import signal
if timeout is None:
- timeout = SHORT_TIMEOUT
+ timeout = LONG_TIMEOUT
t0 = time.monotonic()
sleep = 0.001
max_sleep = 0.1
@@ -1933,7 +2005,7 @@ def wait_process(pid, *, exitcode, timeout=None):
# process is still running
dt = time.monotonic() - t0
- if dt > SHORT_TIMEOUT:
+ if dt > timeout:
try:
os.kill(pid, signal.SIGKILL)
os.waitpid(pid, 0)
@@ -1982,3 +2054,71 @@ def skip_if_broken_multiprocessing_synchronize():
synchronize.Lock(ctx=None)
except OSError as exc:
raise unittest.SkipTest(f"broken multiprocessing SemLock: {exc!r}")
+
+
+@contextlib.contextmanager
+def infinite_recursion(max_depth=75):
+ original_depth = sys.getrecursionlimit()
+ try:
+ sys.setrecursionlimit(max_depth)
+ yield
+ finally:
+ sys.setrecursionlimit(original_depth)
+
+
+def check_disallow_instantiation(testcase, tp, *args, **kwds):
+ """
+ Check that given type cannot be instantiated using *args and **kwds.
+
+ See bpo-43916: Add Py_TPFLAGS_DISALLOW_INSTANTIATION type flag.
+ """
+ mod = tp.__module__
+ name = tp.__name__
+ if mod != 'builtins':
+ qualname = f"{mod}.{name}"
+ else:
+ qualname = f"{name}"
+ msg = f"cannot create '{re.escape(qualname)}' instances"
+ testcase.assertRaisesRegex(TypeError, msg, tp, *args, **kwds)
+
+
+def ignore_deprecations_from(module: str, *, like: str) -> object:
+ token = object()
+ warnings.filterwarnings(
+ "ignore",
+ category=DeprecationWarning,
+ module=module,
+ message=like + fr"(?#support{id(token)})",
+ )
+ return token
+
+
+def clear_ignored_deprecations(*tokens: object) -> None:
+ if not tokens:
+ raise ValueError("Provide token or tokens returned by ignore_deprecations_from")
+
+ new_filters = []
+ endswith = tuple(rf"(?#support{id(token)})" for token in tokens)
+ for action, message, category, module, lineno in warnings.filters:
+ if action == "ignore" and category is DeprecationWarning:
+ if isinstance(message, re.Pattern):
+ msg = message.pattern
+ else:
+ msg = message or ""
+ if msg.endswith(endswith):
+ continue
+ new_filters.append((action, message, category, module, lineno))
+ if warnings.filters != new_filters:
+ warnings.filters[:] = new_filters
+ warnings._filters_mutated()
+
+
+@contextlib.contextmanager
+def adjust_int_max_str_digits(max_digits):
+ """Temporarily change the integer string conversion length limit."""
+ current = sys.get_int_max_str_digits()
+ try:
+ sys.set_int_max_str_digits(max_digits)
+ yield
+ finally:
+ sys.set_int_max_str_digits(current)
diff --git a/Lib/test/support/hashlib_helper.py b/Lib/test/support/hashlib_helper.py
index a28132a565a0b5..a4e6c92203ab50 100644
--- a/Lib/test/support/hashlib_helper.py
+++ b/Lib/test/support/hashlib_helper.py
@@ -21,8 +21,21 @@ def requires_hashdigest(digestname, openssl=None, usedforsecurity=True):
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
ValueError: unsupported hash type md4
"""
- def decorator(func):
- @functools.wraps(func)
+ def decorator(func_or_class):
+ if isinstance(func_or_class, type):
+ setUpClass = func_or_class.__dict__.get('setUpClass')
+ if setUpClass is None:
+ def setUpClass(cls):
+ super(func_or_class, cls).setUpClass()
+ setUpClass.__qualname__ = func_or_class.__qualname__ + '.setUpClass'
+ setUpClass.__module__ = func_or_class.__module__
+ else:
+ setUpClass = setUpClass.__func__
+ setUpClass = classmethod(decorator(setUpClass))
+ func_or_class.setUpClass = setUpClass
+ return func_or_class
+
+ @functools.wraps(func_or_class)
def wrapper(*args, **kwargs):
try:
if openssl and _hashlib is not None:
@@ -33,6 +46,6 @@ def wrapper(*args, **kwargs):
raise unittest.SkipTest(
f"hash digest '{digestname}' is not available."
)
- return func(*args, **kwargs)
+ return func_or_class(*args, **kwargs)
return wrapper
return decorator
diff --git a/Lib/test/support/import_helper.py b/Lib/test/support/import_helper.py
index 5d1e9406879cc6..efa8ffad6a8e0b 100644
--- a/Lib/test/support/import_helper.py
+++ b/Lib/test/support/import_helper.py
@@ -2,6 +2,7 @@
import importlib
import importlib.util
import os
+import shutil
import sys
import unittest
import warnings
@@ -58,7 +59,7 @@ def make_legacy_pyc(source):
pyc_file = importlib.util.cache_from_source(source)
up_one = os.path.dirname(os.path.abspath(source))
legacy_pyc = os.path.join(up_one, source + 'c')
- os.rename(pyc_file, legacy_pyc)
+ shutil.move(pyc_file, legacy_pyc)
return legacy_pyc
@@ -80,33 +81,13 @@ def import_module(name, deprecated=False, *, required_on=()):
raise unittest.SkipTest(str(msg))
-def _save_and_remove_module(name, orig_modules):
- """Helper function to save and remove a module from sys.modules
-
- Raise ImportError if the module can't be imported.
- """
- # try to import the module and raise an error if it can't be imported
- if name not in sys.modules:
- __import__(name)
- del sys.modules[name]
+def _save_and_remove_modules(names):
+ orig_modules = {}
+ prefixes = tuple(name + '.' for name in names)
for modname in list(sys.modules):
- if modname == name or modname.startswith(name + '.'):
- orig_modules[modname] = sys.modules[modname]
- del sys.modules[modname]
-
-
-def _save_and_block_module(name, orig_modules):
- """Helper function to save and block a module in sys.modules
-
- Return True if the module was in sys.modules, False otherwise.
- """
- saved = True
- try:
- orig_modules[name] = sys.modules[name]
- except KeyError:
- saved = False
- sys.modules[name] = None
- return saved
+ if modname in names or modname.startswith(prefixes):
+ orig_modules[modname] = sys.modules.pop(modname)
+ return orig_modules
def import_fresh_module(name, fresh=(), blocked=(), deprecated=False):
@@ -118,7 +99,8 @@ def import_fresh_module(name, fresh=(), blocked=(), deprecated=False):
this operation.
*fresh* is an iterable of additional module names that are also removed
- from the sys.modules cache before doing the import.
+ from the sys.modules cache before doing the import. If one of these
+ modules can't be imported, None is returned.
*blocked* is an iterable of module names that are replaced with None
in the module cache during the import to ensure that attempts to import
@@ -139,24 +121,24 @@ def import_fresh_module(name, fresh=(), blocked=(), deprecated=False):
with _ignore_deprecated_imports(deprecated):
# Keep track of modules saved for later restoration as well
# as those which just need a blocking entry removed
- orig_modules = {}
- names_to_remove = []
- _save_and_remove_module(name, orig_modules)
+ fresh = list(fresh)
+ blocked = list(blocked)
+ names = {name, *fresh, *blocked}
+ orig_modules = _save_and_remove_modules(names)
+ for modname in blocked:
+ sys.modules[modname] = None
+
try:
- for fresh_name in fresh:
- _save_and_remove_module(fresh_name, orig_modules)
- for blocked_name in blocked:
- if not _save_and_block_module(blocked_name, orig_modules):
- names_to_remove.append(blocked_name)
- fresh_module = importlib.import_module(name)
- except ImportError:
- fresh_module = None
+ # Return None when one of the "fresh" modules can not be imported.
+ try:
+ for modname in fresh:
+ __import__(modname)
+ except ImportError:
+ return None
+ return importlib.import_module(name)
finally:
- for orig_name, module in orig_modules.items():
- sys.modules[orig_name] = module
- for name_to_remove in names_to_remove:
- del sys.modules[name_to_remove]
- return fresh_module
+ _save_and_remove_modules(names)
+ sys.modules.update(orig_modules)
class CleanImport(object):
diff --git a/Lib/test/support/os_helper.py b/Lib/test/support/os_helper.py
index d9807a1e114b65..82a6de789c866e 100644
--- a/Lib/test/support/os_helper.py
+++ b/Lib/test/support/os_helper.py
@@ -455,6 +455,17 @@ def create_empty_file(filename):
os.close(fd)
+@contextlib.contextmanager
+def open_dir_fd(path):
+ """Open a file descriptor to a directory."""
+ assert os.path.isdir(path)
+ dir_fd = os.open(path, os.O_RDONLY)
+ try:
+ yield dir_fd
+ finally:
+ os.close(dir_fd)
+
+
def fs_is_case_insensitive(directory):
"""Detects if the file system for the specified directory
is case-insensitive."""
diff --git a/Lib/test/support/socket_helper.py b/Lib/test/support/socket_helper.py
index e78712b74b1377..38c499bf722e85 100644
--- a/Lib/test/support/socket_helper.py
+++ b/Lib/test/support/socket_helper.py
@@ -58,7 +58,7 @@ def find_unused_port(family=socket.AF_INET, socktype=socket.SOCK_STREAM):
http://bugs.python.org/issue2550 for more info. The following site also
has a very thorough description about the implications of both REUSEADDR
and EXCLUSIVEADDRUSE on Windows:
- http://msdn2.microsoft.com/en-us/library/ms740621(VS.85).aspx)
+ https://learn.microsoft.com/windows/win32/winsock/using-so-reuseaddr-and-so-exclusiveaddruse
XXX: although this approach is a vast improvement on previous attempts to
elicit unused ports, it rests heavily on the assumption that the ephemeral
@@ -189,7 +189,7 @@ def get_socket_conn_refused_errs():
@contextlib.contextmanager
def transient_internet(resource_name, *, timeout=_NOT_SET, errnos=()):
"""Return a context manager that raises ResourceDenied when various issues
- with the Internet connection manifest themselves as exceptions."""
+ with the internet connection manifest themselves as exceptions."""
import nntplib
import urllib.error
if timeout is _NOT_SET:
diff --git a/Lib/test/support/testresult.py b/Lib/test/support/testresult.py
index 670afbea2659de..6f2edda0f580ab 100644
--- a/Lib/test/support/testresult.py
+++ b/Lib/test/support/testresult.py
@@ -10,12 +10,11 @@
import unittest
class RegressionTestResult(unittest.TextTestResult):
- separator1 = '=' * 70 + '\n'
- separator2 = '-' * 70 + '\n'
USE_XML = False
def __init__(self, stream, descriptions, verbosity):
- super().__init__(stream=stream, descriptions=descriptions, verbosity=0)
+ super().__init__(stream=stream, descriptions=descriptions,
+ verbosity=2 if verbosity else 0)
self.buffer = True
if self.USE_XML:
from xml.etree import ElementTree as ET
@@ -25,8 +24,6 @@ def __init__(self, stream, descriptions, verbosity):
self.__suite.set('start', datetime.utcnow().isoformat(' '))
self.__e = None
self.__start_time = None
- self.__results = []
- self.__verbose = bool(verbosity)
@classmethod
def __getId(cls, test):
@@ -45,9 +42,6 @@ def startTest(self, test):
if self.USE_XML:
self.__e = e = self.__ET.SubElement(self.__suite, 'testcase')
self.__start_time = time.perf_counter()
- if self.__verbose:
- self.stream.write(f'{self.getDescription(test)} ... ')
- self.stream.flush()
def _add_result(self, test, capture=False, **args):
if not self.USE_XML:
@@ -85,10 +79,6 @@ def _add_result(self, test, capture=False, **args):
else:
e2.text = str(v)
- def __write(self, c, word):
- if self.__verbose:
- self.stream.write(f'{word}\n')
-
@classmethod
def __makeErrorDict(cls, err_type, err_value, err_tb):
if isinstance(err_type, type):
@@ -111,45 +101,26 @@ def __makeErrorDict(cls, err_type, err_value, err_tb):
def addError(self, test, err):
self._add_result(test, True, error=self.__makeErrorDict(*err))
super().addError(test, err)
- self.__write('E', 'ERROR')
def addExpectedFailure(self, test, err):
self._add_result(test, True, output=self.__makeErrorDict(*err))
super().addExpectedFailure(test, err)
- self.__write('x', 'expected failure')
def addFailure(self, test, err):
self._add_result(test, True, failure=self.__makeErrorDict(*err))
super().addFailure(test, err)
- self.__write('F', 'FAIL')
def addSkip(self, test, reason):
self._add_result(test, skipped=reason)
super().addSkip(test, reason)
- self.__write('S', f'skipped {reason!r}')
def addSuccess(self, test):
self._add_result(test)
super().addSuccess(test)
- self.__write('.', 'ok')
def addUnexpectedSuccess(self, test):
self._add_result(test, outcome='UNEXPECTED_SUCCESS')
super().addUnexpectedSuccess(test)
- self.__write('u', 'unexpected success')
-
- def printErrors(self):
- if self.__verbose:
- self.stream.write('\n')
- self.printErrorList('ERROR', self.errors)
- self.printErrorList('FAIL', self.failures)
-
- def printErrorList(self, flavor, errors):
- for test, err in errors:
- self.stream.write(self.separator1)
- self.stream.write(f'{flavor}: {self.getDescription(test)}\n')
- self.stream.write(self.separator2)
- self.stream.write('%s\n' % err)
def get_xml_element(self):
if not self.USE_XML:
diff --git a/Lib/test/support/threading_helper.py b/Lib/test/support/threading_helper.py
index 0632577cdb303c..92a64e8354acbc 100644
--- a/Lib/test/support/threading_helper.py
+++ b/Lib/test/support/threading_helper.py
@@ -157,7 +157,7 @@ class catch_threading_exception:
Context manager catching threading.Thread exception using
threading.excepthook.
- Attributes set when an exception is catched:
+ Attributes set when an exception is caught:
* exc_type
* exc_value
diff --git a/Lib/test/support/warnings_helper.py b/Lib/test/support/warnings_helper.py
index de23e6b452b28d..93a91e6d99f955 100644
--- a/Lib/test/support/warnings_helper.py
+++ b/Lib/test/support/warnings_helper.py
@@ -36,7 +36,7 @@ def check_syntax_warning(testcase, statement, errtext='',
def ignore_warnings(*, category):
- """Decorator to suppress deprecation warnings.
+ """Decorator to suppress warnings.
Use of context managers to hide warnings make diffs
more noisy and tools like 'git blame' less useful.
@@ -187,3 +187,13 @@ def save_restore_warnings_filters():
yield
finally:
warnings.filters[:] = old_filters
+
+
+def _warn_about_deprecation():
+ warnings.warn(
+ "This is used in test_support test to ensure"
+ " support.ignore_deprecations_from() works as expected."
+ " You should not be seeing this.",
+ DeprecationWarning,
+ stacklevel=0,
+ )
diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py
index 15f42d2d114a68..6368ef02253b6d 100644
--- a/Lib/test/test___all__.py
+++ b/Lib/test/test___all__.py
@@ -5,6 +5,13 @@
import sys
+if support.check_sanitizer(address=True, memory=True):
+ # bpo-46633: test___all__ is skipped because importing some modules
+ # directly can trigger known problems with ASAN (like tk or crypt).
+ raise unittest.SkipTest("workaround ASAN build issues on loading tests "
+ "like tk or crypt")
+
+
class NoAll(RuntimeError):
pass
diff --git a/Lib/test/test__xxsubinterpreters.py b/Lib/test/test__xxsubinterpreters.py
index 7baea69a4e5fa7..177a8a64a4329f 100644
--- a/Lib/test/test__xxsubinterpreters.py
+++ b/Lib/test/test__xxsubinterpreters.py
@@ -39,6 +39,20 @@ def _run_output(interp, request, shared=None):
return rpipe.read()
+def _wait_for_interp_to_run(interp, timeout=None):
+ # bpo-37224: Running this test file in multiprocesses will fail randomly.
+ # The failure reason is that the thread can't acquire the cpu to
+ # run subinterpreter eariler than the main thread in multiprocess.
+ if timeout is None:
+ timeout = support.SHORT_TIMEOUT
+ start_time = time.monotonic()
+ deadline = start_time + timeout
+ while not interpreters.is_running(interp):
+ if time.monotonic() > deadline:
+ raise RuntimeError('interp is not running')
+ time.sleep(0.010)
+
+
@contextlib.contextmanager
def _running(interp):
r, w = os.pipe()
@@ -51,6 +65,7 @@ def run():
t = threading.Thread(target=run)
t.start()
+ _wait_for_interp_to_run(interp)
yield
@@ -1206,7 +1221,7 @@ def test_channel_list_interpreters_basic(self):
import _xxsubinterpreters as _interpreters
obj = _interpreters.channel_recv({cid})
"""))
- # Test for channel that has boths ends associated to an interpreter.
+ # Test for channel that has both ends associated to an interpreter.
send_interps = interpreters.channel_list_interpreters(cid, send=True)
recv_interps = interpreters.channel_list_interpreters(cid, send=False)
self.assertEqual(send_interps, [interp0])
diff --git a/Lib/test/test_abc.py b/Lib/test/test_abc.py
index 3d603e7734d870..c1d750dba83f99 100644
--- a/Lib/test/test_abc.py
+++ b/Lib/test/test_abc.py
@@ -459,6 +459,24 @@ class S(metaclass=abc_ABCMeta):
with self.assertRaisesRegex(Exception, exc_msg):
issubclass(int, S)
+ def test_subclasshook(self):
+ class A(metaclass=abc.ABCMeta):
+ @classmethod
+ def __subclasshook__(cls, C):
+ if cls is A:
+ return 'foo' in C.__dict__
+ return NotImplemented
+ self.assertFalse(issubclass(A, A))
+ self.assertFalse(issubclass(A, (A,)))
+ class B:
+ foo = 42
+ self.assertTrue(issubclass(B, A))
+ self.assertTrue(issubclass(B, (A,)))
+ class C:
+ spam = 42
+ self.assertFalse(issubclass(C, A))
+ self.assertFalse(issubclass(C, (A,)))
+
def test_all_new_methods_are_called(self):
class A(metaclass=abc_ABCMeta):
pass
diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py
index 4d0316f73edcd2..5df6f3263075a9 100644
--- a/Lib/test/test_argparse.py
+++ b/Lib/test/test_argparse.py
@@ -1,6 +1,8 @@
# Author: Steven J. Bethard .
import inspect
+import io
+import operator
import os
import shutil
import stat
@@ -10,13 +12,27 @@
import unittest
import argparse
-from io import StringIO
-
-from test import support
from test.support import os_helper
from unittest import mock
-class StdIOBuffer(StringIO):
- pass
+
+
+class StdIOBuffer(io.TextIOWrapper):
+ '''Replacement for writable io.StringIO that behaves more like real file
+
+ Unlike StringIO, provides a buffer attribute that holds the underlying
+ binary data, allowing it to replace sys.stdout/sys.stderr in more
+ contexts.
+ '''
+
+ def __init__(self, initial_value='', newline='\n'):
+ initial_value = initial_value.encode('utf-8')
+ super().__init__(io.BufferedWriter(io.BytesIO(initial_value)),
+ 'utf-8', newline=newline)
+
+ def getvalue(self):
+ self.flush()
+ return self.buffer.raw.getvalue().decode('utf-8')
+
class TestCase(unittest.TestCase):
@@ -43,11 +59,14 @@ def tearDown(self):
os.chmod(os.path.join(self.temp_dir, name), stat.S_IWRITE)
shutil.rmtree(self.temp_dir, True)
- def create_readonly_file(self, filename):
+ def create_writable_file(self, filename):
file_path = os.path.join(self.temp_dir, filename)
with open(file_path, 'w', encoding="utf-8") as file:
file.write(filename)
- os.chmod(file_path, stat.S_IREAD)
+ return file_path
+
+ def create_readonly_file(self, filename):
+ os.chmod(self.create_writable_file(filename), stat.S_IREAD)
class Sig(object):
@@ -97,13 +116,18 @@ def stderr_to_parser_error(parse_args, *args, **kwargs):
try:
result = parse_args(*args, **kwargs)
for key in list(vars(result)):
- if getattr(result, key) is sys.stdout:
+ attr = getattr(result, key)
+ if attr is sys.stdout:
setattr(result, key, old_stdout)
- if getattr(result, key) is sys.stderr:
+ elif attr is sys.stdout.buffer:
+ setattr(result, key, getattr(old_stdout, 'buffer', BIN_STDOUT_SENTINEL))
+ elif attr is sys.stderr:
setattr(result, key, old_stderr)
+ elif attr is sys.stderr.buffer:
+ setattr(result, key, getattr(old_stderr, 'buffer', BIN_STDERR_SENTINEL))
return result
- except SystemExit:
- code = sys.exc_info()[1].code
+ except SystemExit as e:
+ code = e.code
stdout = sys.stdout.getvalue()
stderr = sys.stderr.getvalue()
raise ArgumentParserError(
@@ -271,7 +295,7 @@ class TestOptionalsSingleDashCombined(ParserTestCase):
Sig('-z'),
]
failures = ['a', '--foo', '-xa', '-x --foo', '-x -z', '-z -x',
- '-yx', '-yz a', '-yyyx', '-yyyza', '-xyza']
+ '-yx', '-yz a', '-yyyx', '-yyyza', '-xyza', '-x=']
successes = [
('', NS(x=False, yyy=None, z=None)),
('-x', NS(x=True, yyy=None, z=None)),
@@ -1546,16 +1570,40 @@ def test_r_1_replace(self):
type = argparse.FileType('r', 1, errors='replace')
self.assertEqual("FileType('r', 1, errors='replace')", repr(type))
+
+BIN_STDOUT_SENTINEL = object()
+BIN_STDERR_SENTINEL = object()
+
+
class StdStreamComparer:
def __init__(self, attr):
- self.attr = attr
+ # We try to use the actual stdXXX.buffer attribute as our
+ # marker, but but under some test environments,
+ # sys.stdout/err are replaced by io.StringIO which won't have .buffer,
+ # so we use a sentinel simply to show that the tests do the right thing
+ # for any buffer supporting object
+ self.getattr = operator.attrgetter(attr)
+ if attr == 'stdout.buffer':
+ self.backupattr = BIN_STDOUT_SENTINEL
+ elif attr == 'stderr.buffer':
+ self.backupattr = BIN_STDERR_SENTINEL
+ else:
+ self.backupattr = object() # Not equal to anything
def __eq__(self, other):
- return other == getattr(sys, self.attr)
+ try:
+ return other == self.getattr(sys)
+ except AttributeError:
+ return other == self.backupattr
+
eq_stdin = StdStreamComparer('stdin')
eq_stdout = StdStreamComparer('stdout')
eq_stderr = StdStreamComparer('stderr')
+eq_bstdin = StdStreamComparer('stdin.buffer')
+eq_bstdout = StdStreamComparer('stdout.buffer')
+eq_bstderr = StdStreamComparer('stderr.buffer')
+
class RFile(object):
seen = {}
@@ -1634,7 +1682,7 @@ def setUp(self):
('foo', NS(x=None, spam=RFile('foo'))),
('-x foo bar', NS(x=RFile('foo'), spam=RFile('bar'))),
('bar -x foo', NS(x=RFile('foo'), spam=RFile('bar'))),
- ('-x - -', NS(x=eq_stdin, spam=eq_stdin)),
+ ('-x - -', NS(x=eq_bstdin, spam=eq_bstdin)),
]
@@ -1661,8 +1709,9 @@ class TestFileTypeW(TempDirMixin, ParserTestCase):
"""Test the FileType option/argument type for writing files"""
def setUp(self):
- super(TestFileTypeW, self).setUp()
+ super().setUp()
self.create_readonly_file('readonly')
+ self.create_writable_file('writable')
argument_signatures = [
Sig('-x', type=argparse.FileType('w')),
@@ -1671,13 +1720,37 @@ def setUp(self):
failures = ['-x', '', 'readonly']
successes = [
('foo', NS(x=None, spam=WFile('foo'))),
+ ('writable', NS(x=None, spam=WFile('writable'))),
('-x foo bar', NS(x=WFile('foo'), spam=WFile('bar'))),
('bar -x foo', NS(x=WFile('foo'), spam=WFile('bar'))),
('-x - -', NS(x=eq_stdout, spam=eq_stdout)),
]
+@unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+ "non-root user required")
+class TestFileTypeX(TempDirMixin, ParserTestCase):
+ """Test the FileType option/argument type for writing new files only"""
+
+ def setUp(self):
+ super().setUp()
+ self.create_readonly_file('readonly')
+ self.create_writable_file('writable')
+
+ argument_signatures = [
+ Sig('-x', type=argparse.FileType('x')),
+ Sig('spam', type=argparse.FileType('x')),
+ ]
+ failures = ['-x', '', 'readonly', 'writable']
+ successes = [
+ ('-x foo bar', NS(x=WFile('foo'), spam=WFile('bar'))),
+ ('-x - -', NS(x=eq_stdout, spam=eq_stdout)),
+ ]
+
+@unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+ "non-root user required")
class TestFileTypeWB(TempDirMixin, ParserTestCase):
+ """Test the FileType option/argument type for writing binary files"""
argument_signatures = [
Sig('-x', type=argparse.FileType('wb')),
@@ -1688,7 +1761,22 @@ class TestFileTypeWB(TempDirMixin, ParserTestCase):
('foo', NS(x=None, spam=WFile('foo'))),
('-x foo bar', NS(x=WFile('foo'), spam=WFile('bar'))),
('bar -x foo', NS(x=WFile('foo'), spam=WFile('bar'))),
- ('-x - -', NS(x=eq_stdout, spam=eq_stdout)),
+ ('-x - -', NS(x=eq_bstdout, spam=eq_bstdout)),
+ ]
+
+
+@unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+ "non-root user required")
+class TestFileTypeXB(TestFileTypeX):
+ "Test the FileType option/argument type for writing new binary files only"
+
+ argument_signatures = [
+ Sig('-x', type=argparse.FileType('xb')),
+ Sig('spam', type=argparse.FileType('xb')),
+ ]
+ successes = [
+ ('-x foo bar', NS(x=WFile('foo'), spam=WFile('bar'))),
+ ('-x - -', NS(x=eq_bstdout, spam=eq_bstdout)),
]
@@ -1831,8 +1919,7 @@ def __call__(self, parser, namespace, value, option_string=None):
raise AssertionError('value: %s' % value)
assert expected_ns == namespace, ('expected %s, got %s' %
(expected_ns, namespace))
- except AssertionError:
- e = sys.exc_info()[1]
+ except AssertionError as e:
raise ArgumentParserError('opt_action failed: %s' % e)
setattr(namespace, 'spam', value)
@@ -1857,8 +1944,7 @@ def __call__(self, parser, namespace, value, option_string=None):
raise AssertionError('value: %s' % value)
assert expected_ns == namespace, ('expected %s, got %s' %
(expected_ns, namespace))
- except AssertionError:
- e = sys.exc_info()[1]
+ except AssertionError as e:
raise ArgumentParserError('arg_action failed: %s' % e)
setattr(namespace, 'badger', value)
@@ -2060,6 +2146,30 @@ def test_required_subparsers_default(self):
ret = parser.parse_args(())
self.assertIsNone(ret.command)
+ def test_required_subparsers_no_destination_error(self):
+ parser = ErrorRaisingArgumentParser()
+ subparsers = parser.add_subparsers(required=True)
+ subparsers.add_parser('foo')
+ subparsers.add_parser('bar')
+ with self.assertRaises(ArgumentParserError) as excinfo:
+ parser.parse_args(())
+ self.assertRegex(
+ excinfo.exception.stderr,
+ 'error: the following arguments are required: {foo,bar}\n$'
+ )
+
+ def test_wrong_argument_subparsers_no_destination_error(self):
+ parser = ErrorRaisingArgumentParser()
+ subparsers = parser.add_subparsers(required=True)
+ subparsers.add_parser('foo')
+ subparsers.add_parser('bar')
+ with self.assertRaises(ArgumentParserError) as excinfo:
+ parser.parse_args(('baz',))
+ self.assertRegex(
+ excinfo.exception.stderr,
+ r"error: argument {foo,bar}: invalid choice: 'baz' \(choose from 'foo', 'bar'\)\n$"
+ )
+
def test_optional_subparsers(self):
parser = ErrorRaisingArgumentParser()
subparsers = parser.add_subparsers(dest='command', required=False)
@@ -2122,6 +2232,42 @@ def test_help_non_breaking_spaces(self):
wrap\N{NO-BREAK SPACE}at non-breaking spaces
'''))
+ def test_help_blank(self):
+ # Issue 24444
+ parser = ErrorRaisingArgumentParser(
+ prog='PROG', description='main description')
+ parser.add_argument(
+ 'foo',
+ help=' ')
+ self.assertEqual(parser.format_help(), textwrap.dedent('''\
+ usage: PROG [-h] foo
+
+ main description
+
+ positional arguments:
+ foo
+
+ options:
+ -h, --help show this help message and exit
+ '''))
+
+ parser = ErrorRaisingArgumentParser(
+ prog='PROG', description='main description')
+ parser.add_argument(
+ 'foo', choices=[],
+ help='%(choices)s')
+ self.assertEqual(parser.format_help(), textwrap.dedent('''\
+ usage: PROG [-h] {}
+
+ main description
+
+ positional arguments:
+ {}
+
+ options:
+ -h, --help show this help message and exit
+ '''))
+
def test_help_alternate_prefix_chars(self):
parser = self._get_parser(prefix_chars='+:/')
self.assertEqual(parser.format_usage(),
@@ -2523,6 +2669,13 @@ def test_help(self):
'''
self.assertEqual(parser.format_help(), textwrap.dedent(expected))
+ def test_empty_group(self):
+ # See issue 26952
+ parser = argparse.ArgumentParser()
+ group = parser.add_mutually_exclusive_group()
+ with self.assertRaises(ValueError):
+ parser.parse_args(['-h'])
+
class MEMixin(object):
def test_failures_when_not_required(self):
@@ -3533,6 +3686,8 @@ class TestHelpUsage(HelpTestCase):
Sig('--bar', help='Whether to bar', default=True,
action=argparse.BooleanOptionalAction),
Sig('-f', '--foobar', '--barfoo', action=argparse.BooleanOptionalAction),
+ Sig('--bazz', action=argparse.BooleanOptionalAction,
+ default=argparse.SUPPRESS, help='Bazz!'),
]
argument_group_signatures = [
(Sig('group'), [
@@ -3545,8 +3700,8 @@ class TestHelpUsage(HelpTestCase):
usage = '''\
usage: PROG [-h] [-w W [W ...]] [-x [X ...]] [--foo | --no-foo]
[--bar | --no-bar]
- [-f | --foobar | --no-foobar | --barfoo | --no-barfoo] [-y [Y]]
- [-z Z Z Z]
+ [-f | --foobar | --no-foobar | --barfoo | --no-barfoo]
+ [--bazz | --no-bazz] [-y [Y]] [-z Z Z Z]
a b b [c] [d ...] e [e ...]
'''
help = usage + '''\
@@ -3563,6 +3718,7 @@ class TestHelpUsage(HelpTestCase):
--foo, --no-foo Whether to foo
--bar, --no-bar Whether to bar (default: True)
-f, --foobar, --no-foobar, --barfoo, --no-barfoo
+ --bazz, --no-bazz Bazz!
group:
-y [Y] y
@@ -3573,6 +3729,28 @@ class TestHelpUsage(HelpTestCase):
version = ''
+class TestHelpUsageWithParentheses(HelpTestCase):
+ parser_signature = Sig(prog='PROG')
+ argument_signatures = [
+ Sig('positional', metavar='(example) positional'),
+ Sig('-p', '--optional', metavar='{1 (option A), 2 (option B)}'),
+ ]
+
+ usage = '''\
+ usage: PROG [-h] [-p {1 (option A), 2 (option B)}] (example) positional
+ '''
+ help = usage + '''\
+
+ positional arguments:
+ (example) positional
+
+ options:
+ -h, --help show this help message and exit
+ -p {1 (option A), 2 (option B)}, --optional {1 (option A), 2 (option B)}
+ '''
+ version = ''
+
+
class TestHelpOnlyUserGroups(HelpTestCase):
"""Test basic usage messages"""
@@ -4238,6 +4416,9 @@ class TestHelpArgumentDefaults(HelpTestCase):
argument_signatures = [
Sig('--foo', help='foo help - oh and by the way, %(default)s'),
Sig('--bar', action='store_true', help='bar help'),
+ Sig('--taz', action=argparse.BooleanOptionalAction,
+ help='Whether to taz it', default=True),
+ Sig('--quux', help="Set the quux", default=42),
Sig('spam', help='spam help'),
Sig('badger', nargs='?', default='wooden', help='badger help'),
]
@@ -4246,25 +4427,29 @@ class TestHelpArgumentDefaults(HelpTestCase):
[Sig('--baz', type=int, default=42, help='baz help')]),
]
usage = '''\
- usage: PROG [-h] [--foo FOO] [--bar] [--baz BAZ] spam [badger]
+ usage: PROG [-h] [--foo FOO] [--bar] [--taz | --no-taz] [--quux QUUX]
+ [--baz BAZ]
+ spam [badger]
'''
help = usage + '''\
description
positional arguments:
- spam spam help
- badger badger help (default: wooden)
+ spam spam help
+ badger badger help (default: wooden)
options:
- -h, --help show this help message and exit
- --foo FOO foo help - oh and by the way, None
- --bar bar help (default: False)
+ -h, --help show this help message and exit
+ --foo FOO foo help - oh and by the way, None
+ --bar bar help (default: False)
+ --taz, --no-taz Whether to taz it (default: True)
+ --quux QUUX Set the quux (default: 42)
title:
description
- --baz BAZ baz help (default: 42)
+ --baz BAZ baz help (default: 42)
'''
version = ''
@@ -4710,12 +4895,13 @@ def test_optional(self):
nargs='+',
default=42,
choices=[1, 2, 3],
+ required=False,
help='HELP',
metavar='METAVAR')
string = (
"Action(option_strings=['--foo', '-a', '-b'], dest='b', "
"nargs='+', const=None, default=42, type='int', "
- "choices=[1, 2, 3], help='HELP', metavar='METAVAR')")
+ "choices=[1, 2, 3], required=False, help='HELP', metavar='METAVAR')")
self.assertStringEqual(option, string)
def test_argument(self):
@@ -4726,12 +4912,13 @@ def test_argument(self):
nargs='?',
default=2.5,
choices=[0.5, 1.5, 2.5],
+ required=True,
help='H HH H',
metavar='MV MV MV')
string = (
"Action(option_strings=[], dest='x', nargs='?', "
"const=None, default=2.5, type=%r, choices=[0.5, 1.5, 2.5], "
- "help='H HH H', metavar='MV MV MV')" % float)
+ "required=True, help='H HH H', metavar='MV MV MV')" % float)
self.assertStringEqual(argument, string)
def test_namespace(self):
@@ -5360,13 +5547,11 @@ def test_exit_on_error_with_bad_args(self):
self.parser.parse_args('--integers a'.split())
-def test_main():
- support.run_unittest(__name__)
+def tearDownModule():
# Remove global references to avoid looking like we have refleaks.
RFile.seen = {}
WFile.seen = set()
-
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py
old mode 100644
new mode 100755
index b18467fb889d8b..6a48c1cc975727
--- a/Lib/test/test_array.py
+++ b/Lib/test/test_array.py
@@ -42,9 +42,10 @@ def test_bad_constructor(self):
@support.cpython_only
def test_disallow_instantiation(self):
- # Ensure that the type disallows instantiation (bpo-43916)
- tp = type(iter(array.array('I')))
- self.assertRaises(TypeError, tp)
+ my_array = array.array("I")
+ support.check_disallow_instantiation(
+ self, type(iter(my_array)), my_array
+ )
@support.cpython_only
def test_immutable(self):
@@ -1096,6 +1097,7 @@ def test_weakref(self):
p = weakref.proxy(s)
self.assertEqual(p.tobytes(), s.tobytes())
s = None
+ support.gc_collect() # For PyPy or other GCs.
self.assertRaises(ReferenceError, len, p)
@unittest.skipUnless(hasattr(sys, 'getrefcount'),
diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
index 80d24e94040bcf..36fdb49bd1d7b1 100644
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -1,7 +1,6 @@
import ast
import builtins
import dis
-import enum
import os
import sys
import types
@@ -356,6 +355,24 @@ def test_alias(self):
self.assertEqual(alias.col_offset, 16)
self.assertEqual(alias.end_col_offset, 17)
+ im = ast.parse("from bar import y as z").body[0]
+ alias = im.names[0]
+ self.assertEqual(alias.name, "y")
+ self.assertEqual(alias.asname, "z")
+ self.assertEqual(alias.lineno, 1)
+ self.assertEqual(alias.end_lineno, 1)
+ self.assertEqual(alias.col_offset, 16)
+ self.assertEqual(alias.end_col_offset, 22)
+
+ im = ast.parse("import bar as foo").body[0]
+ alias = im.names[0]
+ self.assertEqual(alias.name, "bar")
+ self.assertEqual(alias.asname, "foo")
+ self.assertEqual(alias.lineno, 1)
+ self.assertEqual(alias.end_lineno, 1)
+ self.assertEqual(alias.col_offset, 7)
+ self.assertEqual(alias.end_col_offset, 17)
+
def test_base_classes(self):
self.assertTrue(issubclass(ast.For, ast.stmt))
self.assertTrue(issubclass(ast.Name, ast.expr))
@@ -687,47 +704,46 @@ def test_ast_asdl_signature(self):
expressions[0] = f"expr = {ast.expr.__subclasses__()[0].__doc__}"
self.assertCountEqual(ast.expr.__doc__.split("\n"), expressions)
- def test_issue40614_feature_version(self):
+ def test_positional_only_feature_version(self):
+ ast.parse('def foo(x, /): ...', feature_version=(3, 8))
+ ast.parse('def bar(x=1, /): ...', feature_version=(3, 8))
+ with self.assertRaises(SyntaxError):
+ ast.parse('def foo(x, /): ...', feature_version=(3, 7))
+ with self.assertRaises(SyntaxError):
+ ast.parse('def bar(x=1, /): ...', feature_version=(3, 7))
+
+ ast.parse('lambda x, /: ...', feature_version=(3, 8))
+ ast.parse('lambda x=1, /: ...', feature_version=(3, 8))
+ with self.assertRaises(SyntaxError):
+ ast.parse('lambda x, /: ...', feature_version=(3, 7))
+ with self.assertRaises(SyntaxError):
+ ast.parse('lambda x=1, /: ...', feature_version=(3, 7))
+
+ def test_parenthesized_with_feature_version(self):
+ ast.parse('with (CtxManager() as example): ...', feature_version=(3, 10))
+ # While advertised as a feature in Python 3.10, this was allowed starting 3.9
+ ast.parse('with (CtxManager() as example): ...', feature_version=(3, 9))
+ with self.assertRaises(SyntaxError):
+ ast.parse('with (CtxManager() as example): ...', feature_version=(3, 8))
+ ast.parse('with CtxManager() as example: ...', feature_version=(3, 8))
+
+ def test_debug_f_string_feature_version(self):
ast.parse('f"{x=}"', feature_version=(3, 8))
with self.assertRaises(SyntaxError):
ast.parse('f"{x=}"', feature_version=(3, 7))
+ def test_assignment_expression_feature_version(self):
+ ast.parse('(x := 0)', feature_version=(3, 8))
+ with self.assertRaises(SyntaxError):
+ ast.parse('(x := 0)', feature_version=(3, 7))
+
def test_constant_as_name(self):
for constant in "True", "False", "None":
expr = ast.Expression(ast.Name(constant, ast.Load()))
ast.fix_missing_locations(expr)
- with self.assertRaisesRegex(ValueError, f"Name node can't be used with '{constant}' constant"):
+ with self.assertRaisesRegex(ValueError, f"identifier field can't represent '{constant}' constant"):
compile(expr, "", "eval")
- def test_precedence_enum(self):
- class _Precedence(enum.IntEnum):
- """Precedence table that originated from python grammar."""
- TUPLE = enum.auto()
- YIELD = enum.auto() # 'yield', 'yield from'
- TEST = enum.auto() # 'if'-'else', 'lambda'
- OR = enum.auto() # 'or'
- AND = enum.auto() # 'and'
- NOT = enum.auto() # 'not'
- CMP = enum.auto() # '<', '>', '==', '>=', '<=', '!=',
- # 'in', 'not in', 'is', 'is not'
- EXPR = enum.auto()
- BOR = EXPR # '|'
- BXOR = enum.auto() # '^'
- BAND = enum.auto() # '&'
- SHIFT = enum.auto() # '<<', '>>'
- ARITH = enum.auto() # '+', '-'
- TERM = enum.auto() # '*', '@', '/', '%', '//'
- FACTOR = enum.auto() # unary '+', '-', '~'
- POWER = enum.auto() # '**'
- AWAIT = enum.auto() # 'await'
- ATOM = enum.auto()
- def next(self):
- try:
- return self.__class__(self + 1)
- except ValueError:
- return self
- enum._test_simple_enum(_Precedence, ast._Precedence)
-
class ASTHelpers_Test(unittest.TestCase):
maxDiff = None
@@ -916,6 +932,18 @@ def test_increment_lineno(self):
self.assertEqual(ast.increment_lineno(src).lineno, 2)
self.assertIsNone(ast.increment_lineno(src).end_lineno)
+ def test_increment_lineno_on_module(self):
+ src = ast.parse(dedent("""\
+ a = 1
+ b = 2 # type: ignore
+ c = 3
+ d = 4 # type: ignore@tag
+ """), type_comments=True)
+ ast.increment_lineno(src, n=5)
+ self.assertEqual(src.type_ignores[0].lineno, 7)
+ self.assertEqual(src.type_ignores[1].lineno, 9)
+ self.assertEqual(src.type_ignores[1].tag, '@tag')
+
def test_iter_fields(self):
node = ast.parse('foo()', mode='eval')
d = dict(ast.iter_fields(node.body))
@@ -1029,6 +1057,14 @@ def test_literal_eval(self):
self.assertRaises(ValueError, ast.literal_eval, '+True')
self.assertRaises(ValueError, ast.literal_eval, '2+3')
+ def test_literal_eval_str_int_limit(self):
+ with support.adjust_int_max_str_digits(4000):
+ ast.literal_eval('3'*4000) # no error
+ with self.assertRaises(SyntaxError) as err_ctx:
+ ast.literal_eval('3'*4001)
+ self.assertIn('Exceeds the limit ', str(err_ctx.exception))
+ self.assertIn(' Consider hexadecimal ', str(err_ctx.exception))
+
def test_literal_eval_complex(self):
# Issue #4907
self.assertEqual(ast.literal_eval('6j'), 6j)
@@ -1074,6 +1110,13 @@ def test_literal_eval_malformed_lineno(self):
with self.assertRaisesRegex(ValueError, msg):
ast.literal_eval(node)
+ def test_literal_eval_syntax_errors(self):
+ with self.assertRaisesRegex(SyntaxError, "unexpected indent"):
+ ast.literal_eval(r'''
+ \
+ (\
+ \ ''')
+
def test_bad_integer(self):
# issue13436: Bad error message with invalid numeric values
body = [ast.ImportFrom(module='time',
@@ -1097,6 +1140,22 @@ def test_level_as_none(self):
exec(code, ns)
self.assertIn('sleep', ns)
+ def test_recursion_direct(self):
+ e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0)
+ e.operand = e
+ with self.assertRaises(RecursionError):
+ with support.infinite_recursion():
+ compile(ast.Expression(e), "", "eval")
+
+ def test_recursion_indirect(self):
+ e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0)
+ f = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0)
+ e.operand = f
+ f.operand = e
+ with self.assertRaises(RecursionError):
+ with support.infinite_recursion():
+ compile(ast.Expression(e), "", "eval")
+
class ASTValidatorTests(unittest.TestCase):
@@ -1490,6 +1549,151 @@ def test_stdlib_validates(self):
mod = ast.parse(source, fn)
compile(mod, fn, "exec")
+ constant_1 = ast.Constant(1)
+ pattern_1 = ast.MatchValue(constant_1)
+
+ constant_x = ast.Constant('x')
+ pattern_x = ast.MatchValue(constant_x)
+
+ constant_true = ast.Constant(True)
+ pattern_true = ast.MatchSingleton(True)
+
+ name_carter = ast.Name('carter', ast.Load())
+
+ _MATCH_PATTERNS = [
+ ast.MatchValue(
+ ast.Attribute(
+ ast.Attribute(
+ ast.Name('x', ast.Store()),
+ 'y', ast.Load()
+ ),
+ 'z', ast.Load()
+ )
+ ),
+ ast.MatchValue(
+ ast.Attribute(
+ ast.Attribute(
+ ast.Name('x', ast.Load()),
+ 'y', ast.Store()
+ ),
+ 'z', ast.Load()
+ )
+ ),
+ ast.MatchValue(
+ ast.Constant(...)
+ ),
+ ast.MatchValue(
+ ast.Constant(True)
+ ),
+ ast.MatchValue(
+ ast.Constant((1,2,3))
+ ),
+ ast.MatchSingleton('string'),
+ ast.MatchSequence([
+ ast.MatchSingleton('string')
+ ]),
+ ast.MatchSequence(
+ [
+ ast.MatchSequence(
+ [
+ ast.MatchSingleton('string')
+ ]
+ )
+ ]
+ ),
+ ast.MatchMapping(
+ [constant_1, constant_true],
+ [pattern_x]
+ ),
+ ast.MatchMapping(
+ [constant_true, constant_1],
+ [pattern_x, pattern_1],
+ rest='True'
+ ),
+ ast.MatchMapping(
+ [constant_true, ast.Starred(ast.Name('lol', ast.Load()), ast.Load())],
+ [pattern_x, pattern_1],
+ rest='legit'
+ ),
+ ast.MatchClass(
+ ast.Attribute(
+ ast.Attribute(
+ constant_x,
+ 'y', ast.Load()),
+ 'z', ast.Load()),
+ patterns=[], kwd_attrs=[], kwd_patterns=[]
+ ),
+ ast.MatchClass(
+ name_carter,
+ patterns=[],
+ kwd_attrs=['True'],
+ kwd_patterns=[pattern_1]
+ ),
+ ast.MatchClass(
+ name_carter,
+ patterns=[],
+ kwd_attrs=[],
+ kwd_patterns=[pattern_1]
+ ),
+ ast.MatchClass(
+ name_carter,
+ patterns=[ast.MatchSingleton('string')],
+ kwd_attrs=[],
+ kwd_patterns=[]
+ ),
+ ast.MatchClass(
+ name_carter,
+ patterns=[ast.MatchStar()],
+ kwd_attrs=[],
+ kwd_patterns=[]
+ ),
+ ast.MatchClass(
+ name_carter,
+ patterns=[],
+ kwd_attrs=[],
+ kwd_patterns=[ast.MatchStar()]
+ ),
+ ast.MatchSequence(
+ [
+ ast.MatchStar("True")
+ ]
+ ),
+ ast.MatchAs(
+ name='False'
+ ),
+ ast.MatchOr(
+ []
+ ),
+ ast.MatchOr(
+ [pattern_1]
+ ),
+ ast.MatchOr(
+ [pattern_1, pattern_x, ast.MatchSingleton('xxx')]
+ ),
+ ast.MatchAs(name="_"),
+ ast.MatchStar(name="x"),
+ ast.MatchSequence([ast.MatchStar("_")]),
+ ast.MatchMapping([], [], rest="_"),
+ ]
+
+ def test_match_validation_pattern(self):
+ name_x = ast.Name('x', ast.Load())
+ for pattern in self._MATCH_PATTERNS:
+ with self.subTest(ast.dump(pattern, indent=4)):
+ node = ast.Match(
+ subject=name_x,
+ cases = [
+ ast.match_case(
+ pattern=pattern,
+ body = [ast.Pass()]
+ )
+ ]
+ )
+ node = ast.fix_missing_locations(node)
+ module = ast.Module([node], [])
+ with self.assertRaises(ValueError):
+ compile(module, "", "exec")
+
class ConstantTests(unittest.TestCase):
"""Tests on the ast.Constant node type."""
diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py
index 77c15c02bc8914..473bce484b47b0 100644
--- a/Lib/test/test_asyncgen.py
+++ b/Lib/test/test_asyncgen.py
@@ -1,11 +1,16 @@
import inspect
import types
import unittest
+import contextlib
from test.support.import_helper import import_module
+from test.support import gc_collect
asyncio = import_module("asyncio")
+_no_default = object()
+
+
class AwaitException(Exception):
pass
@@ -44,6 +49,37 @@ async def iterate():
return run_until_complete(iterate())
+def py_anext(iterator, default=_no_default):
+ """Pure-Python implementation of anext() for testing purposes.
+
+ Closely matches the builtin anext() C implementation.
+ Can be used to compare the built-in implementation of the inner
+ coroutines machinery to C-implementation of __anext__() and send()
+ or throw() on the returned generator.
+ """
+
+ try:
+ __anext__ = type(iterator).__anext__
+ except AttributeError:
+ raise TypeError(f'{iterator!r} is not an async iterator')
+
+ if default is _no_default:
+ return __anext__(iterator)
+
+ async def anext_impl():
+ try:
+ # The C code is way more low-level than this, as it implements
+ # all methods of the iterator protocol. In this implementation
+ # we're relying on higher-level coroutine concepts, but that's
+ # exactly what we want -- crosstest pure-Python high-level
+ # implementation and low-level C anext() iterators.
+ return await __anext__(iterator)
+ except StopAsyncIteration:
+ return default
+
+ return anext_impl()
+
+
class AsyncGenSyntaxTest(unittest.TestCase):
def test_async_gen_syntax_01(self):
@@ -373,6 +409,12 @@ def tearDown(self):
asyncio.set_event_loop_policy(None)
def check_async_iterator_anext(self, ait_class):
+ with self.subTest(anext="pure-Python"):
+ self._check_async_iterator_anext(ait_class, py_anext)
+ with self.subTest(anext="builtin"):
+ self._check_async_iterator_anext(ait_class, anext)
+
+ def _check_async_iterator_anext(self, ait_class, anext):
g = ait_class()
async def consume():
results = []
@@ -405,6 +447,24 @@ async def test_2():
result = self.loop.run_until_complete(test_2())
self.assertEqual(result, "completed")
+ def test_send():
+ p = ait_class()
+ obj = anext(p, "completed")
+ with self.assertRaises(StopIteration):
+ with contextlib.closing(obj.__await__()) as g:
+ g.send(None)
+
+ test_send()
+
+ async def test_throw():
+ p = ait_class()
+ obj = anext(p, "completed")
+ self.assertRaises(SyntaxError, obj.throw, SyntaxError)
+ return "completed"
+
+ result = self.loop.run_until_complete(test_throw())
+ self.assertEqual(result, "completed")
+
def test_async_generator_anext(self):
async def agen():
yield 1
@@ -568,6 +628,119 @@ async def do_test():
result = self.loop.run_until_complete(do_test())
self.assertEqual(result, "completed")
+ def test_anext_iter(self):
+ @types.coroutine
+ def _async_yield(v):
+ return (yield v)
+
+ class MyError(Exception):
+ pass
+
+ async def agenfn():
+ try:
+ await _async_yield(1)
+ except MyError:
+ await _async_yield(2)
+ return
+ yield
+
+ def test1(anext):
+ agen = agenfn()
+ with contextlib.closing(anext(agen, "default").__await__()) as g:
+ self.assertEqual(g.send(None), 1)
+ self.assertEqual(g.throw(MyError, MyError(), None), 2)
+ try:
+ g.send(None)
+ except StopIteration as e:
+ err = e
+ else:
+ self.fail('StopIteration was not raised')
+ self.assertEqual(err.value, "default")
+
+ def test2(anext):
+ agen = agenfn()
+ with contextlib.closing(anext(agen, "default").__await__()) as g:
+ self.assertEqual(g.send(None), 1)
+ self.assertEqual(g.throw(MyError, MyError(), None), 2)
+ with self.assertRaises(MyError):
+ g.throw(MyError, MyError(), None)
+
+ def test3(anext):
+ agen = agenfn()
+ with contextlib.closing(anext(agen, "default").__await__()) as g:
+ self.assertEqual(g.send(None), 1)
+ g.close()
+ with self.assertRaisesRegex(RuntimeError, 'cannot reuse'):
+ self.assertEqual(g.send(None), 1)
+
+ def test4(anext):
+ @types.coroutine
+ def _async_yield(v):
+ yield v * 10
+ return (yield (v * 10 + 1))
+
+ async def agenfn():
+ try:
+ await _async_yield(1)
+ except MyError:
+ await _async_yield(2)
+ return
+ yield
+
+ agen = agenfn()
+ with contextlib.closing(anext(agen, "default").__await__()) as g:
+ self.assertEqual(g.send(None), 10)
+ self.assertEqual(g.throw(MyError, MyError(), None), 20)
+ with self.assertRaisesRegex(MyError, 'val'):
+ g.throw(MyError, MyError('val'), None)
+
+ def test5(anext):
+ @types.coroutine
+ def _async_yield(v):
+ yield v * 10
+ return (yield (v * 10 + 1))
+
+ async def agenfn():
+ try:
+ await _async_yield(1)
+ except MyError:
+ return
+ yield 'aaa'
+
+ agen = agenfn()
+ with contextlib.closing(anext(agen, "default").__await__()) as g:
+ self.assertEqual(g.send(None), 10)
+ with self.assertRaisesRegex(StopIteration, 'default'):
+ g.throw(MyError, MyError(), None)
+
+ def test6(anext):
+ @types.coroutine
+ def _async_yield(v):
+ yield v * 10
+ return (yield (v * 10 + 1))
+
+ async def agenfn():
+ await _async_yield(1)
+ yield 'aaa'
+
+ agen = agenfn()
+ with contextlib.closing(anext(agen, "default").__await__()) as g:
+ with self.assertRaises(MyError):
+ g.throw(MyError, MyError(), None)
+
+ def run_test(test):
+ with self.subTest('pure-Python anext()'):
+ test(py_anext)
+ with self.subTest('builtin anext()'):
+ test(anext)
+
+ run_test(test1)
+ run_test(test2)
+ run_test(test3)
+ run_test(test4)
+ run_test(test5)
+ run_test(test6)
+
def test_aiter_bad_args(self):
async def gen():
yield 1
@@ -871,6 +1044,7 @@ async def run():
await g.__anext__()
await g.__anext__()
del g
+ gc_collect() # For PyPy or other GCs.
await asyncio.sleep(0.1)
@@ -1287,6 +1461,85 @@ async def wait():
self.assertEqual(finalized, 2)
+ def test_async_gen_asyncio_shutdown_02(self):
+ messages = []
+
+ def exception_handler(loop, context):
+ messages.append(context)
+
+ async def async_iterate():
+ yield 1
+ yield 2
+
+ it = async_iterate()
+ async def main():
+ loop = asyncio.get_running_loop()
+ loop.set_exception_handler(exception_handler)
+
+ async for i in it:
+ break
+
+ asyncio.run(main())
+
+ self.assertEqual(messages, [])
+
+ def test_async_gen_asyncio_shutdown_exception_01(self):
+ messages = []
+
+ def exception_handler(loop, context):
+ messages.append(context)
+
+ async def async_iterate():
+ try:
+ yield 1
+ yield 2
+ finally:
+ 1/0
+
+ it = async_iterate()
+ async def main():
+ loop = asyncio.get_running_loop()
+ loop.set_exception_handler(exception_handler)
+
+ async for i in it:
+ break
+
+ asyncio.run(main())
+
+ message, = messages
+ self.assertEqual(message['asyncgen'], it)
+ self.assertIsInstance(message['exception'], ZeroDivisionError)
+ self.assertIn('an error occurred during closing of asynchronous generator',
+ message['message'])
+
+ def test_async_gen_asyncio_shutdown_exception_02(self):
+ messages = []
+
+ def exception_handler(loop, context):
+ messages.append(context)
+
+ async def async_iterate():
+ try:
+ yield 1
+ yield 2
+ finally:
+ 1/0
+
+ async def main():
+ loop = asyncio.get_running_loop()
+ loop.set_exception_handler(exception_handler)
+
+ async for i in async_iterate():
+ break
+ gc_collect()
+
+ asyncio.run(main())
+
+ message, = messages
+ self.assertIsInstance(message['exception'], ZeroDivisionError)
+ self.assertIn('unhandled exception during asyncio.run() shutdown',
+ message['message'])
+
def test_async_gen_expression_01(self):
async def arange(n):
for i in range(n):
diff --git a/Lib/test/test_asynchat.py b/Lib/test/test_asynchat.py
index b32edddc7d5505..973ac1f7d97c9b 100644
--- a/Lib/test/test_asynchat.py
+++ b/Lib/test/test_asynchat.py
@@ -4,8 +4,6 @@
from test.support import socket_helper
from test.support import threading_helper
-import asynchat
-import asyncore
import errno
import socket
import sys
@@ -14,6 +12,12 @@
import unittest
import unittest.mock
+import warnings
+with warnings.catch_warnings():
+ warnings.simplefilter('ignore', DeprecationWarning)
+ import asynchat
+ import asyncore
+
HOST = socket_helper.HOST
SERVER_QUIT = b'QUIT\n'
diff --git a/Lib/test/test_asyncio/functional.py b/Lib/test/test_asyncio/functional.py
index 74490a869de9d9..d19c7a612ccf86 100644
--- a/Lib/test/test_asyncio/functional.py
+++ b/Lib/test/test_asyncio/functional.py
@@ -29,10 +29,6 @@ def setUp(self):
self.loop.set_exception_handler(self.loop_exception_handler)
self.__unhandled_exceptions = []
- # Disable `_get_running_loop`.
- self._old_get_running_loop = asyncio.events._get_running_loop
- asyncio.events._get_running_loop = lambda: None
-
def tearDown(self):
try:
self.loop.close()
@@ -43,7 +39,6 @@ def tearDown(self):
self.fail('unexpected calls to loop.call_exception_handler()')
finally:
- asyncio.events._get_running_loop = self._old_get_running_loop
asyncio.set_event_loop(None)
self.loop = None
diff --git a/Lib/test/test_asyncio/test_asyncio_waitfor.py b/Lib/test/test_asyncio/test_asyncio_waitfor.py
deleted file mode 100644
index 2ca64abbeb527c..00000000000000
--- a/Lib/test/test_asyncio/test_asyncio_waitfor.py
+++ /dev/null
@@ -1,61 +0,0 @@
-import asyncio
-import unittest
-import time
-
-def tearDownModule():
- asyncio.set_event_loop_policy(None)
-
-
-class SlowTask:
- """ Task will run for this defined time, ignoring cancel requests """
- TASK_TIMEOUT = 0.2
-
- def __init__(self):
- self.exited = False
-
- async def run(self):
- exitat = time.monotonic() + self.TASK_TIMEOUT
-
- while True:
- tosleep = exitat - time.monotonic()
- if tosleep <= 0:
- break
-
- try:
- await asyncio.sleep(tosleep)
- except asyncio.CancelledError:
- pass
-
- self.exited = True
-
-class AsyncioWaitForTest(unittest.TestCase):
-
- async def atest_asyncio_wait_for_cancelled(self):
- t = SlowTask()
-
- waitfortask = asyncio.create_task(asyncio.wait_for(t.run(), t.TASK_TIMEOUT * 2))
- await asyncio.sleep(0)
- waitfortask.cancel()
- await asyncio.wait({waitfortask})
-
- self.assertTrue(t.exited)
-
- def test_asyncio_wait_for_cancelled(self):
- asyncio.run(self.atest_asyncio_wait_for_cancelled())
-
- async def atest_asyncio_wait_for_timeout(self):
- t = SlowTask()
-
- try:
- await asyncio.wait_for(t.run(), t.TASK_TIMEOUT / 2)
- except asyncio.TimeoutError:
- pass
-
- self.assertTrue(t.exited)
-
- def test_asyncio_wait_for_timeout(self):
- asyncio.run(self.atest_asyncio_wait_for_timeout())
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index 5691d4250aca9e..7f7e371db1b4f3 100644
--- a/Lib/test/test_asyncio/test_base_events.py
+++ b/Lib/test/test_asyncio/test_base_events.py
@@ -18,7 +18,7 @@
from test.support.script_helper import assert_python_ok
from test.support import os_helper
from test.support import socket_helper
-
+import warnings
MOCK_ANY = mock.ANY
PY34 = sys.version_info >= (3, 4)
@@ -643,7 +643,7 @@ def run_loop():
'Exception in callback.*zero'),
exc_info=(ZeroDivisionError, MOCK_ANY, MOCK_ANY))
- assert not mock_handler.called
+ self.assertFalse(mock_handler.called)
def test_set_exc_handler_broken(self):
def run_loop():
@@ -752,7 +752,7 @@ async def coro():
def test_env_var_debug(self):
code = '\n'.join((
'import asyncio',
- 'loop = asyncio.get_event_loop()',
+ 'loop = asyncio.new_event_loop()',
'print(loop.get_debug())'))
# Test with -E to not fail if the unit test was run with
@@ -801,6 +801,17 @@ def create_task(self, coro):
task._log_destroy_pending = False
coro.close()
+ def test_create_task_error_closes_coro(self):
+ async def test():
+ pass
+ loop = asyncio.new_event_loop()
+ loop.close()
+ with warnings.catch_warnings(record=True) as w:
+ with self.assertRaises(RuntimeError):
+ asyncio.ensure_future(test(), loop=loop)
+ self.assertEqual(len(w), 0)
+
+
def test_create_named_task_with_default_factory(self):
async def test():
pass
@@ -1003,22 +1014,26 @@ def __init__(self, create_future=False):
if create_future:
self.done = asyncio.get_running_loop().create_future()
+ def _assert_state(self, *expected):
+ if self.state not in expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'CONNECTED'
transport.write(b'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n')
def data_received(self, data):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.nbytes += len(data)
def eof_received(self):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.state = 'EOF'
def connection_lost(self, exc):
- assert self.state in ('CONNECTED', 'EOF'), self.state
+ self._assert_state('CONNECTED', 'EOF')
self.state = 'CLOSED'
if self.done:
self.done.set_result(None)
@@ -1033,20 +1048,24 @@ def __init__(self, create_future=False, loop=None):
if create_future:
self.done = loop.create_future()
+ def _assert_state(self, expected):
+ if self.state != expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'INITIALIZED'
def datagram_received(self, data, addr):
- assert self.state == 'INITIALIZED', self.state
+ self._assert_state('INITIALIZED')
self.nbytes += len(data)
def error_received(self, exc):
- assert self.state == 'INITIALIZED', self.state
+ self._assert_state('INITIALIZED')
def connection_lost(self, exc):
- assert self.state == 'INITIALIZED', self.state
+ self._assert_state('INITIALIZED')
self.state = 'CLOSED'
if self.done:
self.done.set_result(None)
@@ -1395,7 +1414,7 @@ def test_create_connection_bluetooth(self, m_socket):
addr = ('00:01:02:03:04:05', 1)
def getaddrinfo(host, port, *args, **kw):
- assert (host, port) == addr
+ self.assertEqual((host, port), addr)
return [(999, 1, 999, '', (addr, 1))]
m_socket.getaddrinfo = getaddrinfo
@@ -1694,7 +1713,7 @@ def test_create_datagram_endpoint_sock_unix(self):
lambda: MyDatagramProto(create_future=True, loop=self.loop),
family=socket.AF_UNIX)
transport, protocol = self.loop.run_until_complete(fut)
- assert transport._sock.family == socket.AF_UNIX
+ self.assertEqual(transport._sock.family, socket.AF_UNIX)
transport.close()
self.loop.run_until_complete(protocol.done)
self.assertEqual('CLOSED', protocol.state)
diff --git a/Lib/test/test_asyncio/test_context.py b/Lib/test/test_asyncio/test_context.py
index 63b1eb320ce16b..6b80721873d95c 100644
--- a/Lib/test/test_asyncio/test_context.py
+++ b/Lib/test/test_asyncio/test_context.py
@@ -32,3 +32,7 @@ async def main():
self.assertEqual(str(r2[0]), '0.333333')
self.assertEqual(str(r2[1]), '0.111111')
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index 55fc266cb714b1..253a6c119c9e83 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -87,23 +87,27 @@ def __init__(self, loop=None):
self.connected = loop.create_future()
self.done = loop.create_future()
+ def _assert_state(self, *expected):
+ if self.state not in expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'CONNECTED'
if self.connected:
self.connected.set_result(None)
def data_received(self, data):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.nbytes += len(data)
def eof_received(self):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.state = 'EOF'
def connection_lost(self, exc):
- assert self.state in ('CONNECTED', 'EOF'), self.state
+ self._assert_state('CONNECTED', 'EOF')
self.state = 'CLOSED'
if self.done:
self.done.set_result(None)
@@ -124,20 +128,24 @@ def __init__(self, loop=None):
if loop is not None:
self.done = loop.create_future()
+ def _assert_state(self, expected):
+ if self.state != expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'INITIALIZED'
def datagram_received(self, data, addr):
- assert self.state == 'INITIALIZED', self.state
+ self._assert_state('INITIALIZED')
self.nbytes += len(data)
def error_received(self, exc):
- assert self.state == 'INITIALIZED', self.state
+ self._assert_state('INITIALIZED')
def connection_lost(self, exc):
- assert self.state == 'INITIALIZED', self.state
+ self._assert_state('INITIALIZED')
self.state = 'CLOSED'
if self.done:
self.done.set_result(None)
@@ -153,23 +161,27 @@ def __init__(self, loop=None):
if loop is not None:
self.done = loop.create_future()
+ def _assert_state(self, expected):
+ if self.state != expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == ['INITIAL'], self.state
+ self._assert_state(['INITIAL'])
self.state.append('CONNECTED')
def data_received(self, data):
- assert self.state == ['INITIAL', 'CONNECTED'], self.state
+ self._assert_state(['INITIAL', 'CONNECTED'])
self.nbytes += len(data)
def eof_received(self):
- assert self.state == ['INITIAL', 'CONNECTED'], self.state
+ self._assert_state(['INITIAL', 'CONNECTED'])
self.state.append('EOF')
def connection_lost(self, exc):
if 'EOF' not in self.state:
self.state.append('EOF') # It is okay if EOF is missed.
- assert self.state == ['INITIAL', 'CONNECTED', 'EOF'], self.state
+ self._assert_state(['INITIAL', 'CONNECTED', 'EOF'])
self.state.append('CLOSED')
if self.done:
self.done.set_result(None)
@@ -184,13 +196,17 @@ def __init__(self, loop=None):
if loop is not None:
self.done = loop.create_future()
+ def _assert_state(self, expected):
+ if self.state != expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'CONNECTED'
def connection_lost(self, exc):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.state = 'CLOSED'
if self.done:
self.done.set_result(None)
@@ -209,31 +225,35 @@ def __init__(self, loop):
self.got_data = {1: asyncio.Event(),
2: asyncio.Event()}
+ def _assert_state(self, expected):
+ if self.state != expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'CONNECTED'
self.connected.set_result(None)
def connection_lost(self, exc):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.state = 'CLOSED'
self.completed.set_result(None)
def pipe_data_received(self, fd, data):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.data[fd] += data
self.got_data[fd].set()
def pipe_connection_lost(self, fd, exc):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
if exc:
self.disconnects[fd].set_exception(exc)
else:
self.disconnects[fd].set_result(exc)
def process_exited(self):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.returncode = self.transport.get_returncode()
@@ -650,6 +670,47 @@ def test_create_connection_local_addr(self):
self.assertEqual(port, expected)
tr.close()
+ def test_create_connection_local_addr_skip_different_family(self):
+ # See https://github.com/python/cpython/issues/86508
+ port1 = socket_helper.find_unused_port()
+ port2 = socket_helper.find_unused_port()
+ getaddrinfo_orig = self.loop.getaddrinfo
+
+ async def getaddrinfo(host, port, *args, **kwargs):
+ if port == port2:
+ return [(socket.AF_INET6, socket.SOCK_STREAM, 0, '', ('::1', 0, 0, 0)),
+ (socket.AF_INET, socket.SOCK_STREAM, 0, '', ('127.0.0.1', 0))]
+ return await getaddrinfo_orig(host, port, *args, **kwargs)
+
+ self.loop.getaddrinfo = getaddrinfo
+
+ f = self.loop.create_connection(
+ lambda: MyProto(loop=self.loop),
+ 'localhost', port1, local_addr=('localhost', port2))
+
+ with self.assertRaises(OSError):
+ self.loop.run_until_complete(f)
+
+ def test_create_connection_local_addr_nomatch_family(self):
+ # See https://github.com/python/cpython/issues/86508
+ port1 = socket_helper.find_unused_port()
+ port2 = socket_helper.find_unused_port()
+ getaddrinfo_orig = self.loop.getaddrinfo
+
+ async def getaddrinfo(host, port, *args, **kwargs):
+ if port == port2:
+ return [(socket.AF_INET6, socket.SOCK_STREAM, 0, '', ('::1', 0, 0, 0))]
+ return await getaddrinfo_orig(host, port, *args, **kwargs)
+
+ self.loop.getaddrinfo = getaddrinfo
+
+ f = self.loop.create_connection(
+ lambda: MyProto(loop=self.loop),
+ 'localhost', port1, local_addr=('localhost', port2))
+
+ with self.assertRaises(OSError):
+ self.loop.run_until_complete(f)
+
def test_create_connection_local_addr_in_use(self):
with test_utils.run_test_server() as httpd:
f = self.loop.create_connection(
@@ -1283,7 +1344,7 @@ def test_create_datagram_endpoint_sock(self):
else:
break
else:
- assert False, 'Can not create socket.'
+ self.fail('Can not create socket.')
f = self.loop.create_datagram_endpoint(
lambda: MyDatagramProto(loop=self.loop), sock=sock)
@@ -1970,10 +2031,11 @@ def test_subprocess_wait_no_same_group(self):
functools.partial(MySubprocessProtocol, self.loop),
'exit 7', stdin=None, stdout=None, stderr=None,
start_new_session=True)
- _, proto = yield self.loop.run_until_complete(connect)
+ transp, proto = self.loop.run_until_complete(connect)
self.assertIsInstance(proto, MySubprocessProtocol)
self.loop.run_until_complete(proto.completed)
self.assertEqual(7, proto.returncode)
+ transp.close()
def test_subprocess_exec_invalid_args(self):
async def connect(**kwds):
@@ -2540,7 +2602,6 @@ def test_event_loop_policy(self):
def test_get_event_loop(self):
policy = asyncio.DefaultEventLoopPolicy()
self.assertIsNone(policy._local._loop)
-
loop = policy.get_event_loop()
self.assertIsInstance(loop, asyncio.AbstractEventLoop)
@@ -2556,6 +2617,7 @@ def test_get_event_loop_calls_set_event_loop(self):
wraps=policy.set_event_loop) as m_set_event_loop:
loop = policy.get_event_loop()
+ self.addCleanup(loop.close)
# policy._local._loop must be set through .set_event_loop()
# (the unix DefaultEventLoopPolicy needs this call to attach
@@ -2589,7 +2651,8 @@ def test_new_event_loop(self):
def test_set_event_loop(self):
policy = asyncio.DefaultEventLoopPolicy()
- old_loop = policy.get_event_loop()
+ old_loop = policy.new_event_loop()
+ policy.set_event_loop(old_loop)
self.assertRaises(AssertionError, policy.set_event_loop, object())
@@ -2702,15 +2765,11 @@ def get_event_loop(self):
asyncio.set_event_loop_policy(Policy())
loop = asyncio.new_event_loop()
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaises(TestError):
- asyncio.get_event_loop()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaises(TestError):
+ asyncio.get_event_loop()
asyncio.set_event_loop(None)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaises(TestError):
- asyncio.get_event_loop()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaises(TestError):
+ asyncio.get_event_loop()
with self.assertRaisesRegex(RuntimeError, 'no running'):
asyncio.get_running_loop()
@@ -2724,16 +2783,11 @@ async def func():
loop.run_until_complete(func())
asyncio.set_event_loop(loop)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaises(TestError):
- asyncio.get_event_loop()
- self.assertEqual(cm.warnings[0].filename, __file__)
-
+ with self.assertRaises(TestError):
+ asyncio.get_event_loop()
asyncio.set_event_loop(None)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaises(TestError):
- asyncio.get_event_loop()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaises(TestError):
+ asyncio.get_event_loop()
finally:
asyncio.set_event_loop_policy(old_policy)
@@ -2752,15 +2806,11 @@ def test_get_event_loop_returns_running_loop2(self):
loop = asyncio.new_event_loop()
self.addCleanup(loop.close)
- with self.assertWarns(DeprecationWarning) as cm:
- loop2 = asyncio.get_event_loop()
+ loop2 = asyncio.get_event_loop()
self.addCleanup(loop2.close)
- self.assertEqual(cm.warnings[0].filename, __file__)
asyncio.set_event_loop(None)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'no current'):
- asyncio.get_event_loop()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current'):
+ asyncio.get_event_loop()
with self.assertRaisesRegex(RuntimeError, 'no running'):
asyncio.get_running_loop()
@@ -2774,15 +2824,11 @@ async def func():
loop.run_until_complete(func())
asyncio.set_event_loop(loop)
- with self.assertWarns(DeprecationWarning) as cm:
- self.assertIs(asyncio.get_event_loop(), loop)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ self.assertIs(asyncio.get_event_loop(), loop)
asyncio.set_event_loop(None)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'no current'):
- asyncio.get_event_loop()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current'):
+ asyncio.get_event_loop()
finally:
asyncio.set_event_loop_policy(old_policy)
diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py
index fe3d44227c83cc..0b47aa28df661e 100644
--- a/Lib/test/test_asyncio/test_futures.py
+++ b/Lib/test/test_asyncio/test_futures.py
@@ -7,7 +7,7 @@
import threading
import unittest
from unittest import mock
-
+from types import GenericAlias
import asyncio
from asyncio import futures
from test.test_asyncio import utils as test_utils
@@ -54,30 +54,30 @@ def done(self):
or self.__exception is not None)
def result(self):
- assert not self.cancelled()
+ self.assertFalse(self.cancelled())
if self.__exception is not None:
raise self.__exception
return self.__result
def exception(self):
- assert not self.cancelled()
+ self.assertFalse(self.cancelled())
return self.__exception
def set_result(self, result):
- assert not self.done()
- assert result is not None
+ self.assertFalse(self.done())
+ self.assertIsNotNone(result)
self.__result = result
def set_exception(self, exception):
- assert not self.done()
- assert exception is not None
+ self.assertFalse(self.done())
+ self.assertIsNotNone(exception)
self.__exception = exception
def __iter__(self):
if not self.done():
self._asyncio_future_blocking = True
yield self
- assert self.done()
+ self.assertTrue(self.done())
return self.result()
@@ -91,12 +91,12 @@ def setUp(self):
def test_wrap_future(self):
f = DuckFuture()
g = asyncio.wrap_future(f)
- assert g is f
+ self.assertIs(g, f)
def test_ensure_future(self):
f = DuckFuture()
g = asyncio.ensure_future(f)
- assert g is f
+ self.assertIs(g, f)
class BaseFutureTests:
@@ -109,6 +109,11 @@ def setUp(self):
self.loop = self.new_test_loop()
self.addCleanup(self.loop.close)
+ def test_generic_alias(self):
+ future = self.cls[str]
+ self.assertEqual(future.__args__, (str,))
+ self.assertIsInstance(future, GenericAlias)
+
def test_isfuture(self):
class MyFuture:
_asyncio_future_blocking = None
@@ -140,10 +145,8 @@ def test_initial_state(self):
self.assertTrue(f.cancelled())
def test_constructor_without_loop(self):
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
- self._new_future()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ self._new_future()
def test_constructor_use_running_loop(self):
async def test():
@@ -153,12 +156,10 @@ async def test():
self.assertIs(f.get_loop(), self.loop)
def test_constructor_use_global_loop(self):
- # Deprecated in 3.10
+ # Deprecated in 3.10, undeprecated in 3.11.1
asyncio.set_event_loop(self.loop)
self.addCleanup(asyncio.set_event_loop, None)
- with self.assertWarns(DeprecationWarning) as cm:
- f = self._new_future()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ f = self._new_future()
self.assertIs(f._loop, self.loop)
self.assertIs(f.get_loop(), self.loop)
@@ -494,10 +495,8 @@ def run(arg):
return (arg, threading.get_ident())
ex = concurrent.futures.ThreadPoolExecutor(1)
f1 = ex.submit(run, 'oi')
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaises(RuntimeError):
- asyncio.wrap_future(f1)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ asyncio.wrap_future(f1)
ex.shutdown(wait=True)
def test_wrap_future_use_running_loop(self):
@@ -512,16 +511,14 @@ async def test():
ex.shutdown(wait=True)
def test_wrap_future_use_global_loop(self):
- # Deprecated in 3.10
+ # Deprecated in 3.10, undeprecated in 3.11.1
asyncio.set_event_loop(self.loop)
self.addCleanup(asyncio.set_event_loop, None)
def run(arg):
return (arg, threading.get_ident())
ex = concurrent.futures.ThreadPoolExecutor(1)
f1 = ex.submit(run, 'oi')
- with self.assertWarns(DeprecationWarning) as cm:
- f2 = asyncio.wrap_future(f1)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ f2 = asyncio.wrap_future(f1)
self.assertIs(self.loop, f2._loop)
ex.shutdown(wait=True)
@@ -822,6 +819,21 @@ def __eq__(self, other):
fut.remove_done_callback(evil())
+ def test_remove_done_callbacks_list_clear(self):
+ # see https://github.com/python/cpython/issues/97592 for details
+
+ fut = self._new_future()
+ fut.add_done_callback(str)
+
+ for _ in range(63):
+ fut.add_done_callback(id)
+
+ class evil:
+ def __eq__(self, other):
+ fut.remove_done_callback(other)
+
+ fut.remove_done_callback(evil())
+
def test_schedule_callbacks_list_mutation_1(self):
# see http://bugs.python.org/issue28963 for details
@@ -918,6 +930,8 @@ def _get_future_cls(self):
return futures._PyFuture
+@unittest.skipUnless(hasattr(futures, '_CFuture'),
+ 'requires the C _asyncio module')
class CFutureInheritanceTests(BaseFutureInheritanceTests,
test_utils.TestCase):
def _get_future_cls(self):
diff --git a/Lib/test/test_asyncio/test_futures2.py b/Lib/test/test_asyncio/test_futures2.py
index 13dbc703277c81..71279b69c7929e 100644
--- a/Lib/test/test_asyncio/test_futures2.py
+++ b/Lib/test/test_asyncio/test_futures2.py
@@ -1,9 +1,42 @@
# IsolatedAsyncioTestCase based tests
import asyncio
+import traceback
import unittest
+from asyncio import tasks
-class FutureTests(unittest.IsolatedAsyncioTestCase):
+def tearDownModule():
+ asyncio.set_event_loop_policy(None)
+
+
+class FutureTests:
+
+ async def test_future_traceback(self):
+
+ async def raise_exc():
+ raise TypeError(42)
+
+ future = self.cls(raise_exc())
+
+ for _ in range(5):
+ try:
+ await future
+ except TypeError as e:
+ tb = ''.join(traceback.format_tb(e.__traceback__))
+ self.assertEqual(tb.count("await future"), 1)
+ else:
+ self.fail('TypeError was not raised')
+
+@unittest.skipUnless(hasattr(tasks, '_CTask'),
+ 'requires the C _asyncio module')
+class CFutureTests(FutureTests, unittest.IsolatedAsyncioTestCase):
+ cls = tasks._CTask
+
+class PyFutureTests(FutureTests, unittest.IsolatedAsyncioTestCase):
+ cls = tasks._PyTask
+
+class FutureReprTests(unittest.IsolatedAsyncioTestCase):
+
async def test_recursive_repr_for_pending_tasks(self):
# The call crashes if the guard for recursive call
# in base_futures:_future_repr_info is absent
@@ -16,3 +49,7 @@ async def func():
# The check for returned string is not very reliable but
# exact comparison for the whole string is even weaker.
self.assertIn('...', repr(await asyncio.wait_for(func(), timeout=10)))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/Lib/test/test_asyncio/test_locks.py b/Lib/test/test_asyncio/test_locks.py
index 6194cd06176dac..50aed902325447 100644
--- a/Lib/test/test_asyncio/test_locks.py
+++ b/Lib/test/test_asyncio/test_locks.py
@@ -5,7 +5,7 @@
import re
import asyncio
-from test.test_asyncio import utils as test_utils
+import collections
STR_RGX_REPR = (
r'^<(?P.*?) object at (?P.*?)'
@@ -20,22 +20,18 @@ def tearDownModule():
asyncio.set_event_loop_policy(None)
-class LockTests(test_utils.TestCase):
+class LockTests(unittest.IsolatedAsyncioTestCase):
- def setUp(self):
- super().setUp()
- self.loop = self.new_test_loop()
-
- def test_repr(self):
+ async def test_repr(self):
lock = asyncio.Lock()
self.assertTrue(repr(lock).endswith('[unlocked]>'))
self.assertTrue(RGX_REPR.match(repr(lock)))
- self.loop.run_until_complete(lock.acquire())
+ await lock.acquire()
self.assertTrue(repr(lock).endswith('[locked]>'))
self.assertTrue(RGX_REPR.match(repr(lock)))
- def test_lock(self):
+ async def test_lock(self):
lock = asyncio.Lock()
with self.assertWarns(DeprecationWarning):
@@ -47,11 +43,11 @@ def acquire_lock():
TypeError,
"object is not iterable"
):
- self.loop.run_until_complete(acquire_lock())
+ await acquire_lock()
self.assertFalse(lock.locked())
- def test_lock_doesnt_accept_loop_parameter(self):
+ async def test_lock_doesnt_accept_loop_parameter(self):
primitives_cls = [
asyncio.Lock,
asyncio.Condition,
@@ -60,17 +56,17 @@ def test_lock_doesnt_accept_loop_parameter(self):
asyncio.BoundedSemaphore,
]
+ loop = asyncio.get_running_loop()
+
for cls in primitives_cls:
with self.assertRaisesRegex(
TypeError,
rf'As of 3.10, the \*loop\* parameter was removed from '
rf'{cls.__name__}\(\) since it is no longer necessary'
):
- cls(loop=self.loop)
+ cls(loop=loop)
- def test_lock_by_with_statement(self):
- loop = asyncio.new_event_loop() # don't use TestLoop quirks
- self.set_event_loop(loop)
+ async def test_lock_by_with_statement(self):
primitives = [
asyncio.Lock(),
asyncio.Condition(),
@@ -91,15 +87,15 @@ def test(lock):
pass
self.assertFalse(lock.locked())
- for primitive in primitives:
- loop.run_until_complete(test(primitive))
- self.assertFalse(primitive.locked())
+ for lock in primitives:
+ await test(lock)
+ self.assertFalse(lock.locked())
- def test_acquire(self):
+ async def test_acquire(self):
lock = asyncio.Lock()
result = []
- self.assertTrue(self.loop.run_until_complete(lock.acquire()))
+ self.assertTrue(await lock.acquire())
async def c1(result):
if await lock.acquire():
@@ -116,27 +112,27 @@ async def c3(result):
result.append(3)
return True
- t1 = self.loop.create_task(c1(result))
- t2 = self.loop.create_task(c2(result))
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
lock.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
- t3 = self.loop.create_task(c3(result))
+ t3 = asyncio.create_task(c3(result))
lock.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1, 2], result)
lock.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1, 2, 3], result)
self.assertTrue(t1.done())
@@ -146,18 +142,17 @@ async def c3(result):
self.assertTrue(t3.done())
self.assertTrue(t3.result())
- def test_acquire_cancel(self):
+ async def test_acquire_cancel(self):
lock = asyncio.Lock()
- self.assertTrue(self.loop.run_until_complete(lock.acquire()))
+ self.assertTrue(await lock.acquire())
- task = self.loop.create_task(lock.acquire())
- self.loop.call_soon(task.cancel)
- self.assertRaises(
- asyncio.CancelledError,
- self.loop.run_until_complete, task)
+ task = asyncio.create_task(lock.acquire())
+ asyncio.get_running_loop().call_soon(task.cancel)
+ with self.assertRaises(asyncio.CancelledError):
+ await task
self.assertFalse(lock._waiters)
- def test_cancel_race(self):
+ async def test_cancel_race(self):
# Several tasks:
# - A acquires the lock
# - B is blocked in acquire()
@@ -182,15 +177,15 @@ async def lockit(name, blocker):
finally:
lock.release()
- fa = self.loop.create_future()
- ta = self.loop.create_task(lockit('A', fa))
- test_utils.run_briefly(self.loop)
+ fa = asyncio.get_running_loop().create_future()
+ ta = asyncio.create_task(lockit('A', fa))
+ await asyncio.sleep(0)
self.assertTrue(lock.locked())
- tb = self.loop.create_task(lockit('B', None))
- test_utils.run_briefly(self.loop)
+ tb = asyncio.create_task(lockit('B', None))
+ await asyncio.sleep(0)
self.assertEqual(len(lock._waiters), 1)
- tc = self.loop.create_task(lockit('C', None))
- test_utils.run_briefly(self.loop)
+ tc = asyncio.create_task(lockit('C', None))
+ await asyncio.sleep(0)
self.assertEqual(len(lock._waiters), 2)
# Create the race and check.
@@ -198,16 +193,17 @@ async def lockit(name, blocker):
fa.set_result(None)
tb.cancel()
self.assertTrue(lock._waiters[0].cancelled())
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertFalse(lock.locked())
self.assertTrue(ta.done())
self.assertTrue(tb.cancelled())
- self.assertTrue(tc.done())
+ await tc
- def test_cancel_release_race(self):
+ async def test_cancel_release_race(self):
# Issue 32734
# Acquire 4 locks, cancel second, release first
# and 2 locks are taken at once.
+ loop = asyncio.get_running_loop()
lock = asyncio.Lock()
lock_count = 0
call_count = 0
@@ -219,27 +215,23 @@ async def lockit():
await lock.acquire()
lock_count += 1
- async def lockandtrigger():
- await lock.acquire()
- self.loop.call_soon(trigger)
-
def trigger():
t1.cancel()
lock.release()
- t0 = self.loop.create_task(lockandtrigger())
- t1 = self.loop.create_task(lockit())
- t2 = self.loop.create_task(lockit())
- t3 = self.loop.create_task(lockit())
+ await lock.acquire()
+
+ t1 = asyncio.create_task(lockit())
+ t2 = asyncio.create_task(lockit())
+ t3 = asyncio.create_task(lockit())
- # First loop acquires all
- test_utils.run_briefly(self.loop)
- self.assertTrue(t0.done())
+ # Start scheduled tasks
+ await asyncio.sleep(0)
- # Second loop calls trigger
- test_utils.run_briefly(self.loop)
- # Third loop calls cancellation
- test_utils.run_briefly(self.loop)
+ loop.call_soon(trigger)
+ with self.assertRaises(asyncio.CancelledError):
+ # Wait for cancellation
+ await t1
# Make sure only one lock was taken
self.assertEqual(lock_count, 1)
@@ -249,62 +241,56 @@ def trigger():
# Cleanup the task that is stuck on acquire.
t3.cancel()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertTrue(t3.cancelled())
- def test_finished_waiter_cancelled(self):
+ async def test_finished_waiter_cancelled(self):
lock = asyncio.Lock()
- ta = self.loop.create_task(lock.acquire())
- test_utils.run_briefly(self.loop)
+ await lock.acquire()
self.assertTrue(lock.locked())
- tb = self.loop.create_task(lock.acquire())
- test_utils.run_briefly(self.loop)
+ tb = asyncio.create_task(lock.acquire())
+ await asyncio.sleep(0)
self.assertEqual(len(lock._waiters), 1)
# Create a second waiter, wake up the first, and cancel it.
# Without the fix, the second was not woken up.
- tc = self.loop.create_task(lock.acquire())
- lock.release()
+ tc = asyncio.create_task(lock.acquire())
tb.cancel()
- test_utils.run_briefly(self.loop)
+ lock.release()
+ await asyncio.sleep(0)
self.assertTrue(lock.locked())
- self.assertTrue(ta.done())
self.assertTrue(tb.cancelled())
- def test_release_not_acquired(self):
+ # Cleanup
+ await tc
+
+ async def test_release_not_acquired(self):
lock = asyncio.Lock()
self.assertRaises(RuntimeError, lock.release)
- def test_release_no_waiters(self):
+ async def test_release_no_waiters(self):
lock = asyncio.Lock()
- self.loop.run_until_complete(lock.acquire())
+ await lock.acquire()
self.assertTrue(lock.locked())
lock.release()
self.assertFalse(lock.locked())
- def test_context_manager(self):
- async def f():
- lock = asyncio.Lock()
- self.assertFalse(lock.locked())
-
- async with lock:
- self.assertTrue(lock.locked())
-
- self.assertFalse(lock.locked())
+ async def test_context_manager(self):
+ lock = asyncio.Lock()
+ self.assertFalse(lock.locked())
- self.loop.run_until_complete(f())
+ async with lock:
+ self.assertTrue(lock.locked())
+ self.assertFalse(lock.locked())
-class EventTests(test_utils.TestCase):
- def setUp(self):
- super().setUp()
- self.loop = self.new_test_loop()
+class EventTests(unittest.IsolatedAsyncioTestCase):
def test_repr(self):
ev = asyncio.Event()
@@ -320,7 +306,7 @@ def test_repr(self):
self.assertTrue('waiters:1' in repr(ev))
self.assertTrue(RGX_REPR.match(repr(ev)))
- def test_wait(self):
+ async def test_wait(self):
ev = asyncio.Event()
self.assertFalse(ev.is_set())
@@ -338,16 +324,16 @@ async def c3(result):
if await ev.wait():
result.append(3)
- t1 = self.loop.create_task(c1(result))
- t2 = self.loop.create_task(c2(result))
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
- t3 = self.loop.create_task(c3(result))
+ t3 = asyncio.create_task(c3(result))
ev.set()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([3, 1, 2], result)
self.assertTrue(t1.done())
@@ -357,24 +343,23 @@ async def c3(result):
self.assertTrue(t3.done())
self.assertIsNone(t3.result())
- def test_wait_on_set(self):
+ async def test_wait_on_set(self):
ev = asyncio.Event()
ev.set()
- res = self.loop.run_until_complete(ev.wait())
+ res = await ev.wait()
self.assertTrue(res)
- def test_wait_cancel(self):
+ async def test_wait_cancel(self):
ev = asyncio.Event()
- wait = self.loop.create_task(ev.wait())
- self.loop.call_soon(wait.cancel)
- self.assertRaises(
- asyncio.CancelledError,
- self.loop.run_until_complete, wait)
+ wait = asyncio.create_task(ev.wait())
+ asyncio.get_running_loop().call_soon(wait.cancel)
+ with self.assertRaises(asyncio.CancelledError):
+ await wait
self.assertFalse(ev._waiters)
- def test_clear(self):
+ async def test_clear(self):
ev = asyncio.Event()
self.assertFalse(ev.is_set())
@@ -384,7 +369,7 @@ def test_clear(self):
ev.clear()
self.assertFalse(ev.is_set())
- def test_clear_with_waiters(self):
+ async def test_clear_with_waiters(self):
ev = asyncio.Event()
result = []
@@ -393,8 +378,8 @@ async def c1(result):
result.append(1)
return True
- t = self.loop.create_task(c1(result))
- test_utils.run_briefly(self.loop)
+ t = asyncio.create_task(c1(result))
+ await asyncio.sleep(0)
self.assertEqual([], result)
ev.set()
@@ -405,7 +390,7 @@ async def c1(result):
ev.set()
self.assertEqual(1, len(ev._waiters))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
self.assertEqual(0, len(ev._waiters))
@@ -413,13 +398,9 @@ async def c1(result):
self.assertTrue(t.result())
-class ConditionTests(test_utils.TestCase):
-
- def setUp(self):
- super().setUp()
- self.loop = self.new_test_loop()
+class ConditionTests(unittest.IsolatedAsyncioTestCase):
- def test_wait(self):
+ async def test_wait(self):
cond = asyncio.Condition()
result = []
@@ -441,37 +422,37 @@ async def c3(result):
result.append(3)
return True
- t1 = self.loop.create_task(c1(result))
- t2 = self.loop.create_task(c2(result))
- t3 = self.loop.create_task(c3(result))
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
+ t3 = asyncio.create_task(c3(result))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
self.assertFalse(cond.locked())
- self.assertTrue(self.loop.run_until_complete(cond.acquire()))
+ self.assertTrue(await cond.acquire())
cond.notify()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
self.assertTrue(cond.locked())
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
self.assertTrue(cond.locked())
cond.notify(2)
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
self.assertTrue(cond.locked())
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1, 2], result)
self.assertTrue(cond.locked())
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1, 2, 3], result)
self.assertTrue(cond.locked())
@@ -482,49 +463,47 @@ async def c3(result):
self.assertTrue(t3.done())
self.assertTrue(t3.result())
- def test_wait_cancel(self):
+ async def test_wait_cancel(self):
cond = asyncio.Condition()
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
- wait = self.loop.create_task(cond.wait())
- self.loop.call_soon(wait.cancel)
- self.assertRaises(
- asyncio.CancelledError,
- self.loop.run_until_complete, wait)
+ wait = asyncio.create_task(cond.wait())
+ asyncio.get_running_loop().call_soon(wait.cancel)
+ with self.assertRaises(asyncio.CancelledError):
+ await wait
self.assertFalse(cond._waiters)
self.assertTrue(cond.locked())
- def test_wait_cancel_contested(self):
+ async def test_wait_cancel_contested(self):
cond = asyncio.Condition()
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
self.assertTrue(cond.locked())
- wait_task = self.loop.create_task(cond.wait())
- test_utils.run_briefly(self.loop)
+ wait_task = asyncio.create_task(cond.wait())
+ await asyncio.sleep(0)
self.assertFalse(cond.locked())
# Notify, but contest the lock before cancelling
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
self.assertTrue(cond.locked())
cond.notify()
- self.loop.call_soon(wait_task.cancel)
- self.loop.call_soon(cond.release)
+ asyncio.get_running_loop().call_soon(wait_task.cancel)
+ asyncio.get_running_loop().call_soon(cond.release)
try:
- self.loop.run_until_complete(wait_task)
+ await wait_task
except asyncio.CancelledError:
# Should not happen, since no cancellation points
pass
self.assertTrue(cond.locked())
- def test_wait_cancel_after_notify(self):
+ async def test_wait_cancel_after_notify(self):
# See bpo-32841
waited = False
cond = asyncio.Condition()
- cond._loop = self.loop
async def wait_on_cond():
nonlocal waited
@@ -532,27 +511,26 @@ async def wait_on_cond():
waited = True # Make sure this area was reached
await cond.wait()
- waiter = asyncio.ensure_future(wait_on_cond(), loop=self.loop)
- test_utils.run_briefly(self.loop) # Start waiting
+ waiter = asyncio.create_task(wait_on_cond())
+ await asyncio.sleep(0) # Start waiting
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
cond.notify()
- test_utils.run_briefly(self.loop) # Get to acquire()
+ await asyncio.sleep(0) # Get to acquire()
waiter.cancel()
- test_utils.run_briefly(self.loop) # Activate cancellation
+ await asyncio.sleep(0) # Activate cancellation
cond.release()
- test_utils.run_briefly(self.loop) # Cancellation should occur
+ await asyncio.sleep(0) # Cancellation should occur
self.assertTrue(waiter.cancelled())
self.assertTrue(waited)
- def test_wait_unacquired(self):
+ async def test_wait_unacquired(self):
cond = asyncio.Condition()
- self.assertRaises(
- RuntimeError,
- self.loop.run_until_complete, cond.wait())
+ with self.assertRaises(RuntimeError):
+ await cond.wait()
- def test_wait_for(self):
+ async def test_wait_for(self):
cond = asyncio.Condition()
presult = False
@@ -568,40 +546,38 @@ async def c1(result):
cond.release()
return True
- t = self.loop.create_task(c1(result))
+ t = asyncio.create_task(c1(result))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
cond.notify()
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
presult = True
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
cond.notify()
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
self.assertTrue(t.done())
self.assertTrue(t.result())
- def test_wait_for_unacquired(self):
+ async def test_wait_for_unacquired(self):
cond = asyncio.Condition()
# predicate can return true immediately
- res = self.loop.run_until_complete(cond.wait_for(lambda: [1, 2, 3]))
+ res = await cond.wait_for(lambda: [1, 2, 3])
self.assertEqual([1, 2, 3], res)
- self.assertRaises(
- RuntimeError,
- self.loop.run_until_complete,
- cond.wait_for(lambda: False))
+ with self.assertRaises(RuntimeError):
+ await cond.wait_for(lambda: False)
- def test_notify(self):
+ async def test_notify(self):
cond = asyncio.Condition()
result = []
@@ -626,24 +602,24 @@ async def c3(result):
cond.release()
return True
- t1 = self.loop.create_task(c1(result))
- t2 = self.loop.create_task(c2(result))
- t3 = self.loop.create_task(c3(result))
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
+ t3 = asyncio.create_task(c3(result))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
cond.notify(1)
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
cond.notify(1)
cond.notify(2048)
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1, 2, 3], result)
self.assertTrue(t1.done())
@@ -653,7 +629,7 @@ async def c3(result):
self.assertTrue(t3.done())
self.assertTrue(t3.result())
- def test_notify_all(self):
+ async def test_notify_all(self):
cond = asyncio.Condition()
result = []
@@ -672,16 +648,16 @@ async def c2(result):
cond.release()
return True
- t1 = self.loop.create_task(c1(result))
- t2 = self.loop.create_task(c2(result))
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([], result)
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
cond.notify_all()
cond.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1, 2], result)
self.assertTrue(t1.done())
@@ -697,12 +673,12 @@ def test_notify_all_unacquired(self):
cond = asyncio.Condition()
self.assertRaises(RuntimeError, cond.notify_all)
- def test_repr(self):
+ async def test_repr(self):
cond = asyncio.Condition()
self.assertTrue('unlocked' in repr(cond))
self.assertTrue(RGX_REPR.match(repr(cond)))
- self.loop.run_until_complete(cond.acquire())
+ await cond.acquire()
self.assertTrue('locked' in repr(cond))
cond._waiters.append(mock.Mock())
@@ -713,69 +689,103 @@ def test_repr(self):
self.assertTrue('waiters:2' in repr(cond))
self.assertTrue(RGX_REPR.match(repr(cond)))
- def test_context_manager(self):
- async def f():
- cond = asyncio.Condition()
+ async def test_context_manager(self):
+ cond = asyncio.Condition()
+ self.assertFalse(cond.locked())
+ async with cond:
+ self.assertTrue(cond.locked())
+ self.assertFalse(cond.locked())
+
+ async def test_explicit_lock(self):
+ async def f(lock=None, cond=None):
+ if lock is None:
+ lock = asyncio.Lock()
+ if cond is None:
+ cond = asyncio.Condition(lock)
+ self.assertIs(cond._lock, lock)
+ self.assertFalse(lock.locked())
self.assertFalse(cond.locked())
async with cond:
+ self.assertTrue(lock.locked())
self.assertTrue(cond.locked())
+ self.assertFalse(lock.locked())
+ self.assertFalse(cond.locked())
+ async with lock:
+ self.assertTrue(lock.locked())
+ self.assertTrue(cond.locked())
+ self.assertFalse(lock.locked())
self.assertFalse(cond.locked())
- self.loop.run_until_complete(f())
-
- def test_explicit_lock(self):
- lock = asyncio.Lock()
- cond = asyncio.Condition(lock)
-
- self.assertIs(cond._lock, lock)
- self.assertIs(cond._loop, lock._loop)
-
- def test_ambiguous_loops(self):
- loop = self.new_test_loop()
- self.addCleanup(loop.close)
-
+ # All should work in the same way.
+ await f()
+ await f(asyncio.Lock())
lock = asyncio.Lock()
- lock._loop = loop
+ await f(lock, asyncio.Condition(lock))
- async def _create_condition():
- with self.assertRaises(ValueError):
- asyncio.Condition(lock)
-
- self.loop.run_until_complete(_create_condition())
-
- def test_timeout_in_block(self):
+ async def test_ambiguous_loops(self):
loop = asyncio.new_event_loop()
self.addCleanup(loop.close)
- async def task_timeout():
- condition = asyncio.Condition()
- async with condition:
- with self.assertRaises(asyncio.TimeoutError):
- await asyncio.wait_for(condition.wait(), timeout=0.5)
+ async def wrong_loop_in_lock():
+ with self.assertRaises(TypeError):
+ asyncio.Lock(loop=loop) # actively disallowed since 3.10
+ lock = asyncio.Lock()
+ lock._loop = loop # use private API for testing
+ async with lock:
+ # acquired immediately via the fast-path
+ # without interaction with any event loop.
+ cond = asyncio.Condition(lock)
+ # cond.acquire() will trigger waiting on the lock
+ # and it will discover the event loop mismatch.
+ with self.assertRaisesRegex(
+ RuntimeError,
+ "is bound to a different event loop",
+ ):
+ await cond.acquire()
- loop.run_until_complete(task_timeout())
+ async def wrong_loop_in_cond():
+ # Same analogy here with the condition's loop.
+ lock = asyncio.Lock()
+ async with lock:
+ with self.assertRaises(TypeError):
+ asyncio.Condition(lock, loop=loop)
+ cond = asyncio.Condition(lock)
+ cond._loop = loop
+ with self.assertRaisesRegex(
+ RuntimeError,
+ "is bound to a different event loop",
+ ):
+ await cond.wait()
+ await wrong_loop_in_lock()
+ await wrong_loop_in_cond()
-class SemaphoreTests(test_utils.TestCase):
+ async def test_timeout_in_block(self):
+ condition = asyncio.Condition()
+ async with condition:
+ with self.assertRaises(asyncio.TimeoutError):
+ await asyncio.wait_for(condition.wait(), timeout=0.5)
- def setUp(self):
- super().setUp()
- self.loop = self.new_test_loop()
+
+class SemaphoreTests(unittest.IsolatedAsyncioTestCase):
def test_initial_value_zero(self):
sem = asyncio.Semaphore(0)
self.assertTrue(sem.locked())
- def test_repr(self):
+ async def test_repr(self):
sem = asyncio.Semaphore()
self.assertTrue(repr(sem).endswith('[unlocked, value:1]>'))
self.assertTrue(RGX_REPR.match(repr(sem)))
- self.loop.run_until_complete(sem.acquire())
+ await sem.acquire()
self.assertTrue(repr(sem).endswith('[locked]>'))
self.assertTrue('waiters' not in repr(sem))
self.assertTrue(RGX_REPR.match(repr(sem)))
+ if sem._waiters is None:
+ sem._waiters = collections.deque()
+
sem._waiters.append(mock.Mock())
self.assertTrue('waiters:1' in repr(sem))
self.assertTrue(RGX_REPR.match(repr(sem)))
@@ -784,7 +794,7 @@ def test_repr(self):
self.assertTrue('waiters:2' in repr(sem))
self.assertTrue(RGX_REPR.match(repr(sem)))
- def test_semaphore(self):
+ async def test_semaphore(self):
sem = asyncio.Semaphore()
self.assertEqual(1, sem._value)
@@ -797,7 +807,7 @@ def acquire_lock():
TypeError,
"'Semaphore' object is not iterable",
):
- self.loop.run_until_complete(acquire_lock())
+ await acquire_lock()
self.assertFalse(sem.locked())
self.assertEqual(1, sem._value)
@@ -805,12 +815,12 @@ def acquire_lock():
def test_semaphore_value(self):
self.assertRaises(ValueError, asyncio.Semaphore, -1)
- def test_acquire(self):
+ async def test_acquire(self):
sem = asyncio.Semaphore(3)
result = []
- self.assertTrue(self.loop.run_until_complete(sem.acquire()))
- self.assertTrue(self.loop.run_until_complete(sem.acquire()))
+ self.assertTrue(await sem.acquire())
+ self.assertTrue(await sem.acquire())
self.assertFalse(sem.locked())
async def c1(result):
@@ -833,23 +843,23 @@ async def c4(result):
result.append(4)
return True
- t1 = self.loop.create_task(c1(result))
- t2 = self.loop.create_task(c2(result))
- t3 = self.loop.create_task(c3(result))
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
+ t3 = asyncio.create_task(c3(result))
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual([1], result)
self.assertTrue(sem.locked())
self.assertEqual(2, len(sem._waiters))
self.assertEqual(0, sem._value)
- t4 = self.loop.create_task(c4(result))
+ t4 = asyncio.create_task(c4(result))
sem.release()
sem.release()
- self.assertEqual(2, sem._value)
+ self.assertEqual(0, sem._value)
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
self.assertEqual(0, sem._value)
self.assertEqual(3, len(result))
self.assertTrue(sem.locked())
@@ -860,73 +870,194 @@ async def c4(result):
self.assertTrue(t1.result())
race_tasks = [t2, t3, t4]
done_tasks = [t for t in race_tasks if t.done() and t.result()]
- self.assertTrue(2, len(done_tasks))
+ self.assertEqual(2, len(done_tasks))
# cleanup locked semaphore
sem.release()
- self.loop.run_until_complete(asyncio.gather(*race_tasks))
+ await asyncio.gather(*race_tasks)
- def test_acquire_cancel(self):
+ async def test_acquire_cancel(self):
sem = asyncio.Semaphore()
- self.loop.run_until_complete(sem.acquire())
+ await sem.acquire()
- acquire = self.loop.create_task(sem.acquire())
- self.loop.call_soon(acquire.cancel)
- self.assertRaises(
- asyncio.CancelledError,
- self.loop.run_until_complete, acquire)
+ acquire = asyncio.create_task(sem.acquire())
+ asyncio.get_running_loop().call_soon(acquire.cancel)
+ with self.assertRaises(asyncio.CancelledError):
+ await acquire
self.assertTrue((not sem._waiters) or
all(waiter.done() for waiter in sem._waiters))
- def test_acquire_cancel_before_awoken(self):
+ async def test_acquire_cancel_before_awoken(self):
sem = asyncio.Semaphore(value=0)
- t1 = self.loop.create_task(sem.acquire())
- t2 = self.loop.create_task(sem.acquire())
- t3 = self.loop.create_task(sem.acquire())
- t4 = self.loop.create_task(sem.acquire())
+ t1 = asyncio.create_task(sem.acquire())
+ t2 = asyncio.create_task(sem.acquire())
+ t3 = asyncio.create_task(sem.acquire())
+ t4 = asyncio.create_task(sem.acquire())
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
- sem.release()
t1.cancel()
t2.cancel()
+ sem.release()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
+ await asyncio.sleep(0)
num_done = sum(t.done() for t in [t3, t4])
self.assertEqual(num_done, 1)
+ self.assertTrue(t3.done())
+ self.assertFalse(t4.done())
t3.cancel()
t4.cancel()
- test_utils.run_briefly(self.loop)
+ await asyncio.sleep(0)
- def test_acquire_hang(self):
+ async def test_acquire_hang(self):
sem = asyncio.Semaphore(value=0)
- t1 = self.loop.create_task(sem.acquire())
- t2 = self.loop.create_task(sem.acquire())
-
- test_utils.run_briefly(self.loop)
+ t1 = asyncio.create_task(sem.acquire())
+ t2 = asyncio.create_task(sem.acquire())
+ await asyncio.sleep(0)
- sem.release()
t1.cancel()
-
- test_utils.run_briefly(self.loop)
+ sem.release()
+ await asyncio.sleep(0)
+ await asyncio.sleep(0)
self.assertTrue(sem.locked())
+ self.assertTrue(t2.done())
+
+ async def test_acquire_no_hang(self):
+
+ sem = asyncio.Semaphore(1)
+
+ async def c1():
+ async with sem:
+ await asyncio.sleep(0)
+ t2.cancel()
+
+ async def c2():
+ async with sem:
+ self.assertFalse(True)
+
+ t1 = asyncio.create_task(c1())
+ t2 = asyncio.create_task(c2())
+
+ r1, r2 = await asyncio.gather(t1, t2, return_exceptions=True)
+ self.assertTrue(r1 is None)
+ self.assertTrue(isinstance(r2, asyncio.CancelledError))
+
+ await asyncio.wait_for(sem.acquire(), timeout=1.0)
def test_release_not_acquired(self):
sem = asyncio.BoundedSemaphore()
self.assertRaises(ValueError, sem.release)
- def test_release_no_waiters(self):
+ async def test_release_no_waiters(self):
sem = asyncio.Semaphore()
- self.loop.run_until_complete(sem.acquire())
+ await sem.acquire()
self.assertTrue(sem.locked())
sem.release()
self.assertFalse(sem.locked())
+ async def test_acquire_fifo_order(self):
+ sem = asyncio.Semaphore(1)
+ result = []
+
+ async def coro(tag):
+ await sem.acquire()
+ result.append(f'{tag}_1')
+ await asyncio.sleep(0.01)
+ sem.release()
+
+ await sem.acquire()
+ result.append(f'{tag}_2')
+ await asyncio.sleep(0.01)
+ sem.release()
+
+ t1 = asyncio.create_task(coro('c1'))
+ t2 = asyncio.create_task(coro('c2'))
+ t3 = asyncio.create_task(coro('c3'))
+
+ await asyncio.gather(t1, t2, t3)
+
+ self.assertEqual(
+ ['c1_1', 'c2_1', 'c3_1', 'c1_2', 'c2_2', 'c3_2'],
+ result
+ )
+
+ async def test_acquire_fifo_order_2(self):
+ sem = asyncio.Semaphore(1)
+ result = []
+
+ async def c1(result):
+ await sem.acquire()
+ result.append(1)
+ return True
+
+ async def c2(result):
+ await sem.acquire()
+ result.append(2)
+ sem.release()
+ await sem.acquire()
+ result.append(4)
+ return True
+
+ async def c3(result):
+ await sem.acquire()
+ result.append(3)
+ return True
+
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
+ t3 = asyncio.create_task(c3(result))
+
+ await asyncio.sleep(0)
+
+ sem.release()
+ sem.release()
+
+ tasks = [t1, t2, t3]
+ await asyncio.gather(*tasks)
+ self.assertEqual([1, 2, 3, 4], result)
+
+ async def test_acquire_fifo_order_3(self):
+ sem = asyncio.Semaphore(0)
+ result = []
+
+ async def c1(result):
+ await sem.acquire()
+ result.append(1)
+ return True
+
+ async def c2(result):
+ await sem.acquire()
+ result.append(2)
+ return True
+
+ async def c3(result):
+ await sem.acquire()
+ result.append(3)
+ return True
+
+ t1 = asyncio.create_task(c1(result))
+ t2 = asyncio.create_task(c2(result))
+ t3 = asyncio.create_task(c3(result))
+
+ await asyncio.sleep(0)
+
+ t1.cancel()
+
+ await asyncio.sleep(0)
+
+ sem.release()
+ sem.release()
+
+ tasks = [t1, t2, t3]
+ await asyncio.gather(*tasks, return_exceptions=True)
+ self.assertEqual([2, 3], result)
+
if __name__ == '__main__':
unittest.main()
diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py
index 4c8906d531ce5c..60b30c6a46a65e 100644
--- a/Lib/test/test_asyncio/test_proactor_events.py
+++ b/Lib/test/test_asyncio/test_proactor_events.py
@@ -241,6 +241,14 @@ def test_close_buffer(self):
test_utils.run_briefly(self.loop)
self.assertFalse(self.protocol.connection_lost.called)
+ def test_close_invalid_sockobj(self):
+ tr = self.socket_transport()
+ self.sock.fileno.return_value = -1
+ tr.close()
+ test_utils.run_briefly(self.loop)
+ self.protocol.connection_lost.assert_called_with(None)
+ self.assertFalse(self.sock.shutdown.called)
+
@mock.patch('asyncio.base_events.logger')
def test_fatal_error(self, m_logging):
tr = self.socket_transport()
@@ -281,7 +289,33 @@ def test_force_close_idempotent(self):
tr._closing = True
tr._force_close(None)
test_utils.run_briefly(self.loop)
+ # See https://github.com/python/cpython/issues/89237
+ # `protocol.connection_lost` should be called even if
+ # the transport was closed forcefully otherwise
+ # the resources held by protocol will never be freed
+ # and waiters will never be notified leading to hang.
+ self.assertTrue(self.protocol.connection_lost.called)
+
+ def test_force_close_protocol_connection_lost_once(self):
+ tr = self.socket_transport()
self.assertFalse(self.protocol.connection_lost.called)
+ tr._closing = True
+ # Calling _force_close twice should not call
+ # protocol.connection_lost twice
+ tr._force_close(None)
+ tr._force_close(None)
+ test_utils.run_briefly(self.loop)
+ self.assertEqual(1, self.protocol.connection_lost.call_count)
+
+ def test_close_protocol_connection_lost_once(self):
+ tr = self.socket_transport()
+ self.assertFalse(self.protocol.connection_lost.called)
+ # Calling close twice should not call
+ # protocol.connection_lost twice
+ tr.close()
+ tr.close()
+ test_utils.run_briefly(self.loop)
+ self.assertEqual(1, self.protocol.connection_lost.call_count)
def test_fatal_error_2(self):
tr = self.socket_transport()
@@ -838,6 +872,7 @@ def datagram_transport(self):
def test_make_datagram_transport(self):
tr = self.datagram_transport()
self.assertIsInstance(tr, _ProactorDatagramTransport)
+ self.assertIsInstance(tr, asyncio.DatagramTransport)
close_transport(tr)
def test_datagram_loop_writing(self):
diff --git a/Lib/test/test_asyncio/test_protocols.py b/Lib/test/test_asyncio/test_protocols.py
index 438111cccd3478..0f232631867db5 100644
--- a/Lib/test/test_asyncio/test_protocols.py
+++ b/Lib/test/test_asyncio/test_protocols.py
@@ -4,6 +4,12 @@
import asyncio
+def tearDownModule():
+ # not needed for the test file but added for uniformness with all other
+ # asyncio test files for the sake of unified cleanup
+ asyncio.set_event_loop_policy(None)
+
+
class ProtocolsAbsTests(unittest.TestCase):
def test_base_protocol(self):
@@ -55,3 +61,7 @@ def test_subprocess_protocol(self):
self.assertIsNone(sp.pipe_connection_lost(1, f))
self.assertIsNone(sp.process_exited())
self.assertFalse(hasattr(sp, '__dict__'))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/Lib/test/test_asyncio/test_queues.py b/Lib/test/test_asyncio/test_queues.py
index 63a9a5f270cc92..b1a53b859c5ccb 100644
--- a/Lib/test/test_asyncio/test_queues.py
+++ b/Lib/test/test_asyncio/test_queues.py
@@ -1,9 +1,8 @@
"""Tests for queues.py"""
import unittest
-from unittest import mock
-
import asyncio
+from types import GenericAlias
from test.test_asyncio import utils as test_utils
@@ -74,6 +73,11 @@ def test_repr(self):
def test_str(self):
self._test_repr_or_str(str, False)
+ def test_generic_alias(self):
+ q = asyncio.Queue[int]
+ self.assertEqual(q.__args__, (int,))
+ self.assertIsInstance(q, GenericAlias)
+
def test_empty(self):
q = asyncio.Queue()
self.assertTrue(q.empty())
diff --git a/Lib/test/test_asyncio/test_runners.py b/Lib/test/test_asyncio/test_runners.py
index b9ae02dc3c04e0..112273662b20b5 100644
--- a/Lib/test/test_asyncio/test_runners.py
+++ b/Lib/test/test_asyncio/test_runners.py
@@ -2,7 +2,11 @@
import unittest
from unittest import mock
-from . import utils as test_utils
+from test.test_asyncio import utils as test_utils
+
+
+def tearDownModule():
+ asyncio.set_event_loop_policy(None)
class TestPolicy(asyncio.AbstractEventLoopPolicy):
@@ -180,3 +184,7 @@ async def main():
self.assertIsNone(spinner.ag_frame)
self.assertFalse(spinner.ag_running)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py
index 1613c753c26ee5..ac221da4432f1f 100644
--- a/Lib/test/test_asyncio/test_selector_events.py
+++ b/Lib/test/test_asyncio/test_selector_events.py
@@ -188,6 +188,24 @@ def test_write_to_self_exception(self):
self.loop._csock.send.side_effect = RuntimeError()
self.assertRaises(RuntimeError, self.loop._write_to_self)
+ @mock.patch('socket.getaddrinfo')
+ def test_sock_connect_resolve_using_socket_params(self, m_gai):
+ addr = ('need-resolution.com', 8080)
+ for sock_type in [socket.SOCK_STREAM, socket.SOCK_DGRAM]:
+ with self.subTest(sock_type):
+ sock = test_utils.mock_nonblocking_socket(type=sock_type)
+
+ m_gai.side_effect = \
+ lambda *args: [(None, None, None, None, ('127.0.0.1', 0))]
+
+ con = self.loop.create_task(self.loop.sock_connect(sock, addr))
+ self.loop.run_until_complete(con)
+ m_gai.assert_called_with(
+ addr[0], addr[1], sock.family, sock.type, sock.proto, 0)
+
+ self.loop.run_until_complete(con)
+ sock.connect.assert_called_with(('127.0.0.1', 0))
+
def test_add_reader(self):
self.loop._selector.get_key.side_effect = KeyError
cb = lambda: True
diff --git a/Lib/test/test_asyncio/test_sendfile.py b/Lib/test/test_asyncio/test_sendfile.py
index 01c698653ec67e..aaaad9df8eb8ad 100644
--- a/Lib/test/test_asyncio/test_sendfile.py
+++ b/Lib/test/test_asyncio/test_sendfile.py
@@ -1,6 +1,7 @@
"""Tests for sendfile functionality."""
import asyncio
+import errno
import os
import socket
import sys
@@ -36,25 +37,29 @@ def __init__(self, loop=None, close_after=0):
self.data = bytearray()
self.close_after = close_after
+ def _assert_state(self, *expected):
+ if self.state not in expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'CONNECTED'
if self.connected:
self.connected.set_result(None)
def eof_received(self):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.state = 'EOF'
def connection_lost(self, exc):
- assert self.state in ('CONNECTED', 'EOF'), self.state
+ self._assert_state('CONNECTED', 'EOF')
self.state = 'CLOSED'
if self.done:
self.done.set_result(None)
def data_received(self, data):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.nbytes += len(data)
self.data.extend(data)
super().data_received(data)
@@ -88,9 +93,13 @@ async def wait_closed(self):
class SendfileBase:
- # 128 KiB plus small unaligned to buffer chunk
- DATA = b"SendfileBaseData" * (1024 * 8 + 1)
-
+ # 256 KiB plus small unaligned to buffer chunk
+ # Newer versions of Windows seems to have increased its internal
+ # buffer and tries to send as much of the data as it can as it
+ # has some form of buffering for this which is less than 256KiB
+ # on newer server versions and Windows 11.
+ # So DATA should be larger than 256 KiB to make this test reliable.
+ DATA = b"x" * (1024 * 256 + 1)
# Reduce socket buffer size to test on relative small data sets.
BUF_SIZE = 4 * 1024 # 4 KiB
@@ -476,8 +485,17 @@ def sendfile_native(transp, file, offset, count):
srv_proto, cli_proto = self.prepare_sendfile(close_after=1024)
with self.assertRaises(ConnectionError):
- self.run_loop(
- self.loop.sendfile(cli_proto.transport, self.file))
+ try:
+ self.run_loop(
+ self.loop.sendfile(cli_proto.transport, self.file))
+ except OSError as e:
+ # macOS may raise OSError of EPROTOTYPE when writing to a
+ # socket that is in the process of closing down.
+ if e.errno == errno.EPROTOTYPE and sys.platform == "darwin":
+ raise ConnectionError
+ else:
+ raise
+
self.run_loop(srv_proto.done)
self.assertTrue(1024 <= srv_proto.nbytes < len(self.DATA),
@@ -561,3 +579,7 @@ class SelectEventLoopTests(SendfileTestsBase,
def create_event_loop(self):
return asyncio.SelectorEventLoop(selectors.SelectSelector())
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/Lib/test/test_asyncio/test_sock_lowlevel.py b/Lib/test/test_asyncio/test_sock_lowlevel.py
index d8a5df8ede1f84..14001a4a5001f8 100644
--- a/Lib/test/test_asyncio/test_sock_lowlevel.py
+++ b/Lib/test/test_asyncio/test_sock_lowlevel.py
@@ -1,5 +1,4 @@
import socket
-import time
import asyncio
import sys
import unittest
@@ -11,6 +10,10 @@
from test.support import socket_helper
+def tearDownModule():
+ asyncio.set_event_loop_policy(None)
+
+
class MyProto(asyncio.Protocol):
connected = None
done = None
@@ -23,24 +26,28 @@ def __init__(self, loop=None):
self.connected = loop.create_future()
self.done = loop.create_future()
+ def _assert_state(self, *expected):
+ if self.state not in expected:
+ raise AssertionError(f'state: {self.state!r}, expected: {expected!r}')
+
def connection_made(self, transport):
self.transport = transport
- assert self.state == 'INITIAL', self.state
+ self._assert_state('INITIAL')
self.state = 'CONNECTED'
if self.connected:
self.connected.set_result(None)
transport.write(b'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n')
def data_received(self, data):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.nbytes += len(data)
def eof_received(self):
- assert self.state == 'CONNECTED', self.state
+ self._assert_state('CONNECTED')
self.state = 'EOF'
def connection_lost(self, exc):
- assert self.state in ('CONNECTED', 'EOF'), self.state
+ self._assert_state('CONNECTED', 'EOF')
self.state = 'CLOSED'
if self.done:
self.done.set_result(None)
@@ -451,7 +458,7 @@ def test_create_connection_sock(self):
else:
break
else:
- assert False, 'Can not create socket.'
+ self.fail('Can not create socket.')
f = self.loop.create_connection(
lambda: MyProto(loop=self.loop), sock=sock)
@@ -508,3 +515,7 @@ class SelectEventLoopTests(BaseSockTestsMixin,
def create_event_loop(self):
return asyncio.SelectorEventLoop(selectors.SelectSelector())
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py
index e87863eb712373..f7411a8142cc87 100644
--- a/Lib/test/test_asyncio/test_sslproto.py
+++ b/Lib/test/test_asyncio/test_sslproto.py
@@ -2,7 +2,6 @@
import logging
import socket
-from test import support
import unittest
import weakref
from unittest import mock
@@ -278,6 +277,7 @@ async def client(addr):
# No garbage is left if SSL is closed uncleanly
client_context = weakref.ref(client_context)
+ support.gc_collect()
self.assertIsNone(client_context())
def test_create_connection_memory_leak(self):
@@ -341,6 +341,7 @@ async def client(addr):
# No garbage is left for SSL client from loop.create_connection, even
# if user stores the SSLTransport in corresponding protocol instance
client_context = weakref.ref(client_context)
+ support.gc_collect()
self.assertIsNone(client_context())
def test_start_tls_client_buf_proto_1(self):
@@ -381,9 +382,9 @@ def connection_made(self, tr):
def get_buffer(self, sizehint):
return self.buf
- def buffer_updated(self, nsize):
- assert nsize == 1
- self.on_data.set_result(bytes(self.buf[:nsize]))
+ def buffer_updated(slf, nsize):
+ self.assertEqual(nsize, 1)
+ slf.on_data.set_result(bytes(slf.buf[:nsize]))
class ClientProtoSecond(asyncio.Protocol):
def __init__(self, on_data, on_eof):
@@ -640,6 +641,7 @@ async def client(addr):
# The 10s handshake timeout should be cancelled to free related
# objects without really waiting for 10s
client_sslctx = weakref.ref(client_sslctx)
+ support.gc_collect()
self.assertIsNone(client_sslctx())
def test_create_connection_ssl_slow_handshake(self):
diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
index 6eaa2899442184..994041c10f0a95 100644
--- a/Lib/test/test_asyncio/test_streams.py
+++ b/Lib/test/test_asyncio/test_streams.py
@@ -711,7 +711,7 @@ def test_read_all_from_pipe_reader(self):
# See asyncio issue 168. This test is derived from the example
# subprocess_attach_read_pipe.py, but we configure the
# StreamReader's limit so that twice it is less than the size
- # of the data writter. Also we must explicitly attach a child
+ # of the data writer. Also we must explicitly attach a child
# watcher to the event loop.
code = """\
@@ -747,10 +747,8 @@ def test_read_all_from_pipe_reader(self):
self.assertEqual(data, b'data')
def test_streamreader_constructor_without_loop(self):
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
- asyncio.StreamReader()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ asyncio.StreamReader()
def test_streamreader_constructor_use_running_loop(self):
# asyncio issue #184: Ensure that StreamReaderProtocol constructor
@@ -764,21 +762,17 @@ async def test():
def test_streamreader_constructor_use_global_loop(self):
# asyncio issue #184: Ensure that StreamReaderProtocol constructor
# retrieves the current loop if the loop parameter is not set
- # Deprecated in 3.10
+ # Deprecated in 3.10, undeprecated in 3.11.1
self.addCleanup(asyncio.set_event_loop, None)
asyncio.set_event_loop(self.loop)
- with self.assertWarns(DeprecationWarning) as cm:
- reader = asyncio.StreamReader()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ reader = asyncio.StreamReader()
self.assertIs(reader._loop, self.loop)
def test_streamreaderprotocol_constructor_without_loop(self):
reader = mock.Mock()
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
- asyncio.StreamReaderProtocol(reader)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ asyncio.StreamReaderProtocol(reader)
def test_streamreaderprotocol_constructor_use_running_loop(self):
# asyncio issue #184: Ensure that StreamReaderProtocol constructor
@@ -792,15 +786,32 @@ async def test():
def test_streamreaderprotocol_constructor_use_global_loop(self):
# asyncio issue #184: Ensure that StreamReaderProtocol constructor
# retrieves the current loop if the loop parameter is not set
- # Deprecated in 3.10
+ # Deprecated in 3.10, undeprecated in 3.11.1
self.addCleanup(asyncio.set_event_loop, None)
asyncio.set_event_loop(self.loop)
reader = mock.Mock()
- with self.assertWarns(DeprecationWarning) as cm:
- protocol = asyncio.StreamReaderProtocol(reader)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ protocol = asyncio.StreamReaderProtocol(reader)
self.assertIs(protocol._loop, self.loop)
+ def test_multiple_drain(self):
+ # See https://github.com/python/cpython/issues/74116
+ drained = 0
+
+ async def drainer(stream):
+ nonlocal drained
+ await stream._drain_helper()
+ drained += 1
+
+ async def main():
+ loop = asyncio.get_running_loop()
+ stream = asyncio.streams.FlowControlMixin(loop)
+ stream.pause_writing()
+ loop.call_later(0.1, stream.resume_writing)
+ await asyncio.gather(*[drainer(stream) for _ in range(10)])
+ self.assertEqual(drained, 10)
+
+ self.loop.run_until_complete(main())
+
def test_drain_raises(self):
# See http://bugs.python.org/issue25441
@@ -987,10 +998,10 @@ def test_eof_feed_when_closing_writer(self):
wr.close()
f = wr.wait_closed()
self.loop.run_until_complete(f)
- assert rd.at_eof()
+ self.assertTrue(rd.at_eof())
f = rd.read()
data = self.loop.run_until_complete(f)
- assert data == b''
+ self.assertEqual(data, b'')
self.assertEqual(messages, [])
diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py
index 225a3babc844b8..7cd80fd68e3ba4 100644
--- a/Lib/test/test_asyncio/test_subprocess.py
+++ b/Lib/test/test_asyncio/test_subprocess.py
@@ -228,7 +228,7 @@ def prepare_broken_pipe_test(self):
# buffer large enough to feed the whole pipe buffer
large_data = b'x' * support.PIPE_MAX_SIZE
- # the program ends before the stdin can be feeded
+ # the program ends before the stdin can be fed
proc = self.loop.run_until_complete(
asyncio.create_subprocess_exec(
sys.executable, '-c', 'pass',
@@ -401,6 +401,26 @@ async def empty_error():
self.assertEqual(output, None)
self.assertEqual(exitcode, 0)
+ @unittest.skipIf(sys.platform != 'linux', "Don't have /dev/stdin")
+ def test_devstdin_input(self):
+
+ async def devstdin_input(message):
+ code = 'file = open("/dev/stdin"); data = file.read(); print(len(data))'
+ proc = await asyncio.create_subprocess_exec(
+ sys.executable, '-c', code,
+ stdin=asyncio.subprocess.PIPE,
+ stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.PIPE,
+ close_fds=False,
+ )
+ stdout, stderr = await proc.communicate(message)
+ exitcode = await proc.wait()
+ return (stdout, exitcode)
+
+ output, exitcode = self.loop.run_until_complete(devstdin_input(b'abc'))
+ self.assertEqual(output.rstrip(), b'3')
+ self.assertEqual(exitcode, 0)
+
def test_cancel_process_wait(self):
# Issue #23140: cancel Process.wait()
@@ -665,6 +685,8 @@ class SubprocessThreadedWatcherTests(SubprocessWatcherMixin,
Watcher = unix_events.ThreadedChildWatcher
+ @unittest.skip("bpo-38323: MultiLoopChildWatcher has a race condition \
+ and these tests can hang the test suite")
class SubprocessMultiLoopWatcherTests(SubprocessWatcherMixin,
test_utils.TestCase):
diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py
index a9e4cf53566ca9..b08068e6173e8f 100644
--- a/Lib/test/test_asyncio/test_tasks.py
+++ b/Lib/test/test_asyncio/test_tasks.py
@@ -15,6 +15,7 @@
import unittest
import weakref
from unittest import mock
+from types import GenericAlias
import asyncio
from asyncio import coroutines
@@ -97,12 +98,6 @@ def __await__(self):
return self
-# The following value can be used as a very small timeout:
-# it passes check "timeout > 0", but has almost
-# no effect on the test performance
-_EPSILON = 0.0001
-
-
class BaseTaskTests:
Task = None
@@ -120,6 +115,11 @@ def setUp(self):
self.loop.set_task_factory(self.new_task)
self.loop.create_future = lambda: self.new_future(self.loop)
+ def test_generic_alias(self):
+ task = self.__class__.Task[str]
+ self.assertEqual(task.__args__, (str,))
+ self.assertIsInstance(task, GenericAlias)
+
def test_task_cancel_message_getter(self):
async def coro():
pass
@@ -210,10 +210,8 @@ async def notmuch():
a = notmuch()
self.addCleanup(a.close)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
- asyncio.ensure_future(a)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ asyncio.ensure_future(a)
async def test():
return asyncio.ensure_future(notmuch())
@@ -223,12 +221,10 @@ async def test():
self.assertTrue(t.done())
self.assertEqual(t.result(), 'ok')
- # Deprecated in 3.10
+ # Deprecated in 3.10.0, undeprecated in 3.10.9
asyncio.set_event_loop(self.loop)
self.addCleanup(asyncio.set_event_loop, None)
- with self.assertWarns(DeprecationWarning) as cm:
- t = asyncio.ensure_future(notmuch())
- self.assertEqual(cm.warnings[0].filename, __file__)
+ t = asyncio.ensure_future(notmuch())
self.assertIs(t._loop, self.loop)
self.loop.run_until_complete(t)
self.assertTrue(t.done())
@@ -247,10 +243,8 @@ def notmuch():
a = notmuch()
self.addCleanup(a.close)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
- asyncio.ensure_future(a)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
+ asyncio.ensure_future(a)
async def test():
return asyncio.ensure_future(notmuch())
@@ -260,12 +254,10 @@ async def test():
self.assertTrue(t.done())
self.assertEqual(t.result(), 'ok')
- # Deprecated in 3.10
+ # Deprecated in 3.10.0, undeprecated in 3.10.9
asyncio.set_event_loop(self.loop)
self.addCleanup(asyncio.set_event_loop, None)
- with self.assertWarns(DeprecationWarning) as cm:
- t = asyncio.ensure_future(notmuch())
- self.assertEqual(cm.warnings[0].filename, __file__)
+ t = asyncio.ensure_future(notmuch())
self.assertIs(t._loop, self.loop)
self.loop.run_until_complete(t)
self.assertTrue(t.done())
@@ -329,7 +321,7 @@ def coro():
self.set_event_loop(loop)
fut = asyncio.ensure_future(Aw(coro()), loop=loop)
loop.run_until_complete(fut)
- assert fut.result() == 'ok'
+ self.assertEqual(fut.result(), 'ok')
def test_ensure_future_neither(self):
with self.assertRaises(TypeError):
@@ -1024,273 +1016,6 @@ async def coro():
task._log_traceback = True
self.loop.run_until_complete(task)
- def test_wait_for_timeout_less_then_0_or_0_future_done(self):
- def gen():
- when = yield
- self.assertAlmostEqual(0, when)
-
- loop = self.new_test_loop(gen)
-
- fut = self.new_future(loop)
- fut.set_result('done')
-
- ret = loop.run_until_complete(asyncio.wait_for(fut, 0))
-
- self.assertEqual(ret, 'done')
- self.assertTrue(fut.done())
- self.assertAlmostEqual(0, loop.time())
-
- def test_wait_for_timeout_less_then_0_or_0_coroutine_do_not_started(self):
- def gen():
- when = yield
- self.assertAlmostEqual(0, when)
-
- loop = self.new_test_loop(gen)
-
- foo_started = False
-
- async def foo():
- nonlocal foo_started
- foo_started = True
-
- with self.assertRaises(asyncio.TimeoutError):
- loop.run_until_complete(asyncio.wait_for(foo(), 0))
-
- self.assertAlmostEqual(0, loop.time())
- self.assertEqual(foo_started, False)
-
- def test_wait_for_timeout_less_then_0_or_0(self):
- def gen():
- when = yield
- self.assertAlmostEqual(0.2, when)
- when = yield 0
- self.assertAlmostEqual(0, when)
-
- for timeout in [0, -1]:
- with self.subTest(timeout=timeout):
- loop = self.new_test_loop(gen)
-
- foo_running = None
-
- async def foo():
- nonlocal foo_running
- foo_running = True
- try:
- await asyncio.sleep(0.2)
- finally:
- foo_running = False
- return 'done'
-
- fut = self.new_task(loop, foo())
-
- with self.assertRaises(asyncio.TimeoutError):
- loop.run_until_complete(asyncio.wait_for(fut, timeout))
- self.assertTrue(fut.done())
- # it should have been cancelled due to the timeout
- self.assertTrue(fut.cancelled())
- self.assertAlmostEqual(0, loop.time())
- self.assertEqual(foo_running, False)
-
- def test_wait_for(self):
-
- def gen():
- when = yield
- self.assertAlmostEqual(0.2, when)
- when = yield 0
- self.assertAlmostEqual(0.1, when)
- when = yield 0.1
-
- loop = self.new_test_loop(gen)
-
- foo_running = None
-
- async def foo():
- nonlocal foo_running
- foo_running = True
- try:
- await asyncio.sleep(0.2)
- finally:
- foo_running = False
- return 'done'
-
- fut = self.new_task(loop, foo())
-
- with self.assertRaises(asyncio.TimeoutError):
- loop.run_until_complete(asyncio.wait_for(fut, 0.1))
- self.assertTrue(fut.done())
- # it should have been cancelled due to the timeout
- self.assertTrue(fut.cancelled())
- self.assertAlmostEqual(0.1, loop.time())
- self.assertEqual(foo_running, False)
-
- def test_wait_for_blocking(self):
- loop = self.new_test_loop()
-
- async def coro():
- return 'done'
-
- res = loop.run_until_complete(asyncio.wait_for(coro(), timeout=None))
- self.assertEqual(res, 'done')
-
- def test_wait_for_race_condition(self):
-
- def gen():
- yield 0.1
- yield 0.1
- yield 0.1
-
- loop = self.new_test_loop(gen)
-
- fut = self.new_future(loop)
- task = asyncio.wait_for(fut, timeout=0.2)
- loop.call_later(0.1, fut.set_result, "ok")
- res = loop.run_until_complete(task)
- self.assertEqual(res, "ok")
-
- def test_wait_for_cancellation_race_condition(self):
- def gen():
- yield 0.1
- yield 0.1
- yield 0.1
- yield 0.1
-
- loop = self.new_test_loop(gen)
-
- fut = self.new_future(loop)
- loop.call_later(0.1, fut.set_result, "ok")
- task = loop.create_task(asyncio.wait_for(fut, timeout=1))
- loop.call_later(0.1, task.cancel)
- res = loop.run_until_complete(task)
- self.assertEqual(res, "ok")
-
- def test_wait_for_waits_for_task_cancellation(self):
- loop = asyncio.new_event_loop()
- self.addCleanup(loop.close)
-
- task_done = False
-
- async def foo():
- async def inner():
- nonlocal task_done
- try:
- await asyncio.sleep(0.2)
- except asyncio.CancelledError:
- await asyncio.sleep(_EPSILON)
- raise
- finally:
- task_done = True
-
- inner_task = self.new_task(loop, inner())
-
- await asyncio.wait_for(inner_task, timeout=_EPSILON)
-
- with self.assertRaises(asyncio.TimeoutError) as cm:
- loop.run_until_complete(foo())
-
- self.assertTrue(task_done)
- chained = cm.exception.__context__
- self.assertEqual(type(chained), asyncio.CancelledError)
-
- def test_wait_for_waits_for_task_cancellation_w_timeout_0(self):
- loop = asyncio.new_event_loop()
- self.addCleanup(loop.close)
-
- task_done = False
-
- async def foo():
- async def inner():
- nonlocal task_done
- try:
- await asyncio.sleep(10)
- except asyncio.CancelledError:
- await asyncio.sleep(_EPSILON)
- raise
- finally:
- task_done = True
-
- inner_task = self.new_task(loop, inner())
- await asyncio.sleep(_EPSILON)
- await asyncio.wait_for(inner_task, timeout=0)
-
- with self.assertRaises(asyncio.TimeoutError) as cm:
- loop.run_until_complete(foo())
-
- self.assertTrue(task_done)
- chained = cm.exception.__context__
- self.assertEqual(type(chained), asyncio.CancelledError)
-
- def test_wait_for_reraises_exception_during_cancellation(self):
- loop = asyncio.new_event_loop()
- self.addCleanup(loop.close)
-
- class FooException(Exception):
- pass
-
- async def foo():
- async def inner():
- try:
- await asyncio.sleep(0.2)
- finally:
- raise FooException
-
- inner_task = self.new_task(loop, inner())
-
- await asyncio.wait_for(inner_task, timeout=_EPSILON)
-
- with self.assertRaises(FooException):
- loop.run_until_complete(foo())
-
- def test_wait_for_raises_timeout_error_if_returned_during_cancellation(self):
- loop = asyncio.new_event_loop()
- self.addCleanup(loop.close)
-
- async def foo():
- async def inner():
- try:
- await asyncio.sleep(0.2)
- except asyncio.CancelledError:
- return 42
-
- inner_task = self.new_task(loop, inner())
-
- await asyncio.wait_for(inner_task, timeout=_EPSILON)
-
- with self.assertRaises(asyncio.TimeoutError):
- loop.run_until_complete(foo())
-
- def test_wait_for_self_cancellation(self):
- loop = asyncio.new_event_loop()
- self.addCleanup(loop.close)
-
- async def foo():
- async def inner():
- try:
- await asyncio.sleep(0.3)
- except asyncio.CancelledError:
- try:
- await asyncio.sleep(0.3)
- except asyncio.CancelledError:
- await asyncio.sleep(0.3)
-
- return 42
-
- inner_task = self.new_task(loop, inner())
-
- wait = asyncio.wait_for(inner_task, timeout=0.1)
-
- # Test that wait_for itself is properly cancellable
- # even when the initial task holds up the initial cancellation.
- task = self.new_task(loop, wait)
- await asyncio.sleep(0.2)
- task.cancel()
-
- with self.assertRaises(asyncio.CancelledError):
- await task
-
- self.assertEqual(await inner_task, 42)
-
- loop.run_until_complete(foo())
-
def test_wait(self):
def gen():
@@ -1755,10 +1480,8 @@ async def coro():
self.addCleanup(a.close)
futs = asyncio.as_completed([a])
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
- list(futs)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ list(futs)
def test_as_completed_coroutine_use_running_loop(self):
loop = self.new_test_loop()
@@ -1774,17 +1497,14 @@ async def test():
loop.run_until_complete(test())
def test_as_completed_coroutine_use_global_loop(self):
- # Deprecated in 3.10
+ # Deprecated in 3.10.0, undeprecated in 3.10.9
async def coro():
return 42
loop = self.new_test_loop()
asyncio.set_event_loop(loop)
self.addCleanup(asyncio.set_event_loop, None)
- futs = asyncio.as_completed([coro()])
- with self.assertWarns(DeprecationWarning) as cm:
- futs = list(futs)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ futs = list(asyncio.as_completed([coro()]))
self.assertEqual(len(futs), 1)
self.assertEqual(loop.run_until_complete(futs[0]), 42)
@@ -1990,6 +1710,7 @@ def fn2():
self.assertTrue(asyncio.iscoroutinefunction(fn2))
self.assertFalse(asyncio.iscoroutinefunction(mock.Mock()))
+ self.assertTrue(asyncio.iscoroutinefunction(mock.AsyncMock()))
def test_yield_vs_yield_from(self):
fut = self.new_future(self.loop)
@@ -2253,10 +1974,8 @@ async def coro():
inner = coro()
self.addCleanup(inner.close)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'):
- asyncio.shield(inner)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ asyncio.shield(inner)
def test_shield_coroutine_use_running_loop(self):
async def coro():
@@ -2270,15 +1989,13 @@ async def test():
self.assertEqual(res, 42)
def test_shield_coroutine_use_global_loop(self):
- # Deprecated in 3.10
+ # Deprecated in 3.10.0, undeprecated in 3.10.9
async def coro():
return 42
asyncio.set_event_loop(self.loop)
self.addCleanup(asyncio.set_event_loop, None)
- with self.assertWarns(DeprecationWarning) as cm:
- outer = asyncio.shield(coro())
- self.assertEqual(cm.warnings[0].filename, __file__)
+ outer = asyncio.shield(coro())
self.assertEqual(outer._loop, self.loop)
res = self.loop.run_until_complete(outer)
self.assertEqual(res, 42)
@@ -2559,32 +2276,6 @@ def test_task_source_traceback(self):
'test_task_source_traceback'))
self.loop.run_until_complete(task)
- def _test_cancel_wait_for(self, timeout):
- loop = asyncio.new_event_loop()
- self.addCleanup(loop.close)
-
- async def blocking_coroutine():
- fut = self.new_future(loop)
- # Block: fut result is never set
- await fut
-
- task = loop.create_task(blocking_coroutine())
-
- wait = loop.create_task(asyncio.wait_for(task, timeout))
- loop.call_soon(wait.cancel)
-
- self.assertRaises(asyncio.CancelledError,
- loop.run_until_complete, wait)
-
- # Python issue #23219: cancelling the wait must also cancel the task
- self.assertTrue(task.cancelled())
-
- def test_cancel_blocking_wait_for(self):
- self._test_cancel_wait_for(None)
-
- def test_cancel_wait_for(self):
- self._test_cancel_wait_for(60.0)
-
def test_cancel_gather_1(self):
"""Ensure that a gathering future refuses to be cancelled once all
children are done"""
@@ -2690,6 +2381,7 @@ def coro():
self.new_task(self.loop, gen)
finally:
gen.close()
+ gc.collect() # For PyPy or other GCs.
self.assertTrue(m_log.error.called)
message = m_log.error.call_args[0][0]
@@ -3241,7 +2933,7 @@ def test_current_task_no_running_loop(self):
self.assertIsNone(asyncio.current_task(loop=self.loop))
def test_current_task_no_running_loop_implicit(self):
- with self.assertRaises(RuntimeError):
+ with self.assertRaisesRegex(RuntimeError, 'no running event loop'):
asyncio.current_task()
def test_current_task_with_implicit_loop(self):
@@ -3275,15 +2967,18 @@ class GenericTaskTests(test_utils.TestCase):
def test_future_subclass(self):
self.assertTrue(issubclass(asyncio.Task, asyncio.Future))
+ @support.cpython_only
def test_asyncio_module_compiled(self):
# Because of circular imports it's easy to make _asyncio
# module non-importable. This is a simple test that will
# fail on systems where C modules were successfully compiled
- # (hence the test for _functools), but _asyncio somehow didn't.
+ # (hence the test for _functools etc), but _asyncio somehow didn't.
try:
import _functools
+ import _json
+ import _pickle
except ImportError:
- pass
+ self.skipTest('C modules are not available')
else:
try:
import _asyncio
@@ -3402,10 +3097,8 @@ def _gather(self, *args, **kwargs):
return asyncio.gather(*args, **kwargs)
def test_constructor_empty_sequence_without_loop(self):
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaises(RuntimeError):
- asyncio.gather()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ asyncio.gather()
def test_constructor_empty_sequence_use_running_loop(self):
async def gather():
@@ -3418,12 +3111,10 @@ async def gather():
self.assertEqual(fut.result(), [])
def test_constructor_empty_sequence_use_global_loop(self):
- # Deprecated in 3.10
+ # Deprecated in 3.10.0, undeprecated in 3.10.9
asyncio.set_event_loop(self.one_loop)
self.addCleanup(asyncio.set_event_loop, None)
- with self.assertWarns(DeprecationWarning) as cm:
- fut = asyncio.gather()
- self.assertEqual(cm.warnings[0].filename, __file__)
+ fut = asyncio.gather()
self.assertIsInstance(fut, asyncio.Future)
self.assertIs(fut._loop, self.one_loop)
self._run_loop(self.one_loop)
@@ -3511,10 +3202,8 @@ async def coro():
self.addCleanup(gen1.close)
gen2 = coro()
self.addCleanup(gen2.close)
- with self.assertWarns(DeprecationWarning) as cm:
- with self.assertRaises(RuntimeError):
- asyncio.gather(gen1, gen2)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ with self.assertRaisesRegex(RuntimeError, 'no current event loop'):
+ asyncio.gather(gen1, gen2)
def test_constructor_use_running_loop(self):
async def coro():
@@ -3528,16 +3217,14 @@ async def gather():
self.one_loop.run_until_complete(fut)
def test_constructor_use_global_loop(self):
- # Deprecated in 3.10
+ # Deprecated in 3.10.0, undeprecated in 3.10.9
async def coro():
return 'abc'
asyncio.set_event_loop(self.other_loop)
self.addCleanup(asyncio.set_event_loop, None)
gen1 = coro()
gen2 = coro()
- with self.assertWarns(DeprecationWarning) as cm:
- fut = asyncio.gather(gen1, gen2)
- self.assertEqual(cm.warnings[0].filename, __file__)
+ fut = asyncio.gather(gen1, gen2)
self.assertIs(fut._loop, self.other_loop)
self.other_loop.run_until_complete(fut)
@@ -3604,6 +3291,20 @@ async def outer():
test_utils.run_briefly(self.one_loop)
self.assertIsInstance(f.exception(), RuntimeError)
+ def test_issue46672(self):
+ with mock.patch(
+ 'asyncio.base_events.BaseEventLoop.call_exception_handler',
+ ):
+ async def coro(s):
+ return s
+ c = coro('abc')
+
+ with self.assertRaises(TypeError):
+ self._gather(c, {})
+ self._run_loop(self.one_loop)
+ # NameError should not happen:
+ self.one_loop.call_exception_handler.assert_not_called()
+
class RunCoroutineThreadsafeTests(test_utils.TestCase):
"""Test case for asyncio.run_coroutine_threadsafe."""
@@ -3669,7 +3370,7 @@ def test_run_coroutine_threadsafe_with_timeout(self):
self.assertTrue(task.done())
def test_run_coroutine_threadsafe_task_cancelled(self):
- """Test coroutine submission from a tread to an event loop
+ """Test coroutine submission from a thread to an event loop
when the task is cancelled."""
callback = lambda: self.target(cancel=True)
future = self.loop.run_in_executor(None, callback)
@@ -3677,7 +3378,7 @@ def test_run_coroutine_threadsafe_task_cancelled(self):
self.loop.run_until_complete(future)
def test_run_coroutine_threadsafe_task_factory_exception(self):
- """Test coroutine submission from a tread to an event loop
+ """Test coroutine submission from a thread to an event loop
when the task factory raise an exception."""
def task_factory(loop, coro):
diff --git a/Lib/test/test_asyncio/test_threads.py b/Lib/test/test_asyncio/test_threads.py
index 2af322421dacfa..1138a93e0f78ec 100644
--- a/Lib/test/test_asyncio/test_threads.py
+++ b/Lib/test/test_asyncio/test_threads.py
@@ -5,87 +5,58 @@
from contextvars import ContextVar
from unittest import mock
-from test.test_asyncio import utils as test_utils
def tearDownModule():
asyncio.set_event_loop_policy(None)
-class ToThreadTests(test_utils.TestCase):
- def setUp(self):
- super().setUp()
- self.loop = asyncio.new_event_loop()
- asyncio.set_event_loop(self.loop)
-
- def tearDown(self):
- self.loop.run_until_complete(
- self.loop.shutdown_default_executor())
- self.loop.close()
- asyncio.set_event_loop(None)
- self.loop = None
- super().tearDown()
-
- def test_to_thread(self):
- async def main():
- return await asyncio.to_thread(sum, [40, 2])
-
- result = self.loop.run_until_complete(main())
+class ToThreadTests(unittest.IsolatedAsyncioTestCase):
+ async def test_to_thread(self):
+ result = await asyncio.to_thread(sum, [40, 2])
self.assertEqual(result, 42)
- def test_to_thread_exception(self):
+ async def test_to_thread_exception(self):
def raise_runtime():
raise RuntimeError("test")
- async def main():
- await asyncio.to_thread(raise_runtime)
-
with self.assertRaisesRegex(RuntimeError, "test"):
- self.loop.run_until_complete(main())
+ await asyncio.to_thread(raise_runtime)
- def test_to_thread_once(self):
+ async def test_to_thread_once(self):
func = mock.Mock()
- async def main():
- await asyncio.to_thread(func)
-
- self.loop.run_until_complete(main())
+ await asyncio.to_thread(func)
func.assert_called_once()
- def test_to_thread_concurrent(self):
+ async def test_to_thread_concurrent(self):
func = mock.Mock()
- async def main():
- futs = []
- for _ in range(10):
- fut = asyncio.to_thread(func)
- futs.append(fut)
- await asyncio.gather(*futs)
+ futs = []
+ for _ in range(10):
+ fut = asyncio.to_thread(func)
+ futs.append(fut)
+ await asyncio.gather(*futs)
- self.loop.run_until_complete(main())
self.assertEqual(func.call_count, 10)
- def test_to_thread_args_kwargs(self):
+ async def test_to_thread_args_kwargs(self):
# Unlike run_in_executor(), to_thread() should directly accept kwargs.
func = mock.Mock()
- async def main():
- await asyncio.to_thread(func, 'test', something=True)
+ await asyncio.to_thread(func, 'test', something=True)
- self.loop.run_until_complete(main())
func.assert_called_once_with('test', something=True)
- def test_to_thread_contextvars(self):
+ async def test_to_thread_contextvars(self):
test_ctx = ContextVar('test_ctx')
def get_ctx():
return test_ctx.get()
- async def main():
- test_ctx.set('parrot')
- return await asyncio.to_thread(get_ctx)
+ test_ctx.set('parrot')
+ result = await asyncio.to_thread(get_ctx)
- result = self.loop.run_until_complete(main())
self.assertEqual(result, 'parrot')
diff --git a/Lib/test/test_asyncio/test_transports.py b/Lib/test/test_asyncio/test_transports.py
index df448557a7b7fc..bbdb218efaa3b6 100644
--- a/Lib/test/test_asyncio/test_transports.py
+++ b/Lib/test/test_asyncio/test_transports.py
@@ -7,6 +7,12 @@
from asyncio import transports
+def tearDownModule():
+ # not needed for the test file but added for uniformness with all other
+ # asyncio test files for the sake of unified cleanup
+ asyncio.set_event_loop_policy(None)
+
+
class TransportTests(unittest.TestCase):
def test_ctor_extra_is_none(self):
diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py
index 643638564e342b..01c1214c7f7cc2 100644
--- a/Lib/test/test_asyncio/test_unix_events.py
+++ b/Lib/test/test_asyncio/test_unix_events.py
@@ -26,6 +26,10 @@
from test.test_asyncio import utils as test_utils
+def tearDownModule():
+ asyncio.set_event_loop_policy(None)
+
+
MOCK_ANY = mock.ANY
@@ -34,14 +38,11 @@ def EXITCODE(exitcode):
def SIGNAL(signum):
- assert 1 <= signum <= 68
+ if not 1 <= signum <= 68:
+ raise AssertionError(f'invalid signum {signum}')
return 32768 - signum
-def tearDownModule():
- asyncio.set_event_loop_policy(None)
-
-
def close_pipe_transport(transport):
# Don't call transport.close() because the event loop and the selector
# are mocked
@@ -1529,7 +1530,7 @@ def test_sigchld_child_reaped_elsewhere(self, m_waitpid):
self.watcher._sig_chld()
if isinstance(self.watcher, asyncio.FastChildWatcher):
- # here the FastChildWatche enters a deadlock
+ # here the FastChildWatcher enters a deadlock
# (there is no way to prevent it)
self.assertFalse(callback.called)
else:
@@ -1739,7 +1740,8 @@ def f():
def test_child_watcher_replace_mainloop_existing(self):
policy = self.create_policy()
- loop = policy.get_event_loop()
+ loop = policy.new_event_loop()
+ policy.set_event_loop(loop)
# Explicitly setup SafeChildWatcher,
# default ThreadedChildWatcher has no _loop property
diff --git a/Lib/test/test_asyncio/test_waitfor.py b/Lib/test/test_asyncio/test_waitfor.py
new file mode 100644
index 00000000000000..45498fa097f6bc
--- /dev/null
+++ b/Lib/test/test_asyncio/test_waitfor.py
@@ -0,0 +1,294 @@
+import asyncio
+import unittest
+import time
+
+
+def tearDownModule():
+ asyncio.set_event_loop_policy(None)
+
+
+# The following value can be used as a very small timeout:
+# it passes check "timeout > 0", but has almost
+# no effect on the test performance
+_EPSILON = 0.0001
+
+
+class SlowTask:
+ """ Task will run for this defined time, ignoring cancel requests """
+ TASK_TIMEOUT = 0.2
+
+ def __init__(self):
+ self.exited = False
+
+ async def run(self):
+ exitat = time.monotonic() + self.TASK_TIMEOUT
+
+ while True:
+ tosleep = exitat - time.monotonic()
+ if tosleep <= 0:
+ break
+
+ try:
+ await asyncio.sleep(tosleep)
+ except asyncio.CancelledError:
+ pass
+
+ self.exited = True
+
+
+class AsyncioWaitForTest(unittest.IsolatedAsyncioTestCase):
+
+ async def test_asyncio_wait_for_cancelled(self):
+ t = SlowTask()
+
+ waitfortask = asyncio.create_task(
+ asyncio.wait_for(t.run(), t.TASK_TIMEOUT * 2))
+ await asyncio.sleep(0)
+ waitfortask.cancel()
+ await asyncio.wait({waitfortask})
+
+ self.assertTrue(t.exited)
+
+ async def test_asyncio_wait_for_timeout(self):
+ t = SlowTask()
+
+ try:
+ await asyncio.wait_for(t.run(), t.TASK_TIMEOUT / 2)
+ except asyncio.TimeoutError:
+ pass
+
+ self.assertTrue(t.exited)
+
+ async def test_wait_for_timeout_less_then_0_or_0_future_done(self):
+ loop = asyncio.get_running_loop()
+
+ fut = loop.create_future()
+ fut.set_result('done')
+
+ t0 = loop.time()
+ ret = await asyncio.wait_for(fut, 0)
+ t1 = loop.time()
+
+ self.assertEqual(ret, 'done')
+ self.assertTrue(fut.done())
+ self.assertLess(t1 - t0, 0.1)
+
+ async def test_wait_for_timeout_less_then_0_or_0_coroutine_do_not_started(self):
+ loop = asyncio.get_running_loop()
+
+ foo_started = False
+
+ async def foo():
+ nonlocal foo_started
+ foo_started = True
+
+ with self.assertRaises(asyncio.TimeoutError):
+ t0 = loop.time()
+ await asyncio.wait_for(foo(), 0)
+ t1 = loop.time()
+
+ self.assertEqual(foo_started, False)
+ self.assertLess(t1 - t0, 0.1)
+
+ async def test_wait_for_timeout_less_then_0_or_0(self):
+ loop = asyncio.get_running_loop()
+
+ for timeout in [0, -1]:
+ with self.subTest(timeout=timeout):
+ foo_running = None
+ started = loop.create_future()
+
+ async def foo():
+ nonlocal foo_running
+ foo_running = True
+ started.set_result(None)
+ try:
+ await asyncio.sleep(10)
+ finally:
+ foo_running = False
+ return 'done'
+
+ fut = asyncio.create_task(foo())
+ await started
+
+ with self.assertRaises(asyncio.TimeoutError):
+ t0 = loop.time()
+ await asyncio.wait_for(fut, timeout)
+ t1 = loop.time()
+
+ self.assertTrue(fut.done())
+ # it should have been cancelled due to the timeout
+ self.assertTrue(fut.cancelled())
+ self.assertEqual(foo_running, False)
+ self.assertLess(t1 - t0, 0.1)
+
+ async def test_wait_for(self):
+ loop = asyncio.get_running_loop()
+ foo_running = None
+
+ async def foo():
+ nonlocal foo_running
+ foo_running = True
+ try:
+ await asyncio.sleep(10)
+ finally:
+ foo_running = False
+ return 'done'
+
+ fut = asyncio.create_task(foo())
+
+ with self.assertRaises(asyncio.TimeoutError):
+ t0 = loop.time()
+ await asyncio.wait_for(fut, 0.1)
+ t1 = loop.time()
+ self.assertTrue(fut.done())
+ # it should have been cancelled due to the timeout
+ self.assertTrue(fut.cancelled())
+ self.assertLess(t1 - t0, 0.5)
+ self.assertEqual(foo_running, False)
+
+ async def test_wait_for_blocking(self):
+ async def coro():
+ return 'done'
+
+ res = await asyncio.wait_for(coro(), timeout=None)
+ self.assertEqual(res, 'done')
+
+ async def test_wait_for_race_condition(self):
+ loop = asyncio.get_running_loop()
+
+ fut = loop.create_future()
+ task = asyncio.wait_for(fut, timeout=0.2)
+ loop.call_later(0.1, fut.set_result, "ok")
+ res = await task
+ self.assertEqual(res, "ok")
+
+ async def test_wait_for_cancellation_race_condition(self):
+ async def inner():
+ with self.assertRaises(asyncio.CancelledError):
+ await asyncio.sleep(1)
+ return 1
+
+ result = await asyncio.wait_for(inner(), timeout=.01)
+ self.assertEqual(result, 1)
+
+ async def test_wait_for_waits_for_task_cancellation(self):
+ task_done = False
+
+ async def inner():
+ nonlocal task_done
+ try:
+ await asyncio.sleep(10)
+ except asyncio.CancelledError:
+ await asyncio.sleep(_EPSILON)
+ raise
+ finally:
+ task_done = True
+
+ inner_task = asyncio.create_task(inner())
+
+ with self.assertRaises(asyncio.TimeoutError) as cm:
+ await asyncio.wait_for(inner_task, timeout=_EPSILON)
+
+ self.assertTrue(task_done)
+ chained = cm.exception.__context__
+ self.assertEqual(type(chained), asyncio.CancelledError)
+
+ async def test_wait_for_waits_for_task_cancellation_w_timeout_0(self):
+ task_done = False
+
+ async def foo():
+ async def inner():
+ nonlocal task_done
+ try:
+ await asyncio.sleep(10)
+ except asyncio.CancelledError:
+ await asyncio.sleep(_EPSILON)
+ raise
+ finally:
+ task_done = True
+
+ inner_task = asyncio.create_task(inner())
+ await asyncio.sleep(_EPSILON)
+ await asyncio.wait_for(inner_task, timeout=0)
+
+ with self.assertRaises(asyncio.TimeoutError) as cm:
+ await foo()
+
+ self.assertTrue(task_done)
+ chained = cm.exception.__context__
+ self.assertEqual(type(chained), asyncio.CancelledError)
+
+ async def test_wait_for_reraises_exception_during_cancellation(self):
+ class FooException(Exception):
+ pass
+
+ async def foo():
+ async def inner():
+ try:
+ await asyncio.sleep(0.2)
+ finally:
+ raise FooException
+
+ inner_task = asyncio.create_task(inner())
+
+ await asyncio.wait_for(inner_task, timeout=_EPSILON)
+
+ with self.assertRaises(FooException):
+ await foo()
+
+ async def test_wait_for_self_cancellation(self):
+ async def inner():
+ try:
+ await asyncio.sleep(0.3)
+ except asyncio.CancelledError:
+ try:
+ await asyncio.sleep(0.3)
+ except asyncio.CancelledError:
+ await asyncio.sleep(0.3)
+
+ return 42
+
+ inner_task = asyncio.create_task(inner())
+
+ wait = asyncio.wait_for(inner_task, timeout=0.1)
+
+ # Test that wait_for itself is properly cancellable
+ # even when the initial task holds up the initial cancellation.
+ task = asyncio.create_task(wait)
+ await asyncio.sleep(0.2)
+ task.cancel()
+
+ with self.assertRaises(asyncio.CancelledError):
+ await task
+
+ self.assertEqual(await inner_task, 42)
+
+ async def _test_cancel_wait_for(self, timeout):
+ loop = asyncio.get_running_loop()
+
+ async def blocking_coroutine():
+ fut = loop.create_future()
+ # Block: fut result is never set
+ await fut
+
+ task = asyncio.create_task(blocking_coroutine())
+
+ wait = asyncio.create_task(asyncio.wait_for(task, timeout))
+ loop.call_soon(wait.cancel)
+
+ with self.assertRaises(asyncio.CancelledError):
+ await wait
+
+ # Python issue #23219: cancelling the wait must also cancel the task
+ self.assertTrue(task.cancelled())
+
+ async def test_cancel_blocking_wait_for(self):
+ await self._test_cancel_wait_for(None)
+
+ async def test_cancel_wait_for(self):
+ await self._test_cancel_wait_for(60.0)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py
index f276cd205a2f8e..daf0f1ec1b2327 100644
--- a/Lib/test/test_asyncio/test_windows_events.py
+++ b/Lib/test/test_asyncio/test_windows_events.py
@@ -36,7 +36,23 @@ def data_received(self, data):
self.trans.close()
-class ProactorLoopCtrlC(test_utils.TestCase):
+class WindowsEventsTestCase(test_utils.TestCase):
+ def _unraisablehook(self, unraisable):
+ # Storing unraisable.object can resurrect an object which is being
+ # finalized. Storing unraisable.exc_value creates a reference cycle.
+ self._unraisable = unraisable
+ print(unraisable)
+
+ def setUp(self):
+ self._prev_unraisablehook = sys.unraisablehook
+ self._unraisable = None
+ sys.unraisablehook = self._unraisablehook
+
+ def tearDown(self):
+ sys.unraisablehook = self._prev_unraisablehook
+ self.assertIsNone(self._unraisable)
+
+class ProactorLoopCtrlC(WindowsEventsTestCase):
def test_ctrl_c(self):
@@ -58,7 +74,7 @@ def SIGINT_after_delay():
thread.join()
-class ProactorMultithreading(test_utils.TestCase):
+class ProactorMultithreading(WindowsEventsTestCase):
def test_run_from_nonmain_thread(self):
finished = False
@@ -79,7 +95,7 @@ def func():
self.assertTrue(finished)
-class ProactorTests(test_utils.TestCase):
+class ProactorTests(WindowsEventsTestCase):
def setUp(self):
super().setUp()
@@ -239,8 +255,83 @@ def test_read_self_pipe_restart(self):
self.close_loop(self.loop)
self.assertFalse(self.loop.call_exception_handler.called)
+ def test_address_argument_type_error(self):
+ # Regression test for https://github.com/python/cpython/issues/98793
+ proactor = self.loop._proactor
+ sock = socket.socket(type=socket.SOCK_DGRAM)
+ bad_address = None
+ with self.assertRaises(TypeError):
+ proactor.connect(sock, bad_address)
+ with self.assertRaises(TypeError):
+ proactor.sendto(sock, b'abc', addr=bad_address)
+ sock.close()
+
+ def test_client_pipe_stat(self):
+ res = self.loop.run_until_complete(self._test_client_pipe_stat())
+ self.assertEqual(res, 'done')
-class WinPolicyTests(test_utils.TestCase):
+ async def _test_client_pipe_stat(self):
+ # Regression test for https://github.com/python/cpython/issues/100573
+ ADDRESS = r'\\.\pipe\test_client_pipe_stat-%s' % os.getpid()
+
+ async def probe():
+ # See https://github.com/python/cpython/pull/100959#discussion_r1068533658
+ h = _overlapped.ConnectPipe(ADDRESS)
+ try:
+ _winapi.CloseHandle(_overlapped.ConnectPipe(ADDRESS))
+ except OSError as e:
+ if e.winerror != _overlapped.ERROR_PIPE_BUSY:
+ raise
+ finally:
+ _winapi.CloseHandle(h)
+
+ with self.assertRaises(FileNotFoundError):
+ await probe()
+
+ [server] = await self.loop.start_serving_pipe(asyncio.Protocol, ADDRESS)
+ self.assertIsInstance(server, windows_events.PipeServer)
+
+ errors = []
+ self.loop.set_exception_handler(lambda _, data: errors.append(data))
+
+ for i in range(5):
+ await self.loop.create_task(probe())
+
+ self.assertEqual(len(errors), 0, errors)
+
+ server.close()
+
+ with self.assertRaises(FileNotFoundError):
+ await probe()
+
+ return "done"
+
+ def test_loop_restart(self):
+ # We're fishing for the "RuntimeError: <_overlapped.Overlapped object at XXX>
+ # still has pending operation at deallocation, the process may crash" error
+ stop = threading.Event()
+ def threadMain():
+ while not stop.is_set():
+ self.loop.call_soon_threadsafe(lambda: None)
+ time.sleep(0.01)
+ thr = threading.Thread(target=threadMain)
+
+ # In 10 60-second runs of this test prior to the fix:
+ # time in seconds until failure: (none), 15.0, 6.4, (none), 7.6, 8.3, 1.7, 22.2, 23.5, 8.3
+ # 10 seconds had a 50% failure rate but longer would be more costly
+ end_time = time.time() + 10 # Run for 10 seconds
+ self.loop.call_soon(thr.start)
+ while not self._unraisable: # Stop if we got an unraisable exc
+ self.loop.stop()
+ self.loop.run_forever()
+ if time.time() >= end_time:
+ break
+
+ stop.set()
+ thr.join()
+
+
+class WinPolicyTests(WindowsEventsTestCase):
def test_selector_win_policy(self):
async def main():
diff --git a/Lib/test/test_asyncio/test_windows_utils.py b/Lib/test/test_asyncio/test_windows_utils.py
index 45c09bb4a2d3a0..eafa5be3829682 100644
--- a/Lib/test/test_asyncio/test_windows_utils.py
+++ b/Lib/test/test_asyncio/test_windows_utils.py
@@ -107,7 +107,8 @@ def test_popen(self):
events = [ovin.event, ovout.event, overr.event]
# Super-long timeout for slow buildbots.
- res = _winapi.WaitForMultipleObjects(events, True, 10000)
+ res = _winapi.WaitForMultipleObjects(events, True,
+ int(support.SHORT_TIMEOUT * 1000))
self.assertEqual(res, _winapi.WAIT_OBJECT_0)
self.assertFalse(ovout.pending)
self.assertFalse(overr.pending)
diff --git a/Lib/test/test_asyncio/utils.py b/Lib/test/test_asyncio/utils.py
index 3765194cd0dd27..0b9cde6878f37a 100644
--- a/Lib/test/test_asyncio/utils.py
+++ b/Lib/test/test_asyncio/utils.py
@@ -409,12 +409,13 @@ def _remove_writer(self, fd):
return False
def assert_writer(self, fd, callback, *args):
- assert fd in self.writers, 'fd {} is not registered'.format(fd)
+ if fd not in self.writers:
+ raise AssertionError(f'fd {fd} is not registered')
handle = self.writers[fd]
- assert handle._callback == callback, '{!r} != {!r}'.format(
- handle._callback, callback)
- assert handle._args == args, '{!r} != {!r}'.format(
- handle._args, args)
+ if handle._callback != callback:
+ raise AssertionError(f'{handle._callback!r} != {callback!r}')
+ if handle._args != args:
+ raise AssertionError(f'{handle._args!r} != {args!r}')
def _ensure_fd_no_transport(self, fd):
if not isinstance(fd, int):
@@ -530,7 +531,8 @@ def close_loop(loop):
thread.join()
def set_event_loop(self, loop, *, cleanup=True):
- assert loop is not None
+ if loop is None:
+ raise AssertionError('loop is None')
# ensure that the event loop is passed explicitly in asyncio
events.set_event_loop(None)
if cleanup:
diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py
index 3bd904d1774bc3..ecd1e120ecb515 100644
--- a/Lib/test/test_asyncore.py
+++ b/Lib/test/test_asyncore.py
@@ -1,4 +1,3 @@
-import asyncore
import unittest
import select
import os
@@ -19,6 +18,11 @@
if support.PGO:
raise unittest.SkipTest("test is not helpful for PGO")
+import warnings
+with warnings.catch_warnings():
+ warnings.simplefilter('ignore', DeprecationWarning)
+ import asyncore
+
HAS_UNIX_SOCKETS = hasattr(socket, 'AF_UNIX')
diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py
index 25ff34bb11298a..10a61c60b57e88 100644
--- a/Lib/test/test_audit.py
+++ b/Lib/test/test_audit.py
@@ -16,6 +16,8 @@
class AuditTest(unittest.TestCase):
+ maxDiff = None
+
def do_test(self, *args):
with subprocess.Popen(
[sys.executable, "-X utf8", AUDIT_TESTS_PY, *args],
@@ -54,6 +56,11 @@ def test_block_add_hook(self):
def test_block_add_hook_baseexception(self):
self.do_test("test_block_add_hook_baseexception")
+ def test_marshal(self):
+ import_helper.import_module("marshal")
+
+ self.do_test("test_marshal")
+
def test_pickle(self):
import_helper.import_module("pickle")
@@ -147,10 +154,7 @@ def test_http(self):
def test_sqlite3(self):
- try:
- import sqlite3
- except ImportError:
- return
+ sqlite3 = import_helper.import_module("sqlite3")
returncode, events, stderr = self.run_python("test_sqlite3")
if returncode:
self.fail(stderr)
@@ -168,5 +172,34 @@ def test_sqlite3(self):
self.assertEqual(actual, expected)
+ def test_syslog(self):
+ syslog = import_helper.import_module("syslog")
+
+ returncode, events, stderr = self.run_python("test_syslog")
+ if returncode:
+ self.fail(stderr)
+
+ if support.verbose:
+ print('Events:', *events, sep='\n ')
+
+ self.assertSequenceEqual(
+ events,
+ [('syslog.openlog', ' ', f'python 0 {syslog.LOG_USER}'),
+ ('syslog.syslog', ' ', f'{syslog.LOG_INFO} test'),
+ ('syslog.setlogmask', ' ', f'{syslog.LOG_DEBUG}'),
+ ('syslog.closelog', '', ''),
+ ('syslog.syslog', ' ', f'{syslog.LOG_INFO} test2'),
+ ('syslog.openlog', ' ', f'audit-tests.py 0 {syslog.LOG_USER}'),
+ ('syslog.openlog', ' ', f'audit-tests.py {syslog.LOG_NDELAY} {syslog.LOG_LOCAL0}'),
+ ('syslog.openlog', ' ', f'None 0 {syslog.LOG_USER}'),
+ ('syslog.closelog', '', '')]
+ )
+
+ def test_not_in_gc(self):
+ returncode, _, stderr = self.run_python("test_not_in_gc")
+ if returncode:
+ self.fail(stderr)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/test_base64.py b/Lib/test/test_base64.py
index 4f62c4115f60aa..418492432a1676 100644
--- a/Lib/test/test_base64.py
+++ b/Lib/test/test_base64.py
@@ -714,6 +714,45 @@ def test_decode_nonascii_str(self):
def test_ErrorHeritage(self):
self.assertTrue(issubclass(binascii.Error, ValueError))
+ def test_RFC4648_test_cases(self):
+ # test cases from RFC 4648 section 10
+ b64encode = base64.b64encode
+ b32hexencode = base64.b32hexencode
+ b32encode = base64.b32encode
+ b16encode = base64.b16encode
+
+ self.assertEqual(b64encode(b""), b"")
+ self.assertEqual(b64encode(b"f"), b"Zg==")
+ self.assertEqual(b64encode(b"fo"), b"Zm8=")
+ self.assertEqual(b64encode(b"foo"), b"Zm9v")
+ self.assertEqual(b64encode(b"foob"), b"Zm9vYg==")
+ self.assertEqual(b64encode(b"fooba"), b"Zm9vYmE=")
+ self.assertEqual(b64encode(b"foobar"), b"Zm9vYmFy")
+
+ self.assertEqual(b32encode(b""), b"")
+ self.assertEqual(b32encode(b"f"), b"MY======")
+ self.assertEqual(b32encode(b"fo"), b"MZXQ====")
+ self.assertEqual(b32encode(b"foo"), b"MZXW6===")
+ self.assertEqual(b32encode(b"foob"), b"MZXW6YQ=")
+ self.assertEqual(b32encode(b"fooba"), b"MZXW6YTB")
+ self.assertEqual(b32encode(b"foobar"), b"MZXW6YTBOI======")
+
+ self.assertEqual(b32hexencode(b""), b"")
+ self.assertEqual(b32hexencode(b"f"), b"CO======")
+ self.assertEqual(b32hexencode(b"fo"), b"CPNG====")
+ self.assertEqual(b32hexencode(b"foo"), b"CPNMU===")
+ self.assertEqual(b32hexencode(b"foob"), b"CPNMUOG=")
+ self.assertEqual(b32hexencode(b"fooba"), b"CPNMUOJ1")
+ self.assertEqual(b32hexencode(b"foobar"), b"CPNMUOJ1E8======")
+
+ self.assertEqual(b16encode(b""), b"")
+ self.assertEqual(b16encode(b"f"), b"66")
+ self.assertEqual(b16encode(b"fo"), b"666F")
+ self.assertEqual(b16encode(b"foo"), b"666F6F")
+ self.assertEqual(b16encode(b"foob"), b"666F6F62")
+ self.assertEqual(b16encode(b"fooba"), b"666F6F6261")
+ self.assertEqual(b16encode(b"foobar"), b"666F6F626172")
+
class TestMain(unittest.TestCase):
def tearDown(self):
diff --git a/Lib/test/test_baseexception.py b/Lib/test/test_baseexception.py
index 8db497a1728507..018626e8db63a5 100644
--- a/Lib/test/test_baseexception.py
+++ b/Lib/test/test_baseexception.py
@@ -114,6 +114,31 @@ def test_interface_no_arg(self):
[repr(exc), exc.__class__.__name__ + '()'])
self.interface_test_driver(results)
+ def test_setstate_refcount_no_crash(self):
+ # gh-97591: Acquire strong reference before calling tp_hash slot
+ # in PyObject_SetAttr.
+ import gc
+ d = {}
+ class HashThisKeyWillClearTheDict(str):
+ def __hash__(self) -> int:
+ d.clear()
+ return super().__hash__()
+ class Value(str):
+ pass
+ exc = Exception()
+
+ d[HashThisKeyWillClearTheDict()] = Value() # refcount of Value() is 1 now
+
+ # Exception.__setstate__ should aquire a strong reference of key and
+ # value in the dict. Otherwise, Value()'s refcount would go below
+ # zero in the tp_hash call in PyObject_SetAttr(), and it would cause
+ # crash in GC.
+ exc.__setstate__(d) # __hash__() is called again here, clearing the dict.
+
+ # This GC would crash if the refcount of Value() goes below zero.
+ gc.collect()
+
+
class UsageTests(unittest.TestCase):
"""Test usage of exceptions"""
diff --git a/Lib/test/test_bdb.py b/Lib/test/test_bdb.py
index 398698c79c16df..042c2daea7f797 100644
--- a/Lib/test/test_bdb.py
+++ b/Lib/test/test_bdb.py
@@ -57,9 +57,9 @@
import linecache
from contextlib import contextmanager
from itertools import islice, repeat
-import test.support
from test.support import import_helper
from test.support import os_helper
+from test.support import patch_list
class BdbException(Exception): pass
@@ -714,9 +714,18 @@ def test_until_in_caller_frame(self):
with TracerRun(self) as tracer:
tracer.runcall(tfunc_main)
+ @patch_list(sys.meta_path)
def test_skip(self):
# Check that tracing is skipped over the import statement in
# 'tfunc_import()'.
+
+ # Remove all but the standard importers.
+ sys.meta_path[:] = (
+ item
+ for item in sys.meta_path
+ if item.__module__.startswith('_frozen_importlib')
+ )
+
code = """
def main():
lno = 3
@@ -1193,13 +1202,12 @@ def main():
with TracerRun(self) as tracer:
tracer.runcall(tfunc_import)
-def test_main():
- test.support.run_unittest(
- StateTestCase,
- RunTestCase,
- BreakpointTestCase,
- IssuesTestCase,
- )
+
+class TestRegressions(unittest.TestCase):
+ def test_format_stack_entry_no_lineno(self):
+ # See gh-101517
+ Bdb().format_stack_entry((sys._getframe(), None))
+
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_bigaddrspace.py b/Lib/test/test_bigaddrspace.py
index aa1f8ca75a9811..50272e99607fd3 100644
--- a/Lib/test/test_bigaddrspace.py
+++ b/Lib/test/test_bigaddrspace.py
@@ -92,10 +92,7 @@ def test_repeat(self):
x = None
-def test_main():
- support.run_unittest(BytesTest, StrTest)
-
if __name__ == '__main__':
if len(sys.argv) > 1:
support.set_memlimit(sys.argv[1])
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_bigmem.py b/Lib/test/test_bigmem.py
index 6a244dd8c94800..859f1539e20b80 100644
--- a/Lib/test/test_bigmem.py
+++ b/Lib/test/test_bigmem.py
@@ -1247,11 +1247,8 @@ def test_sort(self, size):
self.assertEqual(l[:10], [1] * 10)
self.assertEqual(l[-10:], [5] * 10)
-def test_main():
- support.run_unittest(StrTest, BytesTest, BytearrayTest,
- TupleTest, ListTest)
if __name__ == '__main__':
if len(sys.argv) > 1:
support.set_memlimit(sys.argv[1])
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py
index 4d1bf2cce1f1e3..13d4e67e586e98 100644
--- a/Lib/test/test_binascii.py
+++ b/Lib/test/test_binascii.py
@@ -4,7 +4,7 @@
import binascii
import array
import re
-from test.support import warnings_helper
+from test.support import bigmemtest, _1G, _4G, warnings_helper
# Note: "*_hex" functions are aliases for "(un)hexlify"
@@ -449,6 +449,14 @@ class BytearrayBinASCIITest(BinASCIITest):
class MemoryviewBinASCIITest(BinASCIITest):
type2test = memoryview
+class ChecksumBigBufferTestCase(unittest.TestCase):
+ """bpo-38256 - check that inputs >=4 GiB are handled correctly."""
+
+ @bigmemtest(size=_4G + 4, memuse=1, dry_run=False)
+ def test_big_buffer(self, size):
+ data = b"nyan" * (_1G + 1)
+ self.assertEqual(binascii.crc32(data), 1044521549)
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/test_bisect.py b/Lib/test/test_bisect.py
index 20f8b9d7c0aa87..ba108221ebf454 100644
--- a/Lib/test/test_bisect.py
+++ b/Lib/test/test_bisect.py
@@ -257,6 +257,12 @@ def test_insort(self):
target
)
+ def test_insort_keynotNone(self):
+ x = []
+ y = {"a": 2, "b": 1}
+ for f in (self.module.insort_left, self.module.insort_right):
+ self.assertRaises(TypeError, f, x, y, key = "b")
+
class TestBisectPython(TestBisect, unittest.TestCase):
module = py_bisect
diff --git a/Lib/test/test_bool.py b/Lib/test/test_bool.py
index a3214c90ed61e4..5215f775c615c2 100644
--- a/Lib/test/test_bool.py
+++ b/Lib/test/test_bool.py
@@ -1,7 +1,6 @@
# Test properties of bool promised by PEP 285
import unittest
-from test import support
from test.support import os_helper
import os
@@ -41,6 +40,12 @@ def test_float(self):
self.assertEqual(float(True), 1.0)
self.assertIsNot(float(True), True)
+ def test_complex(self):
+ self.assertEqual(complex(False), 0j)
+ self.assertEqual(complex(False), False)
+ self.assertEqual(complex(True), 1+0j)
+ self.assertEqual(complex(True), True)
+
def test_math(self):
self.assertEqual(+False, 0)
self.assertIsNot(+False, False)
@@ -370,8 +375,6 @@ def f(x):
f(x)
self.assertGreaterEqual(x.count, 1)
-def test_main():
- support.run_unittest(BoolTest)
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
index bd8353d038b6fe..f31ab72c02094c 100644
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -159,7 +159,7 @@ def test_import(self):
__import__('string')
__import__(name='sys')
__import__(name='time', level=0)
- self.assertRaises(ImportError, __import__, 'spamspam')
+ self.assertRaises(ModuleNotFoundError, __import__, 'spamspam')
self.assertRaises(TypeError, __import__, 1, 2, 3, 4)
self.assertRaises(ValueError, __import__, '')
self.assertRaises(TypeError, __import__, 'sys', name='sys')
@@ -581,8 +581,8 @@ def __dir__(self):
# dir(traceback)
try:
raise IndexError
- except:
- self.assertEqual(len(dir(sys.exc_info()[2])), 4)
+ except IndexError as e:
+ self.assertEqual(len(dir(e.__traceback__)), 4)
# test that object has a __dir__()
self.assertEqual(sorted([].__dir__()), dir([]))
@@ -799,6 +799,16 @@ def test_filter_pickle(self):
f2 = filter(filter_char, "abcdeabcde")
self.check_iter_pickle(f1, list(f2), proto)
+ def test_filter_dealloc(self):
+ # Tests recursive deallocation of nested filter objects using the
+ # thrashcan mechanism. See gh-102356 for more details.
+ max_iters = 1000000
+ i = filter(bool, range(max_iters))
+ for _ in range(max_iters):
+ i = filter(bool, i)
+ del i
+ gc.collect()
+
def test_getattr(self):
self.assertTrue(getattr(sys, 'stdout') is sys.stdout)
self.assertRaises(TypeError, getattr, sys, 1)
@@ -1861,7 +1871,7 @@ def test_warning_notimplemented(self):
# be evaluated in a boolean context (virtually all such use cases
# are a result of accidental misuse implementing rich comparison
# operations in terms of one another).
- # For the time being, it will continue to evaluate as truthy, but
+ # For the time being, it will continue to evaluate as a true value, but
# issue a deprecation warning (with the eventual intent to make it
# a TypeError).
self.assertWarns(DeprecationWarning, bool, NotImplemented)
@@ -2090,12 +2100,24 @@ def test_input_tty(self):
# is different and invokes GNU readline if available).
self.check_input_tty("prompt", b"quux")
+ def skip_if_readline(self):
+ # bpo-13886: When the readline module is loaded, PyOS_Readline() uses
+ # the readline implementation. In some cases, the Python readline
+ # callback rlhandler() is called by readline with a string without
+ # non-ASCII characters. Skip tests on non-ASCII characters if the
+ # readline module is loaded, since test_builtin is not intented to test
+ # the readline module, but the builtins module.
+ if 'readline' in sys.modules:
+ self.skipTest("the readline module is loaded")
+
def test_input_tty_non_ascii(self):
- # Check stdin/stdout encoding is used when invoking GNU readline
+ self.skip_if_readline()
+ # Check stdin/stdout encoding is used when invoking PyOS_Readline()
self.check_input_tty("prompté", b"quux\xe9", "utf-8")
def test_input_tty_non_ascii_unicode_errors(self):
- # Check stdin/stdout error handler is used when invoking GNU readline
+ self.skip_if_readline()
+ # Check stdin/stdout error handler is used when invoking PyOS_Readline()
self.check_input_tty("prompté", b"quux\xe9", "ascii")
def test_input_no_stdout_fileno(self):
@@ -2236,7 +2258,7 @@ def test_type_name(self):
self.assertEqual(A.__module__, __name__)
with self.assertRaises(ValueError):
type('A\x00B', (), {})
- with self.assertRaises(ValueError):
+ with self.assertRaises(UnicodeEncodeError):
type('A\udcdcB', (), {})
with self.assertRaises(TypeError):
type(b'A', (), {})
@@ -2253,7 +2275,7 @@ def test_type_name(self):
with self.assertRaises(ValueError):
A.__name__ = 'A\x00B'
self.assertEqual(A.__name__, 'C')
- with self.assertRaises(ValueError):
+ with self.assertRaises(UnicodeEncodeError):
A.__name__ = 'A\udcdcB'
self.assertEqual(A.__name__, 'C')
with self.assertRaises(TypeError):
diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py
index 381030fe0e8383..72c5e1849f4f57 100644
--- a/Lib/test/test_bytes.py
+++ b/Lib/test/test_bytes.py
@@ -1169,6 +1169,28 @@ class BufferBlocked(bytearray):
self.assertEqual(bytes(ba), b'ab')
self.assertRaises(TypeError, bytes, bb)
+ def test_repeat_id_preserving(self):
+ a = b'123abc1@'
+ b = b'456zyx-+'
+ self.assertEqual(id(a), id(a))
+ self.assertNotEqual(id(a), id(b))
+ self.assertNotEqual(id(a), id(a * -4))
+ self.assertNotEqual(id(a), id(a * 0))
+ self.assertEqual(id(a), id(a * 1))
+ self.assertEqual(id(a), id(1 * a))
+ self.assertNotEqual(id(a), id(a * 2))
+
+ class SubBytes(bytes):
+ pass
+
+ s = SubBytes(b'qwerty()')
+ self.assertEqual(id(s), id(s))
+ self.assertNotEqual(id(s), id(s * -4))
+ self.assertNotEqual(id(s), id(s * 0))
+ self.assertNotEqual(id(s), id(s * 1))
+ self.assertNotEqual(id(s), id(1 * s))
+ self.assertNotEqual(id(s), id(s * 2))
+
class ByteArrayTest(BaseBytesTest, unittest.TestCase):
type2test = bytearray
@@ -1676,6 +1698,23 @@ def test_repeat_after_setslice(self):
self.assertEqual(b1, b)
self.assertEqual(b3, b'xcxcxc')
+ def test_mutating_index(self):
+ class Boom:
+ def __index__(self):
+ b.clear()
+ return 0
+
+ with self.subTest("tp_as_mapping"):
+ b = bytearray(b'Now you see me...')
+ with self.assertRaises(IndexError):
+ b[0] = Boom()
+
+ with self.subTest("tp_as_sequence"):
+ _testcapi = import_helper.import_module('_testcapi')
+ b = bytearray(b'Now you see me...')
+ with self.assertRaises(IndexError):
+ _testcapi.sequence_setitem(b, 0, Boom())
+
class AssortedBytesTest(unittest.TestCase):
#
diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py
index efed3a859ba217..9965c1fe2e5f17 100644
--- a/Lib/test/test_bz2.py
+++ b/Lib/test/test_bz2.py
@@ -1,6 +1,7 @@
from test import support
from test.support import bigmemtest, _4G
+import array
import unittest
from io import BytesIO, DEFAULT_BUFFER_SIZE
import os
@@ -620,6 +621,14 @@ def test_read_truncated(self):
with BZ2File(BytesIO(truncated[:i])) as f:
self.assertRaises(EOFError, f.read, 1)
+ def test_issue44439(self):
+ q = array.array('Q', [1, 2, 3, 4, 5])
+ LENGTH = len(q) * q.itemsize
+
+ with BZ2File(BytesIO(), 'w') as f:
+ self.assertEqual(f.write(q), LENGTH)
+ self.assertEqual(f.tell(), LENGTH)
+
class BZ2CompressorTest(BaseTest):
def testCompress(self):
@@ -996,15 +1005,9 @@ def test_newline(self):
self.assertEqual(f.readlines(), [text])
-def test_main():
- support.run_unittest(
- BZ2FileTest,
- BZ2CompressorTest,
- BZ2DecompressorTest,
- CompressDecompressTest,
- OpenTest,
- )
+def tearDownModule():
support.reap_children()
+
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_c_locale_coercion.py b/Lib/test/test_c_locale_coercion.py
index fcc85992345dbc..71f934756e26a1 100644
--- a/Lib/test/test_c_locale_coercion.py
+++ b/Lib/test/test_c_locale_coercion.py
@@ -427,12 +427,9 @@ def test_PYTHONCOERCECLOCALE_set_to_one(self):
self.assertEqual(cmd.stdout.rstrip(), loc)
-def test_main():
- support.run_unittest(
- LocaleConfigurationTests,
- LocaleCoercionTests
- )
+def tearDownModule():
support.reap_children()
+
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py
index c641e8c418318d..de54f5af31d51f 100644
--- a/Lib/test/test_calendar.py
+++ b/Lib/test/test_calendar.py
@@ -564,6 +564,23 @@ def test_locale_calendars(self):
new_october = calendar.TextCalendar().formatmonthname(2010, 10, 10)
self.assertEqual(old_october, new_october)
+ def test_locale_calendar_formatweekday(self):
+ try:
+ # formatweekday uses different day names based on the available width.
+ cal = calendar.LocaleTextCalendar(locale='en_US')
+ # For really short widths, the abbreviated name is truncated.
+ self.assertEqual(cal.formatweekday(0, 1), "M")
+ self.assertEqual(cal.formatweekday(0, 2), "Mo")
+ # For short widths, a centered, abbreviated name is used.
+ self.assertEqual(cal.formatweekday(0, 3), "Mon")
+ self.assertEqual(cal.formatweekday(0, 5), " Mon ")
+ self.assertEqual(cal.formatweekday(0, 8), " Mon ")
+ # For long widths, the full day name is used.
+ self.assertEqual(cal.formatweekday(0, 9), " Monday ")
+ self.assertEqual(cal.formatweekday(0, 10), " Monday ")
+ except locale.Error:
+ raise unittest.SkipTest('cannot set the en_US locale')
+
def test_locale_html_calendar_custom_css_class_month_name(self):
try:
cal = calendar.LocaleHTMLCalendar(locale='')
@@ -619,6 +636,14 @@ def test_itermonthdays2(self):
self.assertEqual(days[0][1], firstweekday)
self.assertEqual(days[-1][1], (firstweekday - 1) % 7)
+ def test_iterweekdays(self):
+ week0 = list(range(7))
+ for firstweekday in range(7):
+ cal = calendar.Calendar(firstweekday)
+ week = list(cal.iterweekdays())
+ expected = week0[firstweekday:] + week0[:firstweekday]
+ self.assertEqual(week, expected)
+
class MonthCalendarTestCase(unittest.TestCase):
def setUp(self):
@@ -935,8 +960,7 @@ def test_html_output_year_css(self):
class MiscTestCase(unittest.TestCase):
def test__all__(self):
not_exported = {
- 'mdays', 'January', 'February', 'EPOCH', 'MONDAY', 'TUESDAY',
- 'WEDNESDAY', 'THURSDAY', 'FRIDAY', 'SATURDAY', 'SUNDAY',
+ 'mdays', 'January', 'February', 'EPOCH',
'different_locale', 'c', 'prweek', 'week', 'format',
'formatstring', 'main', 'monthlen', 'prevmonth', 'nextmonth'}
support.check__all__(self, calendar, not_exported=not_exported)
diff --git a/Lib/test/test_call.py b/Lib/test/test_call.py
index 3f459222748b3a..1bf1f79fdad9ee 100644
--- a/Lib/test/test_call.py
+++ b/Lib/test/test_call.py
@@ -11,6 +11,14 @@
import contextlib
+class BadStr(str):
+ def __eq__(self, other):
+ return True
+ def __hash__(self):
+ # Guaranteed different hash
+ return str.__hash__(self) ^ 3
+
+
class FunctionCalls(unittest.TestCase):
def test_kwargs_order(self):
@@ -133,6 +141,18 @@ def test_varargs17_kw(self):
self.assertRaisesRegex(TypeError, msg,
print, 0, sep=1, end=2, file=3, flush=4, foo=5)
+ def test_varargs18_kw(self):
+ # _PyArg_UnpackKeywordsWithVararg()
+ msg = r"invalid keyword argument for print\(\)$"
+ with self.assertRaisesRegex(TypeError, msg):
+ print(0, 1, **{BadStr('foo'): ','})
+
+ def test_varargs19_kw(self):
+ # _PyArg_UnpackKeywords()
+ msg = r"invalid keyword argument for round\(\)$"
+ with self.assertRaisesRegex(TypeError, msg):
+ round(1.75, **{BadStr('foo'): 1})
+
def test_oldargs0_1(self):
msg = r"keys\(\) takes no arguments \(1 given\)"
self.assertRaisesRegex(TypeError, msg, {}.keys, 0)
@@ -526,7 +546,7 @@ def __index__(self):
self.kwargs.clear()
gc.collect()
return 0
- x = IntWithDict(dont_inherit=IntWithDict())
+ x = IntWithDict(optimize=IntWithDict())
# We test the argument handling of "compile" here, the compilation
# itself is not relevant. When we pass flags=x below, x.__index__() is
# called, which changes the keywords dict.
diff --git a/Lib/test/test_capi/__init__.py b/Lib/test/test_capi/__init__.py
new file mode 100644
index 00000000000000..4b16ecc31156a5
--- /dev/null
+++ b/Lib/test/test_capi/__init__.py
@@ -0,0 +1,5 @@
+import os
+from test.support import load_package_tests
+
+def load_tests(*args):
+ return load_package_tests(os.path.dirname(__file__), *args)
diff --git a/Lib/test/test_capi/__main__.py b/Lib/test/test_capi/__main__.py
new file mode 100644
index 00000000000000..05d01775ddf43c
--- /dev/null
+++ b/Lib/test/test_capi/__main__.py
@@ -0,0 +1,3 @@
+import unittest
+
+unittest.main('test.test_capi')
diff --git a/Lib/test/test_capi/test_codecs.py b/Lib/test/test_capi/test_codecs.py
new file mode 100644
index 00000000000000..e46726192aa05b
--- /dev/null
+++ b/Lib/test/test_capi/test_codecs.py
@@ -0,0 +1,54 @@
+import unittest
+from test.support import import_helper
+
+_testcapi = import_helper.import_module('_testcapi')
+
+
+class CAPITest(unittest.TestCase):
+
+ def test_decodeutf8(self):
+ """Test PyUnicode_DecodeUTF8()"""
+ decodeutf8 = _testcapi.unicode_decodeutf8
+
+ for s in ['abc', '\xa1\xa2', '\u4f60\u597d', 'a\U0001f600']:
+ b = s.encode('utf-8')
+ self.assertEqual(decodeutf8(b), s)
+ self.assertEqual(decodeutf8(b, 'strict'), s)
+
+ self.assertRaises(UnicodeDecodeError, decodeutf8, b'\x80')
+ self.assertRaises(UnicodeDecodeError, decodeutf8, b'\xc0')
+ self.assertRaises(UnicodeDecodeError, decodeutf8, b'\xff')
+ self.assertRaises(UnicodeDecodeError, decodeutf8, b'a\xf0\x9f')
+ self.assertEqual(decodeutf8(b'a\xf0\x9f', 'replace'), 'a\ufffd')
+ self.assertEqual(decodeutf8(b'a\xf0\x9fb', 'replace'), 'a\ufffdb')
+
+ self.assertRaises(LookupError, decodeutf8, b'a\x80', 'foo')
+ # TODO: Test PyUnicode_DecodeUTF8() with NULL as data and
+ # negative size.
+
+ def test_decodeutf8stateful(self):
+ """Test PyUnicode_DecodeUTF8Stateful()"""
+ decodeutf8stateful = _testcapi.unicode_decodeutf8stateful
+
+ for s in ['abc', '\xa1\xa2', '\u4f60\u597d', 'a\U0001f600']:
+ b = s.encode('utf-8')
+ self.assertEqual(decodeutf8stateful(b), (s, len(b)))
+ self.assertEqual(decodeutf8stateful(b, 'strict'), (s, len(b)))
+
+ self.assertRaises(UnicodeDecodeError, decodeutf8stateful, b'\x80')
+ self.assertRaises(UnicodeDecodeError, decodeutf8stateful, b'\xc0')
+ self.assertRaises(UnicodeDecodeError, decodeutf8stateful, b'\xff')
+ self.assertEqual(decodeutf8stateful(b'a\xf0\x9f'), ('a', 1))
+ self.assertEqual(decodeutf8stateful(b'a\xf0\x9f', 'replace'), ('a', 1))
+ self.assertRaises(UnicodeDecodeError, decodeutf8stateful, b'a\xf0\x9fb')
+ self.assertEqual(decodeutf8stateful(b'a\xf0\x9fb', 'replace'), ('a\ufffdb', 4))
+
+ self.assertRaises(LookupError, decodeutf8stateful, b'a\x80', 'foo')
+ # TODO: Test PyUnicode_DecodeUTF8Stateful() with NULL as data and
+ # negative size.
+ # TODO: Test PyUnicode_DecodeUTF8Stateful() with NULL as the address of
+ # "consumed".
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_capi/test_getargs.py
similarity index 94%
rename from Lib/test/test_getargs2.py
rename to Lib/test/test_capi/test_getargs.py
index c67e6f51a23464..552fa213524723 100644
--- a/Lib/test/test_getargs2.py
+++ b/Lib/test/test_capi/test_getargs.py
@@ -2,6 +2,7 @@
import math
import string
import sys
+import warnings
from test import support
from test.support import import_helper
from test.support import warnings_helper
@@ -746,6 +747,33 @@ def test_surrogate_keyword(self):
"'\udc80' is an invalid keyword argument for this function"):
getargs_keyword_only(1, 2, **{'\uDC80': 10})
+ def test_weird_str_subclass(self):
+ class BadStr(str):
+ def __eq__(self, other):
+ return True
+ def __hash__(self):
+ # Guaranteed different hash
+ return str.__hash__(self) ^ 3
+ with self.assertRaisesRegex(TypeError,
+ "invalid keyword argument for this function"):
+ getargs_keyword_only(1, 2, **{BadStr("keyword_only"): 3})
+ with self.assertRaisesRegex(TypeError,
+ "invalid keyword argument for this function"):
+ getargs_keyword_only(1, 2, **{BadStr("monster"): 666})
+
+ def test_weird_str_subclass2(self):
+ class BadStr(str):
+ def __eq__(self, other):
+ return False
+ def __hash__(self):
+ return str.__hash__(self)
+ with self.assertRaisesRegex(TypeError,
+ "invalid keyword argument for this function"):
+ getargs_keyword_only(1, 2, **{BadStr("keyword_only"): 3})
+ with self.assertRaisesRegex(TypeError,
+ "invalid keyword argument for this function"):
+ getargs_keyword_only(1, 2, **{BadStr("monster"): 666})
+
class PositionalOnlyAndKeywords_TestCase(unittest.TestCase):
from _testcapi import getargs_positional_only_and_keywords as getargs
@@ -874,6 +902,23 @@ def test_s_hash(self):
self.assertRaises(TypeError, getargs_s_hash, memoryview(b'memoryview'))
self.assertRaises(TypeError, getargs_s_hash, None)
+ def test_s_hash_int(self):
+ # "s#" without PY_SSIZE_T_CLEAN defined.
+ from _testcapi import getargs_s_hash_int
+ from _testcapi import getargs_s_hash_int2
+ buf = bytearray([1, 2])
+ self.assertRaises(SystemError, getargs_s_hash_int, buf, "abc")
+ self.assertRaises(SystemError, getargs_s_hash_int, buf, x=42)
+ self.assertRaises(SystemError, getargs_s_hash_int, buf, x="abc")
+ self.assertRaises(SystemError, getargs_s_hash_int2, buf, ("abc",))
+ self.assertRaises(SystemError, getargs_s_hash_int2, buf, x=42)
+ self.assertRaises(SystemError, getargs_s_hash_int2, buf, x="abc")
+ buf.append(3) # still mutable -- not locked by a buffer export
+ # getargs_s_hash_int(buf) may not raise SystemError because skipitem()
+ # is not called. But it is an implementation detail.
+ # getargs_s_hash_int(buf)
+ # getargs_s_hash_int2(buf)
+
def test_z(self):
from _testcapi import getargs_z
self.assertEqual(getargs_z('abc\xe9'), b'abc\xc3\xa9')
@@ -992,6 +1037,9 @@ def test_u(self):
self.assertRaises(TypeError, getargs_u, memoryview(b'memoryview'))
with self.assertWarns(DeprecationWarning):
self.assertRaises(TypeError, getargs_u, None)
+ with warnings.catch_warnings():
+ warnings.simplefilter('error', DeprecationWarning)
+ self.assertRaises(DeprecationWarning, getargs_u, 'abc\xe9')
@support.requires_legacy_unicode_capi
def test_u_hash(self):
@@ -1008,6 +1056,9 @@ def test_u_hash(self):
self.assertRaises(TypeError, getargs_u_hash, memoryview(b'memoryview'))
with self.assertWarns(DeprecationWarning):
self.assertRaises(TypeError, getargs_u_hash, None)
+ with warnings.catch_warnings():
+ warnings.simplefilter('error', DeprecationWarning)
+ self.assertRaises(DeprecationWarning, getargs_u_hash, 'abc\xe9')
@support.requires_legacy_unicode_capi
def test_Z(self):
@@ -1024,6 +1075,9 @@ def test_Z(self):
self.assertRaises(TypeError, getargs_Z, memoryview(b'memoryview'))
with self.assertWarns(DeprecationWarning):
self.assertIsNone(getargs_Z(None))
+ with warnings.catch_warnings():
+ warnings.simplefilter('error', DeprecationWarning)
+ self.assertRaises(DeprecationWarning, getargs_Z, 'abc\xe9')
@support.requires_legacy_unicode_capi
def test_Z_hash(self):
@@ -1040,6 +1094,13 @@ def test_Z_hash(self):
self.assertRaises(TypeError, getargs_Z_hash, memoryview(b'memoryview'))
with self.assertWarns(DeprecationWarning):
self.assertIsNone(getargs_Z_hash(None))
+ with warnings.catch_warnings():
+ warnings.simplefilter('error', DeprecationWarning)
+ self.assertRaises(DeprecationWarning, getargs_Z_hash, 'abc\xe9')
+
+ def test_gh_99240_clear_args(self):
+ from _testcapi import gh_99240_clear_args
+ self.assertRaises(TypeError, gh_99240_clear_args, 'a', '\0b')
class Object_TestCase(unittest.TestCase):
diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi/test_misc.py
similarity index 90%
rename from Lib/test/test_capi.py
rename to Lib/test/test_capi/test_misc.py
index 1b18bfad553007..404a13a0bcc25f 100644
--- a/Lib/test/test_capi.py
+++ b/Lib/test/test_capi/test_misc.py
@@ -264,7 +264,7 @@ def test_return_result_with_error(self):
def test_getitem_with_error(self):
# Test _Py_CheckSlotResult(). Raise an exception and then calls
- # PyObject_GetItem(): check that the assertion catchs the bug.
+ # PyObject_GetItem(): check that the assertion catches the bug.
# PyObject_GetItem() must not be called with an exception set.
code = textwrap.dedent("""
import _testcapi
@@ -371,6 +371,18 @@ def items(self):
self.assertRaises(TypeError, _testcapi.get_mapping_values, bad_mapping)
self.assertRaises(TypeError, _testcapi.get_mapping_items, bad_mapping)
+ def test_mapping_has_key(self):
+ dct = {'a': 1}
+ self.assertTrue(_testcapi.mapping_has_key(dct, 'a'))
+ self.assertFalse(_testcapi.mapping_has_key(dct, 'b'))
+
+ class SubDict(dict):
+ pass
+
+ dct2 = SubDict({'a': 1})
+ self.assertTrue(_testcapi.mapping_has_key(dct2, 'a'))
+ self.assertFalse(_testcapi.mapping_has_key(dct2, 'b'))
+
@unittest.skipUnless(hasattr(_testcapi, 'negative_refcount'),
'need _testcapi.negative_refcount')
def test_negative_refcount(self):
@@ -619,6 +631,26 @@ def test_fatal_error(self):
''')
self.check_fatal_error(code, expected)
+ def test_pyobject_repr_from_null(self):
+ s = _testcapi.pyobject_repr_from_null()
+ self.assertEqual(s, '')
+
+ def test_pyobject_str_from_null(self):
+ s = _testcapi.pyobject_str_from_null()
+ self.assertEqual(s, '')
+
+ def test_pyobject_bytes_from_null(self):
+ s = _testcapi.pyobject_bytes_from_null()
+ self.assertEqual(s, b'')
+
+ def test_Py_CompileString(self):
+ # Check that Py_CompileString respects the coding cookie
+ _compile = _testcapi.Py_CompileString
+ code = b"# -*- coding: latin1 -*-\nprint('\xc2\xa4')\n"
+ result = _compile(code)
+ expected = compile(code, "", "exec")
+ self.assertEqual(result.co_consts, expected.co_consts)
+
class TestPendingCalls(unittest.TestCase):
@@ -635,11 +667,11 @@ def callback():
#unsuccessful.
while True:
if _testcapi._pending_threadfunc(callback):
- break;
+ break
def pendingcalls_wait(self, l, n, context = None):
#now, stick around until l[0] has grown to 10
- count = 0;
+ count = 0
while len(l) != n:
#this busy loop is where we expect to be interrupted to
#run our callbacks. Note that callbacks are only run on the
@@ -750,6 +782,37 @@ def test_mutate_exception(self):
self.assertFalse(hasattr(binascii.Error, "foobar"))
+ def test_module_state_shared_in_global(self):
+ """
+ bpo-44050: Extension module state should be shared between interpreters
+ when it doesn't support sub-interpreters.
+ """
+ r, w = os.pipe()
+ self.addCleanup(os.close, r)
+ self.addCleanup(os.close, w)
+
+ script = textwrap.dedent(f"""
+ import importlib.machinery
+ import importlib.util
+ import os
+
+ fullname = '_test_module_state_shared'
+ origin = importlib.util.find_spec('_testmultiphase').origin
+ loader = importlib.machinery.ExtensionFileLoader(fullname, origin)
+ spec = importlib.util.spec_from_loader(fullname, loader)
+ module = importlib.util.module_from_spec(spec)
+ attr_id = str(id(module.Error)).encode()
+
+ os.write({w}, attr_id)
+ """)
+ exec(script)
+ main_attr_id = os.read(r, 100)
+
+ ret = support.run_in_subinterp(script)
+ self.assertEqual(ret, 0)
+ subinterp_attr_id = os.read(r, 100)
+ self.assertEqual(main_attr_id, subinterp_attr_id)
+
class TestThreadState(unittest.TestCase):
@@ -774,6 +837,20 @@ def callback():
t.start()
t.join()
+ @threading_helper.reap_threads
+ def test_gilstate_ensure_no_deadlock(self):
+ # See https://github.com/python/cpython/issues/96071
+ code = textwrap.dedent(f"""
+ import _testcapi
+
+ def callback():
+ print('callback called')
+
+ _testcapi._test_thread_state(callback)
+ """)
+ ret = assert_python_ok('-X', 'tracemalloc', '-c', code)
+ self.assertIn(b'callback called', ret.out)
+
class Test_testcapi(unittest.TestCase):
locals().update((name, getattr(_testcapi, name))
@@ -799,8 +876,13 @@ class PyMemDebugTests(unittest.TestCase):
def check(self, code):
with support.SuppressCrashReport():
- out = assert_python_failure('-c', code,
- PYTHONMALLOC=self.PYTHONMALLOC)
+ out = assert_python_failure(
+ '-c', code,
+ PYTHONMALLOC=self.PYTHONMALLOC,
+ # FreeBSD: instruct jemalloc to not fill freed() memory
+ # with junk byte 0x5a, see JEMALLOC(3)
+ MALLOC_CONF="junk:false",
+ )
stderr = out.err
return stderr.decode('ascii', 'replace')
@@ -870,7 +952,11 @@ def check_pyobject_is_freed(self, func_name):
except _testcapi.error:
os._exit(1)
''')
- assert_python_ok('-c', code, PYTHONMALLOC=self.PYTHONMALLOC)
+ assert_python_ok(
+ '-c', code,
+ PYTHONMALLOC=self.PYTHONMALLOC,
+ MALLOC_CONF="junk:false",
+ )
def test_pyobject_null_is_freed(self):
self.check_pyobject_is_freed('check_pyobject_null_is_freed')
@@ -970,6 +1056,22 @@ def test_state_access(self):
with self.assertRaises(TypeError):
increment_count(1, 2, 3)
+ def test_get_module_bad_def(self):
+ # _PyType_GetModuleByDef fails gracefully if it doesn't
+ # find what it's looking for.
+ # see bpo-46433
+ instance = self.module.StateAccessType()
+ with self.assertRaises(TypeError):
+ instance.getmodulebydef_bad_def()
+
+ def test_get_module_static_in_mro(self):
+ # Here, the class _PyType_GetModuleByDef is looking for
+ # appears in the MRO after a static type (Exception).
+ # see bpo-46433
+ class Subclass(BaseException, self.module.StateAccessType):
+ pass
+ self.assertIs(Subclass().get_defining_module(), self.module)
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/test_structmembers.py b/Lib/test/test_capi/test_structmembers.py
similarity index 100%
rename from Lib/test/test_structmembers.py
rename to Lib/test/test_capi/test_structmembers.py
diff --git a/Lib/test/test_capi/test_unicode.py b/Lib/test/test_capi/test_unicode.py
new file mode 100644
index 00000000000000..8c2ada8e5a588c
--- /dev/null
+++ b/Lib/test/test_capi/test_unicode.py
@@ -0,0 +1,502 @@
+import unittest
+import sys
+import warnings
+from test import support
+from test.support import import_helper
+from test.support import warnings_helper
+
+try:
+ import _testcapi
+except ImportError:
+ _testcapi = None
+
+
+class CAPITest(unittest.TestCase):
+
+ # Test PyUnicode_FromFormat()
+ def test_from_format(self):
+ import_helper.import_module('ctypes')
+ from ctypes import (
+ c_char_p,
+ pythonapi, py_object, sizeof,
+ c_int, c_long, c_longlong, c_ssize_t,
+ c_uint, c_ulong, c_ulonglong, c_size_t, c_void_p)
+ name = "PyUnicode_FromFormat"
+ _PyUnicode_FromFormat = getattr(pythonapi, name)
+ _PyUnicode_FromFormat.argtypes = (c_char_p,)
+ _PyUnicode_FromFormat.restype = py_object
+
+ def PyUnicode_FromFormat(format, *args):
+ cargs = tuple(
+ py_object(arg) if isinstance(arg, str) else arg
+ for arg in args)
+ return _PyUnicode_FromFormat(format, *cargs)
+
+ def check_format(expected, format, *args):
+ text = PyUnicode_FromFormat(format, *args)
+ self.assertEqual(expected, text)
+
+ # ascii format, non-ascii argument
+ check_format('ascii\x7f=unicode\xe9',
+ b'ascii\x7f=%U', 'unicode\xe9')
+
+ # non-ascii format, ascii argument: ensure that PyUnicode_FromFormatV()
+ # raises an error
+ self.assertRaisesRegex(ValueError,
+ r'^PyUnicode_FromFormatV\(\) expects an ASCII-encoded format '
+ 'string, got a non-ASCII byte: 0xe9$',
+ PyUnicode_FromFormat, b'unicode\xe9=%s', 'ascii')
+
+ # test "%c"
+ check_format('\uabcd',
+ b'%c', c_int(0xabcd))
+ check_format('\U0010ffff',
+ b'%c', c_int(0x10ffff))
+ with self.assertRaises(OverflowError):
+ PyUnicode_FromFormat(b'%c', c_int(0x110000))
+ # Issue #18183
+ check_format('\U00010000\U00100000',
+ b'%c%c', c_int(0x10000), c_int(0x100000))
+
+ # test "%"
+ check_format('%',
+ b'%')
+ check_format('%',
+ b'%%')
+ check_format('%s',
+ b'%%s')
+ check_format('[%]',
+ b'[%%]')
+ check_format('%abc',
+ b'%%%s', b'abc')
+
+ # truncated string
+ check_format('abc',
+ b'%.3s', b'abcdef')
+ check_format('abc[\ufffd',
+ b'%.5s', 'abc[\u20ac]'.encode('utf8'))
+ check_format("'\\u20acABC'",
+ b'%A', '\u20acABC')
+ check_format("'\\u20",
+ b'%.5A', '\u20acABCDEF')
+ check_format("'\u20acABC'",
+ b'%R', '\u20acABC')
+ check_format("'\u20acA",
+ b'%.3R', '\u20acABCDEF')
+ check_format('\u20acAB',
+ b'%.3S', '\u20acABCDEF')
+ check_format('\u20acAB',
+ b'%.3U', '\u20acABCDEF')
+ check_format('\u20acAB',
+ b'%.3V', '\u20acABCDEF', None)
+ check_format('abc[\ufffd',
+ b'%.5V', None, 'abc[\u20ac]'.encode('utf8'))
+
+ # following tests comes from #7330
+ # test width modifier and precision modifier with %S
+ check_format("repr= abc",
+ b'repr=%5S', 'abc')
+ check_format("repr=ab",
+ b'repr=%.2S', 'abc')
+ check_format("repr= ab",
+ b'repr=%5.2S', 'abc')
+
+ # test width modifier and precision modifier with %R
+ check_format("repr= 'abc'",
+ b'repr=%8R', 'abc')
+ check_format("repr='ab",
+ b'repr=%.3R', 'abc')
+ check_format("repr= 'ab",
+ b'repr=%5.3R', 'abc')
+
+ # test width modifier and precision modifier with %A
+ check_format("repr= 'abc'",
+ b'repr=%8A', 'abc')
+ check_format("repr='ab",
+ b'repr=%.3A', 'abc')
+ check_format("repr= 'ab",
+ b'repr=%5.3A', 'abc')
+
+ # test width modifier and precision modifier with %s
+ check_format("repr= abc",
+ b'repr=%5s', b'abc')
+ check_format("repr=ab",
+ b'repr=%.2s', b'abc')
+ check_format("repr= ab",
+ b'repr=%5.2s', b'abc')
+
+ # test width modifier and precision modifier with %U
+ check_format("repr= abc",
+ b'repr=%5U', 'abc')
+ check_format("repr=ab",
+ b'repr=%.2U', 'abc')
+ check_format("repr= ab",
+ b'repr=%5.2U', 'abc')
+
+ # test width modifier and precision modifier with %V
+ check_format("repr= abc",
+ b'repr=%5V', 'abc', b'123')
+ check_format("repr=ab",
+ b'repr=%.2V', 'abc', b'123')
+ check_format("repr= ab",
+ b'repr=%5.2V', 'abc', b'123')
+ check_format("repr= 123",
+ b'repr=%5V', None, b'123')
+ check_format("repr=12",
+ b'repr=%.2V', None, b'123')
+ check_format("repr= 12",
+ b'repr=%5.2V', None, b'123')
+
+ # test integer formats (%i, %d, %u)
+ check_format('010',
+ b'%03i', c_int(10))
+ check_format('0010',
+ b'%0.4i', c_int(10))
+ check_format('-123',
+ b'%i', c_int(-123))
+ check_format('-123',
+ b'%li', c_long(-123))
+ check_format('-123',
+ b'%lli', c_longlong(-123))
+ check_format('-123',
+ b'%zi', c_ssize_t(-123))
+
+ check_format('-123',
+ b'%d', c_int(-123))
+ check_format('-123',
+ b'%ld', c_long(-123))
+ check_format('-123',
+ b'%lld', c_longlong(-123))
+ check_format('-123',
+ b'%zd', c_ssize_t(-123))
+
+ check_format('123',
+ b'%u', c_uint(123))
+ check_format('123',
+ b'%lu', c_ulong(123))
+ check_format('123',
+ b'%llu', c_ulonglong(123))
+ check_format('123',
+ b'%zu', c_size_t(123))
+
+ # test long output
+ min_longlong = -(2 ** (8 * sizeof(c_longlong) - 1))
+ max_longlong = -min_longlong - 1
+ check_format(str(min_longlong),
+ b'%lld', c_longlong(min_longlong))
+ check_format(str(max_longlong),
+ b'%lld', c_longlong(max_longlong))
+ max_ulonglong = 2 ** (8 * sizeof(c_ulonglong)) - 1
+ check_format(str(max_ulonglong),
+ b'%llu', c_ulonglong(max_ulonglong))
+ PyUnicode_FromFormat(b'%p', c_void_p(-1))
+
+ # test padding (width and/or precision)
+ check_format('123'.rjust(10, '0'),
+ b'%010i', c_int(123))
+ check_format('123'.rjust(100),
+ b'%100i', c_int(123))
+ check_format('123'.rjust(100, '0'),
+ b'%.100i', c_int(123))
+ check_format('123'.rjust(80, '0').rjust(100),
+ b'%100.80i', c_int(123))
+
+ check_format('123'.rjust(10, '0'),
+ b'%010u', c_uint(123))
+ check_format('123'.rjust(100),
+ b'%100u', c_uint(123))
+ check_format('123'.rjust(100, '0'),
+ b'%.100u', c_uint(123))
+ check_format('123'.rjust(80, '0').rjust(100),
+ b'%100.80u', c_uint(123))
+
+ check_format('123'.rjust(10, '0'),
+ b'%010x', c_int(0x123))
+ check_format('123'.rjust(100),
+ b'%100x', c_int(0x123))
+ check_format('123'.rjust(100, '0'),
+ b'%.100x', c_int(0x123))
+ check_format('123'.rjust(80, '0').rjust(100),
+ b'%100.80x', c_int(0x123))
+
+ # test %A
+ check_format(r"%A:'abc\xe9\uabcd\U0010ffff'",
+ b'%%A:%A', 'abc\xe9\uabcd\U0010ffff')
+
+ # test %V
+ check_format('repr=abc',
+ b'repr=%V', 'abc', b'xyz')
+
+ # test %p
+ # We cannot test the exact result,
+ # because it returns a hex representation of a C pointer,
+ # which is going to be different each time. But, we can test the format.
+ p_format_regex = r'^0x[a-zA-Z0-9]{3,}$'
+ p_format1 = PyUnicode_FromFormat(b'%p', 'abc')
+ self.assertIsInstance(p_format1, str)
+ self.assertRegex(p_format1, p_format_regex)
+
+ p_format2 = PyUnicode_FromFormat(b'%p %p', '123456', b'xyz')
+ self.assertIsInstance(p_format2, str)
+ self.assertRegex(p_format2,
+ r'0x[a-zA-Z0-9]{3,} 0x[a-zA-Z0-9]{3,}')
+
+ # Extra args are ignored:
+ p_format3 = PyUnicode_FromFormat(b'%p', '123456', None, b'xyz')
+ self.assertIsInstance(p_format3, str)
+ self.assertRegex(p_format3, p_format_regex)
+
+ # Test string decode from parameter of %s using utf-8.
+ # b'\xe4\xba\xba\xe6\xb0\x91' is utf-8 encoded byte sequence of
+ # '\u4eba\u6c11'
+ check_format('repr=\u4eba\u6c11',
+ b'repr=%V', None, b'\xe4\xba\xba\xe6\xb0\x91')
+
+ #Test replace error handler.
+ check_format('repr=abc\ufffd',
+ b'repr=%V', None, b'abc\xff')
+
+ # not supported: copy the raw format string. these tests are just here
+ # to check for crashes and should not be considered as specifications
+ check_format('%s',
+ b'%1%s', b'abc')
+ check_format('%1abc',
+ b'%1abc')
+ check_format('%+i',
+ b'%+i', c_int(10))
+ check_format('%.%s',
+ b'%.%s', b'abc')
+
+ # Issue #33817: empty strings
+ check_format('',
+ b'')
+ check_format('',
+ b'%s', b'')
+
+ # Test PyUnicode_AsWideChar()
+ @support.cpython_only
+ def test_aswidechar(self):
+ from _testcapi import unicode_aswidechar
+ import_helper.import_module('ctypes')
+ from ctypes import c_wchar, sizeof
+
+ wchar, size = unicode_aswidechar('abcdef', 2)
+ self.assertEqual(size, 2)
+ self.assertEqual(wchar, 'ab')
+
+ wchar, size = unicode_aswidechar('abc', 3)
+ self.assertEqual(size, 3)
+ self.assertEqual(wchar, 'abc')
+
+ wchar, size = unicode_aswidechar('abc', 4)
+ self.assertEqual(size, 3)
+ self.assertEqual(wchar, 'abc\0')
+
+ wchar, size = unicode_aswidechar('abc', 10)
+ self.assertEqual(size, 3)
+ self.assertEqual(wchar, 'abc\0')
+
+ wchar, size = unicode_aswidechar('abc\0def', 20)
+ self.assertEqual(size, 7)
+ self.assertEqual(wchar, 'abc\0def\0')
+
+ nonbmp = chr(0x10ffff)
+ if sizeof(c_wchar) == 2:
+ buflen = 3
+ nchar = 2
+ else: # sizeof(c_wchar) == 4
+ buflen = 2
+ nchar = 1
+ wchar, size = unicode_aswidechar(nonbmp, buflen)
+ self.assertEqual(size, nchar)
+ self.assertEqual(wchar, nonbmp + '\0')
+
+ # Test PyUnicode_AsWideCharString()
+ @support.cpython_only
+ def test_aswidecharstring(self):
+ from _testcapi import unicode_aswidecharstring
+ import_helper.import_module('ctypes')
+ from ctypes import c_wchar, sizeof
+
+ wchar, size = unicode_aswidecharstring('abc')
+ self.assertEqual(size, 3)
+ self.assertEqual(wchar, 'abc\0')
+
+ wchar, size = unicode_aswidecharstring('abc\0def')
+ self.assertEqual(size, 7)
+ self.assertEqual(wchar, 'abc\0def\0')
+
+ nonbmp = chr(0x10ffff)
+ if sizeof(c_wchar) == 2:
+ nchar = 2
+ else: # sizeof(c_wchar) == 4
+ nchar = 1
+ wchar, size = unicode_aswidecharstring(nonbmp)
+ self.assertEqual(size, nchar)
+ self.assertEqual(wchar, nonbmp + '\0')
+
+ # Test PyUnicode_AsUCS4()
+ @support.cpython_only
+ def test_asucs4(self):
+ from _testcapi import unicode_asucs4
+ for s in ['abc', '\xa1\xa2', '\u4f60\u597d', 'a\U0001f600',
+ 'a\ud800b\udfffc', '\ud834\udd1e']:
+ l = len(s)
+ self.assertEqual(unicode_asucs4(s, l, True), s+'\0')
+ self.assertEqual(unicode_asucs4(s, l, False), s+'\uffff')
+ self.assertEqual(unicode_asucs4(s, l+1, True), s+'\0\uffff')
+ self.assertEqual(unicode_asucs4(s, l+1, False), s+'\0\uffff')
+ self.assertRaises(SystemError, unicode_asucs4, s, l-1, True)
+ self.assertRaises(SystemError, unicode_asucs4, s, l-2, False)
+ s = '\0'.join([s, s])
+ self.assertEqual(unicode_asucs4(s, len(s), True), s+'\0')
+ self.assertEqual(unicode_asucs4(s, len(s), False), s+'\uffff')
+
+ # Test PyUnicode_AsUTF8()
+ @support.cpython_only
+ def test_asutf8(self):
+ from _testcapi import unicode_asutf8
+
+ bmp = '\u0100'
+ bmp2 = '\uffff'
+ nonbmp = chr(0x10ffff)
+
+ self.assertEqual(unicode_asutf8(bmp), b'\xc4\x80')
+ self.assertEqual(unicode_asutf8(bmp2), b'\xef\xbf\xbf')
+ self.assertEqual(unicode_asutf8(nonbmp), b'\xf4\x8f\xbf\xbf')
+ self.assertRaises(UnicodeEncodeError, unicode_asutf8, 'a\ud800b\udfffc')
+
+ # Test PyUnicode_AsUTF8AndSize()
+ @support.cpython_only
+ def test_asutf8andsize(self):
+ from _testcapi import unicode_asutf8andsize
+
+ bmp = '\u0100'
+ bmp2 = '\uffff'
+ nonbmp = chr(0x10ffff)
+
+ self.assertEqual(unicode_asutf8andsize(bmp), (b'\xc4\x80', 2))
+ self.assertEqual(unicode_asutf8andsize(bmp2), (b'\xef\xbf\xbf', 3))
+ self.assertEqual(unicode_asutf8andsize(nonbmp), (b'\xf4\x8f\xbf\xbf', 4))
+ self.assertRaises(UnicodeEncodeError, unicode_asutf8andsize, 'a\ud800b\udfffc')
+
+ # Test PyUnicode_FindChar()
+ @support.cpython_only
+ def test_findchar(self):
+ from _testcapi import unicode_findchar
+
+ for str in "\xa1", "\u8000\u8080", "\ud800\udc02", "\U0001f100\U0001f1f1":
+ for i, ch in enumerate(str):
+ self.assertEqual(unicode_findchar(str, ord(ch), 0, len(str), 1), i)
+ self.assertEqual(unicode_findchar(str, ord(ch), 0, len(str), -1), i)
+
+ str = "!>_= end
+ self.assertEqual(unicode_findchar(str, ord('!'), 0, 0, 1), -1)
+ self.assertEqual(unicode_findchar(str, ord('!'), len(str), 0, 1), -1)
+ # negative
+ self.assertEqual(unicode_findchar(str, ord('!'), -len(str), -1, 1), 0)
+ self.assertEqual(unicode_findchar(str, ord('!'), -len(str), -1, -1), 0)
+
+ # Test PyUnicode_CopyCharacters()
+ @support.cpython_only
+ def test_copycharacters(self):
+ from _testcapi import unicode_copycharacters
+
+ strings = [
+ 'abcde', '\xa1\xa2\xa3\xa4\xa5',
+ '\u4f60\u597d\u4e16\u754c\uff01',
+ '\U0001f600\U0001f601\U0001f602\U0001f603\U0001f604'
+ ]
+
+ for idx, from_ in enumerate(strings):
+ # wide -> narrow: exceed maxchar limitation
+ for to in strings[:idx]:
+ self.assertRaises(
+ SystemError,
+ unicode_copycharacters, to, 0, from_, 0, 5
+ )
+ # same kind
+ for from_start in range(5):
+ self.assertEqual(
+ unicode_copycharacters(from_, 0, from_, from_start, 5),
+ (from_[from_start:from_start+5].ljust(5, '\0'),
+ 5-from_start)
+ )
+ for to_start in range(5):
+ self.assertEqual(
+ unicode_copycharacters(from_, to_start, from_, to_start, 5),
+ (from_[to_start:to_start+5].rjust(5, '\0'),
+ 5-to_start)
+ )
+ # narrow -> wide
+ # Tests omitted since this creates invalid strings.
+
+ s = strings[0]
+ self.assertRaises(IndexError, unicode_copycharacters, s, 6, s, 0, 5)
+ self.assertRaises(IndexError, unicode_copycharacters, s, -1, s, 0, 5)
+ self.assertRaises(IndexError, unicode_copycharacters, s, 0, s, 6, 5)
+ self.assertRaises(IndexError, unicode_copycharacters, s, 0, s, -1, 5)
+ self.assertRaises(SystemError, unicode_copycharacters, s, 1, s, 0, 5)
+ self.assertRaises(SystemError, unicode_copycharacters, s, 0, s, 0, -1)
+ self.assertRaises(SystemError, unicode_copycharacters, s, 0, b'', 0, 0)
+
+ @support.cpython_only
+ @support.requires_legacy_unicode_capi
+ def test_encode_decimal(self):
+ from _testcapi import unicode_encodedecimal
+ with warnings_helper.check_warnings():
+ warnings.simplefilter('ignore', DeprecationWarning)
+ self.assertEqual(unicode_encodedecimal('123'),
+ b'123')
+ self.assertEqual(unicode_encodedecimal('\u0663.\u0661\u0664'),
+ b'3.14')
+ self.assertEqual(unicode_encodedecimal(
+ "\N{EM SPACE}3.14\N{EN SPACE}"), b' 3.14 ')
+ self.assertRaises(UnicodeEncodeError,
+ unicode_encodedecimal, "123\u20ac", "strict")
+ self.assertRaisesRegex(
+ ValueError,
+ "^'decimal' codec can't encode character",
+ unicode_encodedecimal, "123\u20ac", "replace")
+
+ @support.cpython_only
+ @support.requires_legacy_unicode_capi
+ def test_transform_decimal(self):
+ from _testcapi import unicode_transformdecimaltoascii as transform_decimal
+ with warnings_helper.check_warnings():
+ warnings.simplefilter('ignore', DeprecationWarning)
+ self.assertEqual(transform_decimal('123'),
+ '123')
+ self.assertEqual(transform_decimal('\u0663.\u0661\u0664'),
+ '3.14')
+ self.assertEqual(transform_decimal("\N{EM SPACE}3.14\N{EN SPACE}"),
+ "\N{EM SPACE}3.14\N{EN SPACE}")
+ self.assertEqual(transform_decimal('123\u20ac'),
+ '123\u20ac')
+
+ @support.cpython_only
+ def test_pep393_utf8_caching_bug(self):
+ # Issue #25709: Problem with string concatenation and utf-8 cache
+ from _testcapi import getargs_s_hash
+ for k in 0x24, 0xa4, 0x20ac, 0x1f40d:
+ s = ''
+ for i in range(5):
+ # Due to CPython specific optimization the 's' string can be
+ # resized in-place.
+ s += chr(k)
+ # Parsing with the "s#" format code calls indirectly
+ # PyUnicode_AsUTF8AndSize() which creates the UTF-8
+ # encoded string cached in the Unicode object.
+ self.assertEqual(getargs_s_hash(s), chr(k).encode() * (i + 1))
+ # Check that the second call returns the same result
+ self.assertEqual(getargs_s_hash(s), chr(k).encode() * (i + 1))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/Lib/test/test_check_c_globals.py b/Lib/test/test_check_c_globals.py
index 030debc452e409..898807a5e69259 100644
--- a/Lib/test/test_check_c_globals.py
+++ b/Lib/test/test_check_c_globals.py
@@ -1,9 +1,14 @@
import unittest
import test.test_tools
+from test.support.warnings_helper import save_restore_warnings_filters
test.test_tools.skip_if_missing('c-analyzer')
with test.test_tools.imports_under_tool('c-analyzer'):
- from cpython.__main__ import main
+ # gh-95349: Save/restore warnings filters to leave them unchanged.
+ # Importing the c-analyzer imports docutils which imports pkg_resources
+ # which adds a warnings filter.
+ with save_restore_warnings_filters():
+ from cpython.__main__ import main
class ActualChecks(unittest.TestCase):
diff --git a/Lib/test/test_clinic.py b/Lib/test/test_clinic.py
index 4aa9691a4829d1..feebb6add09a1d 100644
--- a/Lib/test/test_clinic.py
+++ b/Lib/test/test_clinic.py
@@ -3,7 +3,7 @@
# Licensed to the PSF under a contributor agreement.
from test import support, test_tools
-from test.support import os_helper
+from test.support import import_helper, os_helper
from unittest import TestCase
import collections
import inspect
@@ -820,5 +820,413 @@ def test_external(self):
self.assertEqual(new_mtime_ns, old_mtime_ns)
+ac_tester = import_helper.import_module('_testclinic')
+
+
+class ClinicFunctionalTest(unittest.TestCase):
+ locals().update((name, getattr(ac_tester, name))
+ for name in dir(ac_tester) if name.startswith('test_'))
+
+ def test_objects_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.objects_converter()
+ self.assertEqual(ac_tester.objects_converter(1, 2), (1, 2))
+ self.assertEqual(ac_tester.objects_converter([], 'whatever class'), ([], 'whatever class'))
+ self.assertEqual(ac_tester.objects_converter(1), (1, None))
+
+ def test_bytes_object_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.bytes_object_converter(1)
+ self.assertEqual(ac_tester.bytes_object_converter(b'BytesObject'), (b'BytesObject',))
+
+ def test_byte_array_object_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.byte_array_object_converter(1)
+ byte_arr = bytearray(b'ByteArrayObject')
+ self.assertEqual(ac_tester.byte_array_object_converter(byte_arr), (byte_arr,))
+
+ def test_unicode_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.unicode_converter(1)
+ self.assertEqual(ac_tester.unicode_converter('unicode'), ('unicode',))
+
+ def test_bool_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.bool_converter(False, False, 'not a int')
+ self.assertEqual(ac_tester.bool_converter(), (True, True, True))
+ self.assertEqual(ac_tester.bool_converter('', [], 5), (False, False, True))
+ self.assertEqual(ac_tester.bool_converter(('not empty',), {1: 2}, 0), (True, True, False))
+
+ def test_char_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.char_converter(1)
+ with self.assertRaises(TypeError):
+ ac_tester.char_converter(b'ab')
+ chars = [b'A', b'\a', b'\b', b'\t', b'\n', b'\v', b'\f', b'\r', b'"', b"'", b'?', b'\\', b'\000', b'\377']
+ expected = tuple(ord(c) for c in chars)
+ self.assertEqual(ac_tester.char_converter(), expected)
+ chars = [b'1', b'2', b'3', b'4', b'5', b'6', b'7', b'8', b'9', b'0', b'a', b'b', b'c', b'd']
+ expected = tuple(ord(c) for c in chars)
+ self.assertEqual(ac_tester.char_converter(*chars), expected)
+
+ def test_unsigned_char_converter(self):
+ from _testcapi import UCHAR_MAX
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_char_converter(-1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_char_converter(UCHAR_MAX + 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_char_converter(0, UCHAR_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.unsigned_char_converter([])
+ self.assertEqual(ac_tester.unsigned_char_converter(), (12, 34, 56))
+ self.assertEqual(ac_tester.unsigned_char_converter(0, 0, UCHAR_MAX + 1), (0, 0, 0))
+ self.assertEqual(ac_tester.unsigned_char_converter(0, 0, (UCHAR_MAX + 1) * 3 + 123), (0, 0, 123))
+
+ def test_short_converter(self):
+ from _testcapi import SHRT_MIN, SHRT_MAX
+ with self.assertRaises(OverflowError):
+ ac_tester.short_converter(SHRT_MIN - 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.short_converter(SHRT_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.short_converter([])
+ self.assertEqual(ac_tester.short_converter(-1234), (-1234,))
+ self.assertEqual(ac_tester.short_converter(4321), (4321,))
+
+ def test_unsigned_short_converter(self):
+ from _testcapi import USHRT_MAX
+ with self.assertRaises(ValueError):
+ ac_tester.unsigned_short_converter(-1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_short_converter(USHRT_MAX + 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_short_converter(0, USHRT_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.unsigned_short_converter([])
+ self.assertEqual(ac_tester.unsigned_short_converter(), (12, 34, 56))
+ self.assertEqual(ac_tester.unsigned_short_converter(0, 0, USHRT_MAX + 1), (0, 0, 0))
+ self.assertEqual(ac_tester.unsigned_short_converter(0, 0, (USHRT_MAX + 1) * 3 + 123), (0, 0, 123))
+
+ def test_int_converter(self):
+ from _testcapi import INT_MIN, INT_MAX
+ with self.assertRaises(OverflowError):
+ ac_tester.int_converter(INT_MIN - 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.int_converter(INT_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.int_converter(1, 2, 3)
+ with self.assertRaises(TypeError):
+ ac_tester.int_converter([])
+ self.assertEqual(ac_tester.int_converter(), (12, 34, 45))
+ self.assertEqual(ac_tester.int_converter(1, 2, '3'), (1, 2, ord('3')))
+
+ def test_unsigned_int_converter(self):
+ from _testcapi import UINT_MAX
+ with self.assertRaises(ValueError):
+ ac_tester.unsigned_int_converter(-1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_int_converter(UINT_MAX + 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_int_converter(0, UINT_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.unsigned_int_converter([])
+ self.assertEqual(ac_tester.unsigned_int_converter(), (12, 34, 56))
+ self.assertEqual(ac_tester.unsigned_int_converter(0, 0, UINT_MAX + 1), (0, 0, 0))
+ self.assertEqual(ac_tester.unsigned_int_converter(0, 0, (UINT_MAX + 1) * 3 + 123), (0, 0, 123))
+
+ def test_long_converter(self):
+ from _testcapi import LONG_MIN, LONG_MAX
+ with self.assertRaises(OverflowError):
+ ac_tester.long_converter(LONG_MIN - 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.long_converter(LONG_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.long_converter([])
+ self.assertEqual(ac_tester.long_converter(), (12,))
+ self.assertEqual(ac_tester.long_converter(-1234), (-1234,))
+
+ def test_unsigned_long_converter(self):
+ from _testcapi import ULONG_MAX
+ with self.assertRaises(ValueError):
+ ac_tester.unsigned_long_converter(-1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_long_converter(ULONG_MAX + 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_long_converter(0, ULONG_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.unsigned_long_converter([])
+ self.assertEqual(ac_tester.unsigned_long_converter(), (12, 34, 56))
+ self.assertEqual(ac_tester.unsigned_long_converter(0, 0, ULONG_MAX + 1), (0, 0, 0))
+ self.assertEqual(ac_tester.unsigned_long_converter(0, 0, (ULONG_MAX + 1) * 3 + 123), (0, 0, 123))
+
+ def test_long_long_converter(self):
+ from _testcapi import LLONG_MIN, LLONG_MAX
+ with self.assertRaises(OverflowError):
+ ac_tester.long_long_converter(LLONG_MIN - 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.long_long_converter(LLONG_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.long_long_converter([])
+ self.assertEqual(ac_tester.long_long_converter(), (12,))
+ self.assertEqual(ac_tester.long_long_converter(-1234), (-1234,))
+
+ def test_unsigned_long_long_converter(self):
+ from _testcapi import ULLONG_MAX
+ with self.assertRaises(ValueError):
+ ac_tester.unsigned_long_long_converter(-1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_long_long_converter(ULLONG_MAX + 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.unsigned_long_long_converter(0, ULLONG_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.unsigned_long_long_converter([])
+ self.assertEqual(ac_tester.unsigned_long_long_converter(), (12, 34, 56))
+ self.assertEqual(ac_tester.unsigned_long_long_converter(0, 0, ULLONG_MAX + 1), (0, 0, 0))
+ self.assertEqual(ac_tester.unsigned_long_long_converter(0, 0, (ULLONG_MAX + 1) * 3 + 123), (0, 0, 123))
+
+ def test_py_ssize_t_converter(self):
+ from _testcapi import PY_SSIZE_T_MIN, PY_SSIZE_T_MAX
+ with self.assertRaises(OverflowError):
+ ac_tester.py_ssize_t_converter(PY_SSIZE_T_MIN - 1)
+ with self.assertRaises(OverflowError):
+ ac_tester.py_ssize_t_converter(PY_SSIZE_T_MAX + 1)
+ with self.assertRaises(TypeError):
+ ac_tester.py_ssize_t_converter([])
+ self.assertEqual(ac_tester.py_ssize_t_converter(), (12, 34, 56))
+ self.assertEqual(ac_tester.py_ssize_t_converter(1, 2, None), (1, 2, 56))
+
+ def test_slice_index_converter(self):
+ from _testcapi import PY_SSIZE_T_MIN, PY_SSIZE_T_MAX
+ with self.assertRaises(TypeError):
+ ac_tester.slice_index_converter([])
+ self.assertEqual(ac_tester.slice_index_converter(), (12, 34, 56))
+ self.assertEqual(ac_tester.slice_index_converter(1, 2, None), (1, 2, 56))
+ self.assertEqual(ac_tester.slice_index_converter(PY_SSIZE_T_MAX, PY_SSIZE_T_MAX + 1, PY_SSIZE_T_MAX + 1234),
+ (PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX))
+ self.assertEqual(ac_tester.slice_index_converter(PY_SSIZE_T_MIN, PY_SSIZE_T_MIN - 1, PY_SSIZE_T_MIN - 1234),
+ (PY_SSIZE_T_MIN, PY_SSIZE_T_MIN, PY_SSIZE_T_MIN))
+
+ def test_size_t_converter(self):
+ with self.assertRaises(ValueError):
+ ac_tester.size_t_converter(-1)
+ with self.assertRaises(TypeError):
+ ac_tester.size_t_converter([])
+ self.assertEqual(ac_tester.size_t_converter(), (12,))
+
+ def test_float_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.float_converter([])
+ self.assertEqual(ac_tester.float_converter(), (12.5,))
+ self.assertEqual(ac_tester.float_converter(-0.5), (-0.5,))
+
+ def test_double_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.double_converter([])
+ self.assertEqual(ac_tester.double_converter(), (12.5,))
+ self.assertEqual(ac_tester.double_converter(-0.5), (-0.5,))
+
+ def test_py_complex_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.py_complex_converter([])
+ self.assertEqual(ac_tester.py_complex_converter(complex(1, 2)), (complex(1, 2),))
+ self.assertEqual(ac_tester.py_complex_converter(complex('-1-2j')), (complex('-1-2j'),))
+ self.assertEqual(ac_tester.py_complex_converter(-0.5), (-0.5,))
+ self.assertEqual(ac_tester.py_complex_converter(10), (10,))
+
+ def test_str_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.str_converter(1)
+ with self.assertRaises(TypeError):
+ ac_tester.str_converter('a', 'b', 'c')
+ with self.assertRaises(ValueError):
+ ac_tester.str_converter('a', b'b\0b', 'c')
+ self.assertEqual(ac_tester.str_converter('a', b'b', 'c'), ('a', 'b', 'c'))
+ self.assertEqual(ac_tester.str_converter('a', b'b', b'c'), ('a', 'b', 'c'))
+ self.assertEqual(ac_tester.str_converter('a', b'b', 'c\0c'), ('a', 'b', 'c\0c'))
+
+ def test_str_converter_encoding(self):
+ with self.assertRaises(TypeError):
+ ac_tester.str_converter_encoding(1)
+ self.assertEqual(ac_tester.str_converter_encoding('a', 'b', 'c'), ('a', 'b', 'c'))
+ with self.assertRaises(TypeError):
+ ac_tester.str_converter_encoding('a', b'b\0b', 'c')
+ self.assertEqual(ac_tester.str_converter_encoding('a', b'b', bytearray([ord('c')])), ('a', 'b', 'c'))
+ self.assertEqual(ac_tester.str_converter_encoding('a', b'b', bytearray([ord('c'), 0, ord('c')])),
+ ('a', 'b', 'c\x00c'))
+ self.assertEqual(ac_tester.str_converter_encoding('a', b'b', b'c\x00c'), ('a', 'b', 'c\x00c'))
+
+ def test_py_buffer_converter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.py_buffer_converter('a', 'b')
+ self.assertEqual(ac_tester.py_buffer_converter('abc', bytearray([1, 2, 3])), (b'abc', b'\x01\x02\x03'))
+
+ def test_keywords(self):
+ self.assertEqual(ac_tester.keywords(1, 2), (1, 2))
+ self.assertEqual(ac_tester.keywords(1, b=2), (1, 2))
+ self.assertEqual(ac_tester.keywords(a=1, b=2), (1, 2))
+
+ def test_keywords_kwonly(self):
+ with self.assertRaises(TypeError):
+ ac_tester.keywords_kwonly(1, 2)
+ self.assertEqual(ac_tester.keywords_kwonly(1, b=2), (1, 2))
+ self.assertEqual(ac_tester.keywords_kwonly(a=1, b=2), (1, 2))
+
+ def test_keywords_opt(self):
+ self.assertEqual(ac_tester.keywords_opt(1), (1, None, None))
+ self.assertEqual(ac_tester.keywords_opt(1, 2), (1, 2, None))
+ self.assertEqual(ac_tester.keywords_opt(1, 2, 3), (1, 2, 3))
+ self.assertEqual(ac_tester.keywords_opt(1, b=2), (1, 2, None))
+ self.assertEqual(ac_tester.keywords_opt(1, 2, c=3), (1, 2, 3))
+ self.assertEqual(ac_tester.keywords_opt(a=1, c=3), (1, None, 3))
+ self.assertEqual(ac_tester.keywords_opt(a=1, b=2, c=3), (1, 2, 3))
+
+ def test_keywords_opt_kwonly(self):
+ self.assertEqual(ac_tester.keywords_opt_kwonly(1), (1, None, None, None))
+ self.assertEqual(ac_tester.keywords_opt_kwonly(1, 2), (1, 2, None, None))
+ with self.assertRaises(TypeError):
+ ac_tester.keywords_opt_kwonly(1, 2, 3)
+ self.assertEqual(ac_tester.keywords_opt_kwonly(1, b=2), (1, 2, None, None))
+ self.assertEqual(ac_tester.keywords_opt_kwonly(1, 2, c=3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.keywords_opt_kwonly(a=1, c=3), (1, None, 3, None))
+ self.assertEqual(ac_tester.keywords_opt_kwonly(a=1, b=2, c=3, d=4), (1, 2, 3, 4))
+
+ def test_keywords_kwonly_opt(self):
+ self.assertEqual(ac_tester.keywords_kwonly_opt(1), (1, None, None))
+ with self.assertRaises(TypeError):
+ ac_tester.keywords_kwonly_opt(1, 2)
+ self.assertEqual(ac_tester.keywords_kwonly_opt(1, b=2), (1, 2, None))
+ self.assertEqual(ac_tester.keywords_kwonly_opt(a=1, c=3), (1, None, 3))
+ self.assertEqual(ac_tester.keywords_kwonly_opt(a=1, b=2, c=3), (1, 2, 3))
+
+ def test_posonly_keywords(self):
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords(1)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords(a=1, b=2)
+ self.assertEqual(ac_tester.posonly_keywords(1, 2), (1, 2))
+ self.assertEqual(ac_tester.posonly_keywords(1, b=2), (1, 2))
+
+ def test_posonly_kwonly(self):
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_kwonly(1)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_kwonly(1, 2)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_kwonly(a=1, b=2)
+ self.assertEqual(ac_tester.posonly_kwonly(1, b=2), (1, 2))
+
+ def test_posonly_keywords_kwonly(self):
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly(1)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly(1, 2, 3)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly(a=1, b=2, c=3)
+ self.assertEqual(ac_tester.posonly_keywords_kwonly(1, 2, c=3), (1, 2, 3))
+ self.assertEqual(ac_tester.posonly_keywords_kwonly(1, b=2, c=3), (1, 2, 3))
+
+ def test_posonly_keywords_opt(self):
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_opt(1)
+ self.assertEqual(ac_tester.posonly_keywords_opt(1, 2), (1, 2, None, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt(1, 2, 3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt(1, 2, 3, 4), (1, 2, 3, 4))
+ self.assertEqual(ac_tester.posonly_keywords_opt(1, b=2), (1, 2, None, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt(1, 2, c=3), (1, 2, 3, None))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_opt(a=1, b=2, c=3, d=4)
+ self.assertEqual(ac_tester.posonly_keywords_opt(1, b=2, c=3, d=4), (1, 2, 3, 4))
+
+ def test_posonly_opt_keywords_opt(self):
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt(1), (1, None, None, None))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt(1, 2), (1, 2, None, None))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt(1, 2, 3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt(1, 2, 3, 4), (1, 2, 3, 4))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_opt_keywords_opt(1, b=2)
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt(1, 2, c=3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt(1, 2, c=3, d=4), (1, 2, 3, 4))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_opt_keywords_opt(a=1, b=2, c=3, d=4)
+
+ def test_posonly_kwonly_opt(self):
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_kwonly_opt(1)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_kwonly_opt(1, 2)
+ self.assertEqual(ac_tester.posonly_kwonly_opt(1, b=2), (1, 2, None, None))
+ self.assertEqual(ac_tester.posonly_kwonly_opt(1, b=2, c=3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.posonly_kwonly_opt(1, b=2, c=3, d=4), (1, 2, 3, 4))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_kwonly_opt(a=1, b=2, c=3, d=4)
+
+ def test_posonly_opt_kwonly_opt(self):
+ self.assertEqual(ac_tester.posonly_opt_kwonly_opt(1), (1, None, None, None))
+ self.assertEqual(ac_tester.posonly_opt_kwonly_opt(1, 2), (1, 2, None, None))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_opt_kwonly_opt(1, 2, 3)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_opt_kwonly_opt(1, b=2)
+ self.assertEqual(ac_tester.posonly_opt_kwonly_opt(1, 2, c=3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.posonly_opt_kwonly_opt(1, 2, c=3, d=4), (1, 2, 3, 4))
+
+ def test_posonly_keywords_kwonly_opt(self):
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly_opt(1)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly_opt(1, 2)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly_opt(1, b=2)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly_opt(1, 2, 3)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_kwonly_opt(a=1, b=2, c=3)
+ self.assertEqual(ac_tester.posonly_keywords_kwonly_opt(1, 2, c=3), (1, 2, 3, None, None))
+ self.assertEqual(ac_tester.posonly_keywords_kwonly_opt(1, b=2, c=3), (1, 2, 3, None, None))
+ self.assertEqual(ac_tester.posonly_keywords_kwonly_opt(1, 2, c=3, d=4), (1, 2, 3, 4, None))
+ self.assertEqual(ac_tester.posonly_keywords_kwonly_opt(1, 2, c=3, d=4, e=5), (1, 2, 3, 4, 5))
+
+ def test_posonly_keywords_opt_kwonly_opt(self):
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_opt_kwonly_opt(1)
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, 2), (1, 2, None, None, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, b=2), (1, 2, None, None, None))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_opt_kwonly_opt(1, 2, 3, 4)
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_keywords_opt_kwonly_opt(a=1, b=2)
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, 2, c=3), (1, 2, 3, None, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, b=2, c=3), (1, 2, 3, None, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, 2, 3, d=4), (1, 2, 3, 4, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, 2, c=3, d=4), (1, 2, 3, 4, None))
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, 2, 3, d=4, e=5), (1, 2, 3, 4, 5))
+ self.assertEqual(ac_tester.posonly_keywords_opt_kwonly_opt(1, 2, c=3, d=4, e=5), (1, 2, 3, 4, 5))
+
+ def test_posonly_opt_keywords_opt_kwonly_opt(self):
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt_kwonly_opt(1), (1, None, None, None))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt_kwonly_opt(1, 2), (1, 2, None, None))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_opt_keywords_opt_kwonly_opt(1, b=2)
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt_kwonly_opt(1, 2, 3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt_kwonly_opt(1, 2, c=3), (1, 2, 3, None))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt_kwonly_opt(1, 2, 3, d=4), (1, 2, 3, 4))
+ self.assertEqual(ac_tester.posonly_opt_keywords_opt_kwonly_opt(1, 2, c=3, d=4), (1, 2, 3, 4))
+ with self.assertRaises(TypeError):
+ ac_tester.posonly_opt_keywords_opt_kwonly_opt(1, 2, 3, 4)
+
+ def test_keyword_only_parameter(self):
+ with self.assertRaises(TypeError):
+ ac_tester.keyword_only_parameter()
+ with self.assertRaises(TypeError):
+ ac_tester.keyword_only_parameter(1)
+ self.assertEqual(ac_tester.keyword_only_parameter(a=1), (1,))
+
+ def test_gh_99240_double_free(self):
+ expected_error = r'gh_99240_double_free\(\) argument 2 must be encoded string without null bytes, not str'
+ with self.assertRaisesRegex(TypeError, expected_error):
+ ac_tester.gh_99240_double_free('a', '\0b')
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
index cbf9ff2a863c3c..14de3d40b9d30d 100644
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -137,6 +137,17 @@ def test_run_module_bug1764407(self):
self.assertTrue(data.find(b'1 loop') != -1)
self.assertTrue(data.find(b'__main__.Timer') != -1)
+ def test_relativedir_bug46421(self):
+ # Test `python -m unittest` with a relative directory beginning with ./
+ # Note: We have to switch to the project's top module's directory, as per
+ # the python unittest wiki. We will switch back when we are done.
+ defaultwd = os.getcwd()
+ projectlibpath = os.path.dirname(__file__).removesuffix("test")
+ with os_helper.change_cwd(projectlibpath):
+ # Testing with and without ./
+ assert_python_ok('-m', 'unittest', "test/test_longexp.py")
+ assert_python_ok('-m', 'unittest', "./test/test_longexp.py")
+
def test_run_code(self):
# Test expected operation of the '-c' switch
# Switch needs an argument
@@ -814,6 +825,41 @@ def test_parsing_error(self):
self.assertTrue(proc.stderr.startswith(err_msg), proc.stderr)
self.assertNotEqual(proc.returncode, 0)
+ def test_int_max_str_digits(self):
+ code = "import sys; print(sys.flags.int_max_str_digits, sys.get_int_max_str_digits())"
+
+ assert_python_failure('-X', 'int_max_str_digits', '-c', code)
+ assert_python_failure('-X', 'int_max_str_digits=foo', '-c', code)
+ assert_python_failure('-X', 'int_max_str_digits=100', '-c', code)
+ assert_python_failure('-X', 'int_max_str_digits', '-c', code,
+ PYTHONINTMAXSTRDIGITS='4000')
+
+ assert_python_failure('-c', code, PYTHONINTMAXSTRDIGITS='foo')
+ assert_python_failure('-c', code, PYTHONINTMAXSTRDIGITS='100')
+
+ def res2int(res):
+ out = res.out.strip().decode("utf-8")
+ return tuple(int(i) for i in out.split())
+
+ res = assert_python_ok('-c', code)
+ self.assertEqual(res2int(res), (-1, sys.get_int_max_str_digits()))
+ res = assert_python_ok('-X', 'int_max_str_digits=0', '-c', code)
+ self.assertEqual(res2int(res), (0, 0))
+ res = assert_python_ok('-X', 'int_max_str_digits=4000', '-c', code)
+ self.assertEqual(res2int(res), (4000, 4000))
+ res = assert_python_ok('-X', 'int_max_str_digits=100000', '-c', code)
+ self.assertEqual(res2int(res), (100000, 100000))
+
+ res = assert_python_ok('-c', code, PYTHONINTMAXSTRDIGITS='0')
+ self.assertEqual(res2int(res), (0, 0))
+ res = assert_python_ok('-c', code, PYTHONINTMAXSTRDIGITS='4000')
+ self.assertEqual(res2int(res), (4000, 4000))
+ res = assert_python_ok(
+ '-X', 'int_max_str_digits=6000', '-c', code,
+ PYTHONINTMAXSTRDIGITS='4000'
+ )
+ self.assertEqual(res2int(res), (6000, 6000))
+
@unittest.skipIf(interpreter_requires_environment(),
'Cannot run -I tests when PYTHON env vars are required.')
@@ -864,9 +910,10 @@ def test_tokenizer_error_with_stdin(self):
def test_decoding_error_at_the_end_of_the_line(self):
self.check_string(br"'\u1f'")
-def test_main():
- support.run_unittest(CmdLineTest, IgnoreEnvironmentTest, SyntaxErrorTests)
+
+def tearDownModule():
support.reap_children()
+
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
index af29c171d4282f..86bcafcf0e3cb3 100644
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -651,7 +651,7 @@ def test_syntaxerror_invalid_escape_sequence_multi_line(self):
stderr.splitlines()[-3:],
[ b' foo = """\\q"""',
b' ^^^^^^^^',
- b'SyntaxError: invalid escape sequence \\q'
+ b'SyntaxError: invalid escape sequence \'\\q\''
],
)
@@ -738,10 +738,27 @@ def test_nonexisting_script(self):
self.assertIn(": can't open file ", err)
self.assertNotEqual(proc.returncode, 0)
+ @unittest.skipUnless(os.path.exists('/dev/fd/0'), 'requires /dev/fd platform')
+ @unittest.skipIf(sys.platform.startswith("freebsd") and
+ os.stat("/dev").st_dev == os.stat("/dev/fd").st_dev,
+ "Requires fdescfs mounted on /dev/fd on FreeBSD")
+ def test_script_as_dev_fd(self):
+ # GH-87235: On macOS passing a non-trivial script to /dev/fd/N can cause
+ # problems because all open /dev/fd/N file descriptors share the same
+ # offset.
+ script = 'print("12345678912345678912345")'
+ with os_helper.temp_dir() as work_dir:
+ script_name = _make_test_script(work_dir, 'script.py', script)
+ with open(script_name, "r") as fp:
+ p = spawn_python(f"/dev/fd/{fp.fileno()}", close_fds=False, pass_fds=(0,1,2,fp.fileno()))
+ out, err = p.communicate()
+ self.assertEqual(out, b"12345678912345678912345\n")
+
-def test_main():
- support.run_unittest(CmdLineTest)
+
+def tearDownModule():
support.reap_children()
+
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py
index 6aaf04ca6dde5e..b9c4f8bdcb382e 100644
--- a/Lib/test/test_code.py
+++ b/Lib/test/test_code.py
@@ -135,7 +135,7 @@
except ImportError:
ctypes = None
from test.support import (run_doctest, run_unittest, cpython_only,
- check_impl_detail)
+ check_impl_detail, gc_collect)
def consts(t):
@@ -213,7 +213,7 @@ def func(): pass
CodeType = type(co)
# test code constructor
- return CodeType(co.co_argcount,
+ CodeType(co.co_argcount,
co.co_posonlyargcount,
co.co_kwonlyargcount,
co.co_nlocals,
@@ -343,6 +343,7 @@ def callback(code):
coderef = weakref.ref(f.__code__, callback)
self.assertTrue(bool(coderef()))
del f
+ gc_collect() # For PyPy or other GCs.
self.assertFalse(bool(coderef()))
self.assertTrue(self.called)
diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
index 243f002c4ecadd..4991330489d139 100644
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -1,5 +1,6 @@
import codecs
import html.entities
+import itertools
import sys
import unicodedata
import unittest
@@ -22,6 +23,18 @@ def handle(self, exc):
self.pos = len(exc.object)
return ("", oldpos)
+class RepeatedPosReturn:
+ def __init__(self, repl=""):
+ self.repl = repl
+ self.pos = 0
+ self.count = 0
+
+ def handle(self, exc):
+ if self.count > 0:
+ self.count -= 1
+ return (self.repl, self.pos)
+ return (self.repl, exc.end)
+
# A UnicodeEncodeError object with a bad start attribute
class BadStartUnicodeEncodeError(UnicodeEncodeError):
def __init__(self):
@@ -783,20 +796,104 @@ def test_lookup(self):
codecs.lookup_error("namereplace")
)
- def test_unencodablereplacement(self):
+ def test_encode_nonascii_replacement(self):
+ def handle(exc):
+ if isinstance(exc, UnicodeEncodeError):
+ return (repl, exc.end)
+ raise TypeError("don't know how to handle %r" % exc)
+ codecs.register_error("test.replacing", handle)
+
+ for enc, input, repl in (
+ ("ascii", "[¤]", "abc"),
+ ("iso-8859-1", "[€]", "½¾"),
+ ("iso-8859-15", "[¤]", "œŸ"),
+ ):
+ res = input.encode(enc, "test.replacing")
+ self.assertEqual(res, ("[" + repl + "]").encode(enc))
+
+ for enc, input, repl in (
+ ("utf-8", "[\udc80]", "\U0001f40d"),
+ ("utf-16", "[\udc80]", "\U0001f40d"),
+ ("utf-32", "[\udc80]", "\U0001f40d"),
+ ):
+ with self.subTest(encoding=enc):
+ with self.assertRaises(UnicodeEncodeError) as cm:
+ input.encode(enc, "test.replacing")
+ exc = cm.exception
+ self.assertEqual(exc.start, 1)
+ self.assertEqual(exc.end, 2)
+ self.assertEqual(exc.object, input)
+
+ def test_encode_unencodable_replacement(self):
def unencrepl(exc):
if isinstance(exc, UnicodeEncodeError):
- return ("\u4242", exc.end)
+ return (repl, exc.end)
else:
raise TypeError("don't know how to handle %r" % exc)
codecs.register_error("test.unencreplhandler", unencrepl)
- for enc in ("ascii", "iso-8859-1", "iso-8859-15"):
- self.assertRaises(
- UnicodeEncodeError,
- "\u4242".encode,
- enc,
- "test.unencreplhandler"
- )
+
+ for enc, input, repl in (
+ ("ascii", "[¤]", "½"),
+ ("iso-8859-1", "[€]", "œ"),
+ ("iso-8859-15", "[¤]", "½"),
+ ("utf-8", "[\udc80]", "\udcff"),
+ ("utf-16", "[\udc80]", "\udcff"),
+ ("utf-32", "[\udc80]", "\udcff"),
+ ):
+ with self.subTest(encoding=enc):
+ with self.assertRaises(UnicodeEncodeError) as cm:
+ input.encode(enc, "test.unencreplhandler")
+ exc = cm.exception
+ self.assertEqual(exc.start, 1)
+ self.assertEqual(exc.end, 2)
+ self.assertEqual(exc.object, input)
+
+ def test_encode_bytes_replacement(self):
+ def handle(exc):
+ if isinstance(exc, UnicodeEncodeError):
+ return (repl, exc.end)
+ raise TypeError("don't know how to handle %r" % exc)
+ codecs.register_error("test.replacing", handle)
+
+ # It works even if the bytes sequence is not decodable.
+ for enc, input, repl in (
+ ("ascii", "[¤]", b"\xbd\xbe"),
+ ("iso-8859-1", "[€]", b"\xbd\xbe"),
+ ("iso-8859-15", "[¤]", b"\xbd\xbe"),
+ ("utf-8", "[\udc80]", b"\xbd\xbe"),
+ ("utf-16le", "[\udc80]", b"\xbd\xbe"),
+ ("utf-16be", "[\udc80]", b"\xbd\xbe"),
+ ("utf-32le", "[\udc80]", b"\xbc\xbd\xbe\xbf"),
+ ("utf-32be", "[\udc80]", b"\xbc\xbd\xbe\xbf"),
+ ):
+ with self.subTest(encoding=enc):
+ res = input.encode(enc, "test.replacing")
+ self.assertEqual(res, "[".encode(enc) + repl + "]".encode(enc))
+
+ def test_encode_odd_bytes_replacement(self):
+ def handle(exc):
+ if isinstance(exc, UnicodeEncodeError):
+ return (repl, exc.end)
+ raise TypeError("don't know how to handle %r" % exc)
+ codecs.register_error("test.replacing", handle)
+
+ input = "[\udc80]"
+ # Tests in which the replacement bytestring contains not whole number
+ # of code units.
+ for enc, repl in (
+ *itertools.product(("utf-16le", "utf-16be"),
+ [b"a", b"abc"]),
+ *itertools.product(("utf-32le", "utf-32be"),
+ [b"a", b"ab", b"abc", b"abcde"]),
+ ):
+ with self.subTest(encoding=enc, repl=repl):
+ with self.assertRaises(UnicodeEncodeError) as cm:
+ input.encode(enc, "test.replacing")
+ exc = cm.exception
+ self.assertEqual(exc.start, 1)
+ self.assertEqual(exc.end, 2)
+ self.assertEqual(exc.object, input)
+ self.assertEqual(exc.reason, "surrogates not allowed")
def test_badregistercall(self):
# enhance coverage of:
@@ -940,6 +1037,68 @@ def __getitem__(self, key):
self.assertRaises(ValueError, codecs.charmap_encode, "\xff", err, D())
self.assertRaises(TypeError, codecs.charmap_encode, "\xff", err, {0xff: 300})
+ def test_decodehelper_bug36819(self):
+ handler = RepeatedPosReturn("x")
+ codecs.register_error("test.bug36819", handler.handle)
+
+ testcases = [
+ ("ascii", b"\xff"),
+ ("utf-8", b"\xff"),
+ ("utf-16be", b'\xdc\x80'),
+ ("utf-32be", b'\x00\x00\xdc\x80'),
+ ("iso-8859-6", b"\xff"),
+ ]
+ for enc, bad in testcases:
+ input = "abcd".encode(enc) + bad
+ with self.subTest(encoding=enc):
+ handler.count = 50
+ decoded = input.decode(enc, "test.bug36819")
+ self.assertEqual(decoded, 'abcdx' * 51)
+
+ def test_encodehelper_bug36819(self):
+ handler = RepeatedPosReturn()
+ codecs.register_error("test.bug36819", handler.handle)
+
+ input = "abcd\udc80"
+ encodings = ["ascii", "latin1", "utf-8", "utf-16", "utf-32"] # built-in
+ encodings += ["iso-8859-15"] # charmap codec
+ if sys.platform == 'win32':
+ encodings = ["mbcs", "oem"] # code page codecs
+
+ handler.repl = "\udcff"
+ for enc in encodings:
+ with self.subTest(encoding=enc):
+ handler.count = 50
+ with self.assertRaises(UnicodeEncodeError) as cm:
+ input.encode(enc, "test.bug36819")
+ exc = cm.exception
+ self.assertEqual(exc.start, 4)
+ self.assertEqual(exc.end, 5)
+ self.assertEqual(exc.object, input)
+ if sys.platform == "win32":
+ handler.count = 50
+ with self.assertRaises(UnicodeEncodeError) as cm:
+ codecs.code_page_encode(437, input, "test.bug36819")
+ exc = cm.exception
+ self.assertEqual(exc.start, 4)
+ self.assertEqual(exc.end, 5)
+ self.assertEqual(exc.object, input)
+
+ handler.repl = "x"
+ for enc in encodings:
+ with self.subTest(encoding=enc):
+ # The interpreter should segfault after a handful of attempts.
+ # 50 was chosen to try to ensure a segfault without a fix,
+ # but not OOM a machine with one.
+ handler.count = 50
+ encoded = input.encode(enc, "test.bug36819")
+ self.assertEqual(encoded.decode(enc), "abcdx" * 51)
+ if sys.platform == "win32":
+ handler.count = 50
+ encoded = codecs.code_page_encode(437, input, "test.bug36819")
+ self.assertEqual(encoded[0].decode(), "abcdx" * 51)
+ self.assertEqual(encoded[1], len(input))
+
def test_translatehelper(self):
# enhance coverage of:
# Objects/unicodeobject.c::unicode_encode_call_errorhandler()
diff --git a/Lib/test/test_codecencodings_iso2022.py b/Lib/test/test_codecencodings_iso2022.py
index 00ea1c39dd6fb6..027dbecc6134df 100644
--- a/Lib/test/test_codecencodings_iso2022.py
+++ b/Lib/test/test_codecencodings_iso2022.py
@@ -24,6 +24,52 @@ class Test_ISO2022_JP2(multibytecodec_support.TestBase, unittest.TestCase):
(b'ab\x1BNdef', 'replace', 'abdef'),
)
+class Test_ISO2022_JP3(multibytecodec_support.TestBase, unittest.TestCase):
+ encoding = 'iso2022_jp_3'
+ tstring = multibytecodec_support.load_teststring('iso2022_jp')
+ codectests = COMMON_CODEC_TESTS + (
+ (b'ab\x1BNdef', 'replace', 'ab\x1BNdef'),
+ (b'\x1B$(O\x2E\x23\x1B(B', 'strict', '\u3402' ),
+ (b'\x1B$(O\x2E\x22\x1B(B', 'strict', '\U0002000B' ),
+ (b'\x1B$(O\x24\x77\x1B(B', 'strict', '\u304B\u309A'),
+ (b'\x1B$(P\x21\x22\x1B(B', 'strict', '\u4E02' ),
+ (b'\x1B$(P\x7E\x76\x1B(B', 'strict', '\U0002A6B2' ),
+ ('\u3402', 'strict', b'\x1B$(O\x2E\x23\x1B(B'),
+ ('\U0002000B', 'strict', b'\x1B$(O\x2E\x22\x1B(B'),
+ ('\u304B\u309A', 'strict', b'\x1B$(O\x24\x77\x1B(B'),
+ ('\u4E02', 'strict', b'\x1B$(P\x21\x22\x1B(B'),
+ ('\U0002A6B2', 'strict', b'\x1B$(P\x7E\x76\x1B(B'),
+ (b'ab\x1B$(O\x2E\x21\x1B(Bdef', 'replace', 'ab\uFFFDdef'),
+ ('ab\u4FF1def', 'replace', b'ab?def'),
+ )
+ xmlcharnametest = (
+ '\xAB\u211C\xBB = \u2329\u1234\u232A',
+ b'\x1B$(O\x29\x28\x1B(Bℜ\x1B$(O\x29\x32\x1B(B = ⟨ሴ⟩'
+ )
+
+class Test_ISO2022_JP2004(multibytecodec_support.TestBase, unittest.TestCase):
+ encoding = 'iso2022_jp_2004'
+ tstring = multibytecodec_support.load_teststring('iso2022_jp')
+ codectests = COMMON_CODEC_TESTS + (
+ (b'ab\x1BNdef', 'replace', 'ab\x1BNdef'),
+ (b'\x1B$(Q\x2E\x23\x1B(B', 'strict', '\u3402' ),
+ (b'\x1B$(Q\x2E\x22\x1B(B', 'strict', '\U0002000B' ),
+ (b'\x1B$(Q\x24\x77\x1B(B', 'strict', '\u304B\u309A'),
+ (b'\x1B$(P\x21\x22\x1B(B', 'strict', '\u4E02' ),
+ (b'\x1B$(P\x7E\x76\x1B(B', 'strict', '\U0002A6B2' ),
+ ('\u3402', 'strict', b'\x1B$(Q\x2E\x23\x1B(B'),
+ ('\U0002000B', 'strict', b'\x1B$(Q\x2E\x22\x1B(B'),
+ ('\u304B\u309A', 'strict', b'\x1B$(Q\x24\x77\x1B(B'),
+ ('\u4E02', 'strict', b'\x1B$(P\x21\x22\x1B(B'),
+ ('\U0002A6B2', 'strict', b'\x1B$(P\x7E\x76\x1B(B'),
+ (b'ab\x1B$(Q\x2E\x21\x1B(Bdef', 'replace', 'ab\u4FF1def'),
+ ('ab\u4FF1def', 'replace', b'ab\x1B$(Q\x2E\x21\x1B(Bdef'),
+ )
+ xmlcharnametest = (
+ '\xAB\u211C\xBB = \u2329\u1234\u232A',
+ b'\x1B$(Q\x29\x28\x1B(Bℜ\x1B$(Q\x29\x32\x1B(B = ⟨ሴ⟩'
+ )
+
class Test_ISO2022_KR(multibytecodec_support.TestBase, unittest.TestCase):
encoding = 'iso2022_kr'
tstring = multibytecodec_support.load_teststring('iso2022_kr')
diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
index 328a47b2e37669..240756726133fc 100644
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -32,7 +32,7 @@ def check(input, expect):
# On small versions of Windows like Windows IoT or Windows Nano Server not all codepages are present
def is_code_page_present(cp):
from ctypes import POINTER, WINFUNCTYPE, WinDLL
- from ctypes.wintypes import BOOL, UINT, BYTE, WCHAR, UINT, DWORD
+ from ctypes.wintypes import BOOL, BYTE, WCHAR, UINT, DWORD
MAX_LEADBYTES = 12 # 5 ranges, 2 bytes ea., 0 term.
MAX_DEFAULTCHAR = 2 # single or double byte
@@ -114,7 +114,7 @@ def check_partial(self, input, partialresults):
q = Queue(b"")
r = codecs.getreader(self.encoding)(q)
result = ""
- for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
+ for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True):
q.write(bytes([c]))
result += r.read()
self.assertEqual(result, partialresult)
@@ -125,7 +125,7 @@ def check_partial(self, input, partialresults):
# do the check again, this time using an incremental decoder
d = codecs.getincrementaldecoder(self.encoding)()
result = ""
- for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
+ for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True):
result += d.decode(bytes([c]))
self.assertEqual(result, partialresult)
# check that there's nothing left in the buffers
@@ -135,7 +135,7 @@ def check_partial(self, input, partialresults):
# Check whether the reset method works properly
d.reset()
result = ""
- for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
+ for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True):
result += d.decode(bytes([c]))
self.assertEqual(result, partialresult)
# check that there's nothing left in the buffers
@@ -705,7 +705,8 @@ def test_decoder_state(self):
"spamspam", self.spambe)
def test_bug691291(self):
- # Files are always opened in binary mode, even if no binary mode was
+ # If encoding is not None, then
+ # files are always opened in binary mode, even if no binary mode was
# specified. This means that no automatic conversion of '\n' is done
# on reading and writing.
s1 = 'Hello\r\nworld\r\n'
@@ -1534,6 +1535,12 @@ def test_builtin_encode(self):
self.assertEqual("pyth\xf6n.org".encode("idna"), b"xn--pythn-mua.org")
self.assertEqual("pyth\xf6n.org.".encode("idna"), b"xn--pythn-mua.org.")
+ def test_builtin_decode_length_limit(self):
+ with self.assertRaisesRegex(UnicodeError, "too long"):
+ (b"xn--016c"+b"a"*1100).decode("idna")
+ with self.assertRaisesRegex(UnicodeError, "too long"):
+ (b"xn--016c"+b"a"*70).decode("idna")
+
def test_stream(self):
r = codecs.getreader("idna")(io.BytesIO(b"abc"))
r.read(3)
@@ -2341,7 +2348,11 @@ def test_unicode_escape(self):
(r"\x5c\x55\x30\x30\x31\x31\x30\x30\x30\x30", 10))
-class UnicodeEscapeTest(unittest.TestCase):
+class UnicodeEscapeTest(ReadTest, unittest.TestCase):
+ encoding = "unicode-escape"
+
+ test_lone_surrogates = None
+
def test_empty(self):
self.assertEqual(codecs.unicode_escape_encode(""), (b"", 0))
self.assertEqual(codecs.unicode_escape_decode(b""), ("", 0))
@@ -2428,8 +2439,50 @@ def test_decode_errors(self):
self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10))
self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10))
+ def test_partial(self):
+ self.check_partial(
+ "\x00\t\n\r\\\xff\uffff\U00010000",
+ [
+ '',
+ '',
+ '',
+ '\x00',
+ '\x00',
+ '\x00\t',
+ '\x00\t',
+ '\x00\t\n',
+ '\x00\t\n',
+ '\x00\t\n\r',
+ '\x00\t\n\r',
+ '\x00\t\n\r\\',
+ '\x00\t\n\r\\',
+ '\x00\t\n\r\\',
+ '\x00\t\n\r\\',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff\U00010000',
+ ]
+ )
+
+class RawUnicodeEscapeTest(ReadTest, unittest.TestCase):
+ encoding = "raw-unicode-escape"
+
+ test_lone_surrogates = None
-class RawUnicodeEscapeTest(unittest.TestCase):
def test_empty(self):
self.assertEqual(codecs.raw_unicode_escape_encode(""), (b"", 0))
self.assertEqual(codecs.raw_unicode_escape_decode(b""), ("", 0))
@@ -2478,6 +2531,35 @@ def test_decode_errors(self):
self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10))
self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10))
+ def test_partial(self):
+ self.check_partial(
+ "\x00\t\n\r\\\xff\uffff\U00010000",
+ [
+ '\x00',
+ '\x00\t',
+ '\x00\t\n',
+ '\x00\t\n\r',
+ '\x00\t\n\r',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff',
+ '\x00\t\n\r\\\xff\uffff\U00010000',
+ ]
+ )
+
class EscapeEncodeTest(unittest.TestCase):
diff --git a/Lib/test/test_codeop.py b/Lib/test/test_codeop.py
index 07b46afbe1bd89..133096d25a44bc 100644
--- a/Lib/test/test_codeop.py
+++ b/Lib/test/test_codeop.py
@@ -182,21 +182,21 @@ def test_incomplete(self):
ai("from a import (b,c")
ai("from a import (b,c,")
- ai("[");
- ai("[a");
- ai("[a,");
- ai("[a,b");
- ai("[a,b,");
-
- ai("{");
- ai("{a");
- ai("{a:");
- ai("{a:b");
- ai("{a:b,");
- ai("{a:b,c");
- ai("{a:b,c:");
- ai("{a:b,c:d");
- ai("{a:b,c:d,");
+ ai("[")
+ ai("[a")
+ ai("[a,")
+ ai("[a,b")
+ ai("[a,b,")
+
+ ai("{")
+ ai("{a")
+ ai("{a:")
+ ai("{a:b")
+ ai("{a:b,")
+ ai("{a:b,c")
+ ai("{a:b,c:")
+ ai("{a:b,c:d")
+ ai("{a:b,c:d,")
ai("a(")
ai("a(b")
@@ -321,6 +321,26 @@ def test_warning(self):
warnings.simplefilter('error', SyntaxWarning)
compile_command('1 is 1', symbol='exec')
+ # Check DeprecationWarning treated as an SyntaxError
+ with warnings.catch_warnings(), self.assertRaises(SyntaxError):
+ warnings.simplefilter('error', DeprecationWarning)
+ compile_command(r"'\e'", symbol='exec')
+
+ def test_incomplete_warning(self):
+ with warnings.catch_warnings(record=True) as w:
+ warnings.simplefilter('always')
+ self.assertIncomplete("'\\e' + (")
+ self.assertEqual(w, [])
+
+ def test_invalid_warning(self):
+ with warnings.catch_warnings(record=True) as w:
+ warnings.simplefilter('always')
+ self.assertInvalid("'\\e' 1")
+ self.assertEqual(len(w), 1)
+ self.assertEqual(w[0].category, DeprecationWarning)
+ self.assertRegex(str(w[0].message), 'invalid escape sequence')
+ self.assertEqual(w[0].filename, '')
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
index f98048b34a7a1d..f5af55a364b1c8 100644
--- a/Lib/test/test_collections.py
+++ b/Lib/test/test_collections.py
@@ -668,6 +668,7 @@ class Point(namedtuple('_Point', ['x', 'y'])):
a.w = 5
self.assertEqual(a.__dict__, {'w': 5})
+ @support.cpython_only
def test_field_descriptor(self):
Point = namedtuple('Point', 'x y')
p = Point(11, 22)
@@ -787,6 +788,8 @@ def throw(self, typ, val=None, tb=None):
def __await__(self):
yield
+ self.validate_abstract_methods(Awaitable, '__await__')
+
non_samples = [None, int(), gen(), object()]
for x in non_samples:
self.assertNotIsInstance(x, Awaitable)
@@ -837,6 +840,8 @@ def throw(self, typ, val=None, tb=None):
def __await__(self):
yield
+ self.validate_abstract_methods(Coroutine, '__await__', 'send', 'throw')
+
non_samples = [None, int(), gen(), object(), Bar()]
for x in non_samples:
self.assertNotIsInstance(x, Coroutine)
@@ -1579,6 +1584,7 @@ def __len__(self):
containers = [
seq,
ItemsView({1: nan, 2: obj}),
+ KeysView({1: nan, 2: obj}),
ValuesView({1: nan, 2: obj})
]
for container in containers:
@@ -1594,7 +1600,7 @@ def assertSameSet(self, s1, s2):
self.assertSetEqual(set(s1), set(s2))
def test_Set_from_iterable(self):
- """Verify _from_iterable overriden to an instance method works."""
+ """Verify _from_iterable overridden to an instance method works."""
class SetUsingInstanceFromIterable(MutableSet):
def __init__(self, values, created_by):
if not created_by:
@@ -1801,6 +1807,18 @@ def __repr__(self):
self.assertTrue(f1 != l1)
self.assertTrue(f1 != l2)
+ def test_Set_hash_matches_frozenset(self):
+ sets = [
+ {}, {1}, {None}, {-1}, {0.0}, {"abc"}, {1, 2, 3},
+ {10**100, 10**101}, {"a", "b", "ab", ""}, {False, True},
+ {object(), object(), object()}, {float("nan")}, {frozenset()},
+ {*range(1000)}, {*range(1000)} - {100, 200, 300},
+ {*range(sys.maxsize - 10, sys.maxsize + 10)},
+ ]
+ for s in sets:
+ fs = frozenset(s)
+ self.assertEqual(hash(fs), Set._hash(fs), msg=s)
+
def test_Mapping(self):
for sample in [dict]:
self.assertIsInstance(sample(), Mapping)
@@ -1830,6 +1848,8 @@ def test_MutableMapping_subclass(self):
mymap['red'] = 5
self.assertIsInstance(mymap.keys(), Set)
self.assertIsInstance(mymap.keys(), KeysView)
+ self.assertIsInstance(mymap.values(), Collection)
+ self.assertIsInstance(mymap.values(), ValuesView)
self.assertIsInstance(mymap.items(), Set)
self.assertIsInstance(mymap.items(), ItemsView)
@@ -1905,6 +1925,7 @@ def test_ByteString(self):
self.assertFalse(issubclass(sample, ByteString))
self.assertNotIsInstance(memoryview(b""), ByteString)
self.assertFalse(issubclass(memoryview, ByteString))
+ self.validate_abstract_methods(ByteString, '__getitem__', '__len__')
def test_MutableSequence(self):
for sample in [tuple, str, bytes]:
diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py
index d40347c3c6bb3e..d329cb1708093e 100644
--- a/Lib/test/test_compile.py
+++ b/Lib/test/test_compile.py
@@ -162,7 +162,7 @@ def test_literals_with_leading_zeroes(self):
for arg in ["077787", "0xj", "0x.", "0e", "090000000000000",
"080000000000000", "000000000000009", "000000000000008",
"0b42", "0BADCAFE", "0o123456789", "0b1.1", "0o4.2",
- "0b101j2", "0o153j2", "0b100e1", "0o777e1", "0777",
+ "0b101j", "0o153j", "0b100e1", "0o777e1", "0777",
"000777", "000000000000007"]:
self.assertRaises(SyntaxError, eval, arg)
@@ -191,6 +191,19 @@ def test_literals_with_leading_zeroes(self):
self.assertEqual(eval("0o777"), 511)
self.assertEqual(eval("-0o0000010"), -8)
+ def test_int_literals_too_long(self):
+ n = 3000
+ source = f"a = 1\nb = 2\nc = {'3'*n}\nd = 4"
+ with support.adjust_int_max_str_digits(n):
+ compile(source, "", "exec") # no errors.
+ with support.adjust_int_max_str_digits(n-1):
+ with self.assertRaises(SyntaxError) as err_ctx:
+ compile(source, "", "exec")
+ exc = err_ctx.exception
+ self.assertEqual(exc.lineno, 3)
+ self.assertIn('Exceeds the limit ', str(exc))
+ self.assertIn(' Consider hexadecimal ', str(exc))
+
def test_unary_minus(self):
# Verify treatment of unary minus on negative numbers SF bug #660455
if sys.maxsize == 2147483647:
@@ -502,6 +515,7 @@ def test_single_statement(self):
self.compile_single("if x:\n f(x)")
self.compile_single("if x:\n f(x)\nelse:\n g(x)")
self.compile_single("class T:\n pass")
+ self.compile_single("c = '''\na=1\nb=2\nc=3\n'''")
def test_bad_single_statement(self):
self.assertInvalidSingle('1\n2')
@@ -512,6 +526,7 @@ def test_bad_single_statement(self):
self.assertInvalidSingle('f()\n# blah\nblah()')
self.assertInvalidSingle('f()\nxy # blah\nblah()')
self.assertInvalidSingle('x = 5 # comment\nx = 6\n')
+ self.assertInvalidSingle("c = '''\nd=1\n'''\na = 1\n\nb = 2\n")
def test_particularly_evil_undecodable(self):
# Issue 24022
@@ -648,6 +663,17 @@ def test_merge_code_attrs(self):
self.assertIs(f1.__code__.co_linetable, f2.__code__.co_linetable)
self.assertIs(f1.__code__.co_code, f2.__code__.co_code)
+ # Stripping unused constants is not a strict requirement for the
+ # Python semantics, it's a more an implementation detail.
+ @support.cpython_only
+ def test_strip_unused_consts(self):
+ # Python 3.10rc1 appended None to co_consts when None is not used
+ # at all. See bpo-45056.
+ def f1():
+ "docstring"
+ return 42
+ self.assertEqual(f1.__code__.co_consts, ("docstring", 42))
+
# This is a regression test for a CPython specific peephole optimizer
# implementation bug present in a few releases. It's assertion verifies
# that peephole optimization was actually done though that isn't an
@@ -895,6 +921,31 @@ def aug_store_attr():
for (_, _, line) in func.__code__.co_lines() ]
self.assertEqual(lines, code_lines)
+ def test_line_number_genexp(self):
+
+ def return_genexp():
+ return (1
+ for
+ x
+ in
+ y)
+ genexp_lines = [None, 1, 3, 1]
+
+ genexp_code = return_genexp.__code__.co_consts[1]
+ code_lines = [ None if line is None else line-return_genexp.__code__.co_firstlineno
+ for (_, _, line) in genexp_code.co_lines() ]
+ self.assertEqual(genexp_lines, code_lines)
+
+ def test_line_number_implicit_return_after_async_for(self):
+
+ async def test(aseq):
+ async for i in aseq:
+ body
+
+ expected_lines = [None, 1, 2, 1]
+ code_lines = [ None if line is None else line-test.__code__.co_firstlineno
+ for (_, _, line) in test.__code__.co_lines() ]
+ self.assertEqual(expected_lines, code_lines)
def test_big_dict_literal(self):
# The compiler has a flushing point in "compiler_dict" that calls compiles
@@ -933,6 +984,38 @@ def if_else_break():
elif instr.opname in HANDLED_JUMPS:
self.assertNotEqual(instr.arg, (line + 1)*INSTR_SIZE)
+ def test_compare_positions(self):
+ for opname, op in [
+ ("COMPARE_OP", "<"),
+ ("COMPARE_OP", "<="),
+ ("COMPARE_OP", ">"),
+ ("COMPARE_OP", ">="),
+ ("CONTAINS_OP", "in"),
+ ("CONTAINS_OP", "not in"),
+ ("IS_OP", "is"),
+ ("IS_OP", "is not"),
+ ]:
+ expr = f'a {op} b {op} c'
+ expected_lines = 2 * [2]
+ for source in [
+ f"\\\n{expr}", f'if \\\n{expr}: x', f"x if \\\n{expr} else y"
+ ]:
+ code = compile(source, "", "exec")
+ all_lines = (
+ line
+ for start, stop, line in code.co_lines()
+ for _ in range(start, stop, 2)
+ )
+ actual_lines = [
+ line
+ for instruction, line in zip(
+ dis.get_instructions(code), all_lines, strict=True
+ )
+ if instruction.opname == opname
+ ]
+ with self.subTest(source):
+ self.assertEqual(actual_lines, expected_lines)
+
class TestExpressionStackSize(unittest.TestCase):
# These tests check that the computed stack size for a code object
@@ -985,7 +1068,7 @@ def test_func_kwargs(self):
kwargs = (f'a{i}=x' for i in range(self.N))
self.check_stack_size("f(" + ", ".join(kwargs) + ")")
- def test_func_args(self):
+ def test_meth_args(self):
self.check_stack_size("o.m(" + "x, " * self.N + ")")
def test_meth_kwargs(self):
@@ -997,6 +1080,12 @@ def test_func_and(self):
code += " x and x\n" * self.N
self.check_stack_size(code)
+ def test_stack_3050(self):
+ M = 3050
+ code = "x," * M + "=t"
+ # This raised on 3.10.0 to 3.10.5
+ compile(code, "", "single")
+
class TestStackSizeStability(unittest.TestCase):
# Check that repeating certain snippets doesn't increase the stack size
diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py
index 96a0f8f729aa4e..554e2b27eea2d7 100644
--- a/Lib/test/test_compileall.py
+++ b/Lib/test/test_compileall.py
@@ -3,7 +3,6 @@
import filecmp
import importlib.util
import io
-import itertools
import os
import pathlib
import py_compile
@@ -29,9 +28,8 @@
from test import support
from test.support import os_helper
from test.support import script_helper
-
-from .test_py_compile import without_source_date_epoch
-from .test_py_compile import SourceDateEpochTestMeta
+from test.test_py_compile import without_source_date_epoch
+from test.test_py_compile import SourceDateEpochTestMeta
def get_pyc(script, opt):
@@ -80,9 +78,30 @@ def timestamp_metadata(self):
with open(self.bc_path, 'rb') as file:
data = file.read(12)
mtime = int(os.stat(self.source_path).st_mtime)
- compare = struct.pack('<4sll', importlib.util.MAGIC_NUMBER, 0, mtime)
+ compare = struct.pack('<4sLL', importlib.util.MAGIC_NUMBER, 0,
+ mtime & 0xFFFF_FFFF)
return data, compare
+ def test_year_2038_mtime_compilation(self):
+ # Test to make sure we can handle mtimes larger than what a 32-bit
+ # signed number can hold as part of bpo-34990
+ try:
+ os.utime(self.source_path, (2**32 - 1, 2**32 - 1))
+ except (OverflowError, OSError):
+ self.skipTest("filesystem doesn't support timestamps near 2**32")
+ with contextlib.redirect_stdout(io.StringIO()):
+ self.assertTrue(compileall.compile_file(self.source_path))
+
+ def test_larger_than_32_bit_times(self):
+ # This is similar to the test above but we skip it if the OS doesn't
+ # support modification times larger than 32-bits.
+ try:
+ os.utime(self.source_path, (2**35, 2**35))
+ except (OverflowError, OSError):
+ self.skipTest("filesystem doesn't support large timestamps")
+ with contextlib.redirect_stdout(io.StringIO()):
+ self.assertTrue(compileall.compile_file(self.source_path))
+
def recreation_check(self, metadata):
"""Check that compileall recreates bytecode when the new metadata is
used."""
@@ -101,7 +120,7 @@ def recreation_check(self, metadata):
def test_mtime(self):
# Test a change in mtime leads to a new .pyc.
- self.recreation_check(struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
+ self.recreation_check(struct.pack('<4sLL', importlib.util.MAGIC_NUMBER,
0, 1))
def test_magic_number(self):
@@ -148,6 +167,20 @@ def test_compile_file_pathlike_ddir(self):
quiet=2))
self.assertTrue(os.path.isfile(self.bc_path))
+ def test_compile_file_pathlike_stripdir(self):
+ self.assertFalse(os.path.isfile(self.bc_path))
+ self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path),
+ stripdir=pathlib.Path('stripdir_path'),
+ quiet=2))
+ self.assertTrue(os.path.isfile(self.bc_path))
+
+ def test_compile_file_pathlike_prependdir(self):
+ self.assertFalse(os.path.isfile(self.bc_path))
+ self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path),
+ prependdir=pathlib.Path('prependdir_path'),
+ quiet=2))
+ self.assertTrue(os.path.isfile(self.bc_path))
+
def test_compile_path(self):
with test.test_importlib.util.import_state(path=[self.directory]):
self.assertTrue(compileall.compile_path(quiet=2))
@@ -169,6 +202,14 @@ def test_no_pycache_in_non_package(self):
compileall.compile_file(data_file)
self.assertFalse(os.path.exists(os.path.join(data_dir, '__pycache__')))
+
+ def test_compile_file_encoding_fallback(self):
+ # Bug 44666 reported that compile_file failed when sys.stdout.encoding is None
+ self.add_bad_source_file()
+ with contextlib.redirect_stdout(io.StringIO()):
+ self.assertFalse(compileall.compile_file(self.bad_source_path))
+
+
def test_optimize(self):
# make sure compiling with different optimization settings than the
# interpreter's creates the correct file names
@@ -192,6 +233,20 @@ def test_compile_dir_pathlike(self):
self.assertRegex(line, r'Listing ([^WindowsPath|PosixPath].*)')
self.assertTrue(os.path.isfile(self.bc_path))
+ def test_compile_dir_pathlike_stripdir(self):
+ self.assertFalse(os.path.isfile(self.bc_path))
+ self.assertTrue(compileall.compile_dir(pathlib.Path(self.directory),
+ stripdir=pathlib.Path('stripdir_path'),
+ quiet=2))
+ self.assertTrue(os.path.isfile(self.bc_path))
+
+ def test_compile_dir_pathlike_prependdir(self):
+ self.assertFalse(os.path.isfile(self.bc_path))
+ self.assertTrue(compileall.compile_dir(pathlib.Path(self.directory),
+ prependdir=pathlib.Path('prependdir_path'),
+ quiet=2))
+ self.assertTrue(os.path.isfile(self.bc_path))
+
@skipUnless(_have_multiprocessing, "requires multiprocessing")
@mock.patch('concurrent.futures.ProcessPoolExecutor')
def test_compile_pool_called(self, pool_mock):
@@ -430,31 +485,29 @@ def test_error(self):
class CommandLineTestsBase:
"""Test compileall's CLI."""
- @classmethod
- def setUpClass(cls):
- for path in filter(os.path.isdir, sys.path):
- directory_created = False
- directory = pathlib.Path(path) / '__pycache__'
- path = directory / 'test.try'
- try:
- if not directory.is_dir():
- directory.mkdir()
- directory_created = True
- path.write_text('# for test_compileall', encoding="utf-8")
- except OSError:
- sys_path_writable = False
- break
- finally:
- os_helper.unlink(str(path))
- if directory_created:
- directory.rmdir()
- else:
- sys_path_writable = True
- cls._sys_path_writable = sys_path_writable
-
- def _skip_if_sys_path_not_writable(self):
- if not self._sys_path_writable:
- raise unittest.SkipTest('not all entries on sys.path are writable')
+ def setUp(self):
+ self.directory = tempfile.mkdtemp()
+ self.addCleanup(os_helper.rmtree, self.directory)
+ self.pkgdir = os.path.join(self.directory, 'foo')
+ os.mkdir(self.pkgdir)
+ self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__')
+ # Create the __init__.py and a package module.
+ self.initfn = script_helper.make_script(self.pkgdir, '__init__', '')
+ self.barfn = script_helper.make_script(self.pkgdir, 'bar', '')
+
+ @contextlib.contextmanager
+ def temporary_pycache_prefix(self):
+ """Adjust and restore sys.pycache_prefix."""
+ old_prefix = sys.pycache_prefix
+ new_prefix = os.path.join(self.directory, '__testcache__')
+ try:
+ sys.pycache_prefix = new_prefix
+ yield {
+ 'PYTHONPATH': self.directory,
+ 'PYTHONPYCACHEPREFIX': new_prefix,
+ }
+ finally:
+ sys.pycache_prefix = old_prefix
def _get_run_args(self, args):
return [*support.optim_args_from_interpreter_flags(),
@@ -482,49 +535,39 @@ def assertNotCompiled(self, fn):
path = importlib.util.cache_from_source(fn)
self.assertFalse(os.path.exists(path))
- def setUp(self):
- self.directory = tempfile.mkdtemp()
- self.addCleanup(os_helper.rmtree, self.directory)
- self.pkgdir = os.path.join(self.directory, 'foo')
- os.mkdir(self.pkgdir)
- self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__')
- # Create the __init__.py and a package module.
- self.initfn = script_helper.make_script(self.pkgdir, '__init__', '')
- self.barfn = script_helper.make_script(self.pkgdir, 'bar', '')
-
def test_no_args_compiles_path(self):
# Note that -l is implied for the no args case.
- self._skip_if_sys_path_not_writable()
bazfn = script_helper.make_script(self.directory, 'baz', '')
- self.assertRunOK(PYTHONPATH=self.directory)
- self.assertCompiled(bazfn)
- self.assertNotCompiled(self.initfn)
- self.assertNotCompiled(self.barfn)
+ with self.temporary_pycache_prefix() as env:
+ self.assertRunOK(**env)
+ self.assertCompiled(bazfn)
+ self.assertNotCompiled(self.initfn)
+ self.assertNotCompiled(self.barfn)
@without_source_date_epoch # timestamp invalidation test
def test_no_args_respects_force_flag(self):
- self._skip_if_sys_path_not_writable()
bazfn = script_helper.make_script(self.directory, 'baz', '')
- self.assertRunOK(PYTHONPATH=self.directory)
- pycpath = importlib.util.cache_from_source(bazfn)
+ with self.temporary_pycache_prefix() as env:
+ self.assertRunOK(**env)
+ pycpath = importlib.util.cache_from_source(bazfn)
# Set atime/mtime backward to avoid file timestamp resolution issues
os.utime(pycpath, (time.time()-60,)*2)
mtime = os.stat(pycpath).st_mtime
# Without force, no recompilation
- self.assertRunOK(PYTHONPATH=self.directory)
+ self.assertRunOK(**env)
mtime2 = os.stat(pycpath).st_mtime
self.assertEqual(mtime, mtime2)
# Now force it.
- self.assertRunOK('-f', PYTHONPATH=self.directory)
+ self.assertRunOK('-f', **env)
mtime2 = os.stat(pycpath).st_mtime
self.assertNotEqual(mtime, mtime2)
def test_no_args_respects_quiet_flag(self):
- self._skip_if_sys_path_not_writable()
script_helper.make_script(self.directory, 'baz', '')
- noisy = self.assertRunOK(PYTHONPATH=self.directory)
+ with self.temporary_pycache_prefix() as env:
+ noisy = self.assertRunOK(**env)
self.assertIn(b'Listing ', noisy)
- quiet = self.assertRunOK('-q', PYTHONPATH=self.directory)
+ quiet = self.assertRunOK('-q', **env)
self.assertNotIn(b'Listing ', quiet)
# Ensure that the default behavior of compileall's CLI is to create
diff --git a/Lib/test/test_complex.py b/Lib/test/test_complex.py
index af39ee878dc913..c9fd6a5c0a3bbe 100644
--- a/Lib/test/test_complex.py
+++ b/Lib/test/test_complex.py
@@ -1,4 +1,5 @@
import unittest
+import sys
from test import support
from test.test_grammar import (VALID_UNDERSCORE_LITERALS,
INVALID_UNDERSCORE_LITERALS)
@@ -248,6 +249,54 @@ def test_pow(self):
b = 5.1+2.3j
self.assertRaises(ValueError, pow, a, b, 0)
+ # Check some boundary conditions; some of these used to invoke
+ # undefined behaviour (https://bugs.python.org/issue44698). We're
+ # not actually checking the results of these operations, just making
+ # sure they don't crash (for example when using clang's
+ # UndefinedBehaviourSanitizer).
+ values = (sys.maxsize, sys.maxsize+1, sys.maxsize-1,
+ -sys.maxsize, -sys.maxsize+1, -sys.maxsize+1)
+ for real in values:
+ for imag in values:
+ with self.subTest(real=real, imag=imag):
+ c = complex(real, imag)
+ try:
+ c ** real
+ except OverflowError:
+ pass
+ try:
+ c ** c
+ except OverflowError:
+ pass
+
+ def test_pow_with_small_integer_exponents(self):
+ # Check that small integer exponents are handled identically
+ # regardless of their type.
+ values = [
+ complex(5.0, 12.0),
+ complex(5.0e100, 12.0e100),
+ complex(-4.0, INF),
+ complex(INF, 0.0),
+ ]
+ exponents = [-19, -5, -3, -2, -1, 0, 1, 2, 3, 5, 19]
+ for value in values:
+ for exponent in exponents:
+ with self.subTest(value=value, exponent=exponent):
+ try:
+ int_pow = value**exponent
+ except OverflowError:
+ int_pow = "overflow"
+ try:
+ float_pow = value**float(exponent)
+ except OverflowError:
+ float_pow = "overflow"
+ try:
+ complex_pow = value**complex(exponent)
+ except OverflowError:
+ complex_pow = "overflow"
+ self.assertEqual(str(float_pow), str(int_pow))
+ self.assertEqual(str(complex_pow), str(int_pow))
+
def test_boolcontext(self):
for i in range(100):
self.assertTrue(complex(random() + 1e-6, random() + 1e-6))
@@ -257,15 +306,10 @@ def test_conjugate(self):
self.assertClose(complex(5.3, 9.8).conjugate(), 5.3-9.8j)
def test_constructor(self):
- class OS:
- def __init__(self, value): self.value = value
- def __complex__(self): return self.value
- class NS(object):
+ class NS:
def __init__(self, value): self.value = value
def __complex__(self): return self.value
- self.assertEqual(complex(OS(1+10j)), 1+10j)
self.assertEqual(complex(NS(1+10j)), 1+10j)
- self.assertRaises(TypeError, complex, OS(None))
self.assertRaises(TypeError, complex, NS(None))
self.assertRaises(TypeError, complex, {})
self.assertRaises(TypeError, complex, NS(1.5))
@@ -728,8 +772,6 @@ def test_format(self):
self.assertEqual(format(complex(INF, 1), 'F'), 'INF+1.000000j')
self.assertEqual(format(complex(INF, -1), 'F'), 'INF-1.000000j')
-def test_main():
- support.run_unittest(ComplexTest)
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py
index 99651f5f4ed4da..6eae8c9cce3854 100644
--- a/Lib/test/test_concurrent_futures.py
+++ b/Lib/test/test_concurrent_futures.py
@@ -14,6 +14,7 @@
from logging.handlers import QueueHandler
import os
import queue
+import signal
import sys
import threading
import time
@@ -26,10 +27,16 @@
PENDING, RUNNING, CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED, Future,
BrokenExecutor)
from concurrent.futures.process import BrokenProcessPool, _check_system_limits
-from multiprocessing import get_context
import multiprocessing.process
import multiprocessing.util
+import multiprocessing as mp
+
+
+if support.check_sanitizer(address=True, memory=True):
+ # bpo-46633: Skip the test because it is too slow when Python is built
+ # with ASAN/MSAN: between 5 and 20 minutes on GitHub Actions.
+ raise unittest.SkipTest("test too slow on ASAN/MSAN build")
def create_future(state=PENDING, exception=None, result=None):
@@ -125,7 +132,6 @@ def setUp(self):
self.executor = self.executor_type(
max_workers=self.worker_count,
**self.executor_kwargs)
- self._prime_executor()
def tearDown(self):
self.executor.shutdown(wait=True)
@@ -139,15 +145,7 @@ def tearDown(self):
super().tearDown()
def get_context(self):
- return get_context(self.ctx)
-
- def _prime_executor(self):
- # Make sure that the executor is ready to do work before running the
- # tests. This should reduce the probability of timeouts in the tests.
- futures = [self.executor.submit(time.sleep, 0.1)
- for _ in range(self.worker_count)]
- for f in futures:
- f.result()
+ return mp.get_context(self.ctx)
class ThreadPoolMixin(ExecutorMixin):
@@ -270,9 +268,6 @@ def test_initializer(self):
with self.assertRaises(BrokenExecutor):
self.executor.submit(get_init_status)
- def _prime_executor(self):
- pass
-
@contextlib.contextmanager
def _assert_logged(self, msg):
if self.log_queue is not None:
@@ -359,14 +354,14 @@ def test_hang_issue12364(self):
f.result()
def test_cancel_futures(self):
- executor = self.executor_type(max_workers=3)
- fs = [executor.submit(time.sleep, .1) for _ in range(50)]
- executor.shutdown(cancel_futures=True)
+ assert self.worker_count <= 5, "test needs few workers"
+ fs = [self.executor.submit(time.sleep, .1) for _ in range(50)]
+ self.executor.shutdown(cancel_futures=True)
# We can't guarantee the exact number of cancellations, but we can
- # guarantee that *some* were cancelled. With setting max_workers to 3,
- # most of the submitted futures should have been cancelled.
+ # guarantee that *some* were cancelled. With few workers, many of
+ # the submitted futures should have been cancelled.
cancelled = [fut for fut in fs if fut.cancelled()]
- self.assertTrue(len(cancelled) >= 35, msg=f"{len(cancelled)=}")
+ self.assertGreater(len(cancelled), 20)
# Ensure the other futures were able to finish.
# Use "not fut.cancelled()" instead of "fut.done()" to include futures
@@ -379,33 +374,59 @@ def test_cancel_futures(self):
# Similar to the number of cancelled futures, we can't guarantee the
# exact number that completed. But, we can guarantee that at least
# one finished.
- self.assertTrue(len(others) > 0, msg=f"{len(others)=}")
+ self.assertGreater(len(others), 0)
- def test_hang_issue39205(self):
+ def test_hang_gh83386(self):
"""shutdown(wait=False) doesn't hang at exit with running futures.
- See https://bugs.python.org/issue39205.
+ See https://github.com/python/cpython/issues/83386.
"""
if self.executor_type == futures.ProcessPoolExecutor:
raise unittest.SkipTest(
- "Hangs due to https://bugs.python.org/issue39205")
+ "Hangs, see https://github.com/python/cpython/issues/83386")
rc, out, err = assert_python_ok('-c', """if True:
from concurrent.futures import {executor_type}
from test.test_concurrent_futures import sleep_and_print
if __name__ == "__main__":
+ if {context!r}: multiprocessing.set_start_method({context!r})
t = {executor_type}(max_workers=3)
t.submit(sleep_and_print, 1.0, "apple")
t.shutdown(wait=False)
- """.format(executor_type=self.executor_type.__name__))
+ """.format(executor_type=self.executor_type.__name__,
+ context=getattr(self, 'ctx', None)))
self.assertFalse(err)
self.assertEqual(out.strip(), b"apple")
+ def test_hang_gh94440(self):
+ """shutdown(wait=True) doesn't hang when a future was submitted and
+ quickly canceled right before shutdown.
-class ThreadPoolShutdownTest(ThreadPoolMixin, ExecutorShutdownTest, BaseTestCase):
- def _prime_executor(self):
- pass
+ See https://github.com/python/cpython/issues/94440.
+ """
+ if not hasattr(signal, 'alarm'):
+ raise unittest.SkipTest(
+ "Tested platform does not support the alarm signal")
+
+ def timeout(_signum, _frame):
+ raise RuntimeError("timed out waiting for shutdown")
+ kwargs = {}
+ if getattr(self, 'ctx', None):
+ kwargs['mp_context'] = self.get_context()
+ executor = self.executor_type(max_workers=1, **kwargs)
+ executor.submit(int).result()
+ old_handler = signal.signal(signal.SIGALRM, timeout)
+ try:
+ signal.alarm(5)
+ executor.submit(int).cancel()
+ executor.shutdown(wait=True)
+ finally:
+ signal.alarm(0)
+ signal.signal(signal.SIGALRM, old_handler)
+
+
+class ThreadPoolShutdownTest(ThreadPoolMixin, ExecutorShutdownTest, BaseTestCase):
def test_threads_terminate(self):
def acquire_lock(lock):
lock.acquire()
@@ -463,6 +484,7 @@ def test_thread_names_assigned(self):
executor.map(abs, range(-5, 5))
threads = executor._threads
del executor
+ support.gc_collect() # For PyPy or other GCs.
for t in threads:
self.assertRegex(t.name, r'^SpecialPool_[0-4]$')
@@ -473,6 +495,7 @@ def test_thread_names_default(self):
executor.map(abs, range(-5, 5))
threads = executor._threads
del executor
+ support.gc_collect() # For PyPy or other GCs.
for t in threads:
# Ensure that our default name is reasonably sane and unique when
@@ -490,7 +513,7 @@ def test_cancel_futures_wait_false(self):
t = ThreadPoolExecutor()
t.submit(sleep_and_print, .1, "apple")
t.shutdown(wait=False, cancel_futures=True)
- """.format(executor_type=self.executor_type.__name__))
+ """)
# Errors in atexit hooks don't change the process exit code, check
# stderr manually.
self.assertFalse(err)
@@ -498,18 +521,21 @@ def test_cancel_futures_wait_false(self):
class ProcessPoolShutdownTest(ExecutorShutdownTest):
- def _prime_executor(self):
- pass
-
def test_processes_terminate(self):
def acquire_lock(lock):
lock.acquire()
- mp_context = get_context()
+ mp_context = self.get_context()
+ if mp_context.get_start_method(allow_none=False) == "fork":
+ # fork pre-spawns, not on demand.
+ expected_num_processes = self.worker_count
+ else:
+ expected_num_processes = 3
+
sem = mp_context.Semaphore(0)
for _ in range(3):
self.executor.submit(acquire_lock, sem)
- self.assertEqual(len(self.executor._processes), 3)
+ self.assertEqual(len(self.executor._processes), expected_num_processes)
for _ in range(3):
sem.release()
processes = self.executor._processes
@@ -519,7 +545,8 @@ def acquire_lock(lock):
p.join()
def test_context_manager_shutdown(self):
- with futures.ProcessPoolExecutor(max_workers=5) as e:
+ with futures.ProcessPoolExecutor(
+ max_workers=5, mp_context=self.get_context()) as e:
processes = e._processes
self.assertEqual(list(e.map(abs, range(-5, 5))),
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4])
@@ -528,13 +555,15 @@ def test_context_manager_shutdown(self):
p.join()
def test_del_shutdown(self):
- executor = futures.ProcessPoolExecutor(max_workers=5)
+ executor = futures.ProcessPoolExecutor(
+ max_workers=5, mp_context=self.get_context())
res = executor.map(abs, range(-5, 5))
executor_manager_thread = executor._executor_manager_thread
processes = executor._processes
call_queue = executor._call_queue
executor_manager_thread = executor._executor_manager_thread
del executor
+ support.gc_collect() # For PyPy or other GCs.
# Make sure that all the executor resources were properly cleaned by
# the shutdown process
@@ -550,7 +579,8 @@ def test_del_shutdown(self):
def test_shutdown_no_wait(self):
# Ensure that the executor cleans up the processes when calling
# shutdown with wait=False
- executor = futures.ProcessPoolExecutor(max_workers=5)
+ executor = futures.ProcessPoolExecutor(
+ max_workers=5, mp_context=self.get_context())
res = executor.map(abs, range(-5, 5))
processes = executor._processes
call_queue = executor._call_queue
@@ -576,6 +606,14 @@ def test_shutdown_no_wait(self):
class WaitTests:
+ def test_20369(self):
+ # See https://bugs.python.org/issue20369
+ future = self.executor.submit(time.sleep, 1.5)
+ done, not_done = futures.wait([future, future],
+ return_when=futures.ALL_COMPLETED)
+ self.assertEqual({future}, done)
+ self.assertEqual(set(), not_done)
+
def test_first_completed(self):
future1 = self.executor.submit(mul, 21, 2)
@@ -759,6 +797,7 @@ def test_free_reference_yielded_future(self):
futures_list.remove(future)
wr = weakref.ref(future)
del future
+ support.gc_collect() # For PyPy or other GCs.
self.assertIsNone(wr())
futures_list[0].set_result("test")
@@ -766,6 +805,7 @@ def test_free_reference_yielded_future(self):
futures_list.remove(future)
wr = weakref.ref(future)
del future
+ support.gc_collect() # For PyPy or other GCs.
self.assertIsNone(wr())
if futures_list:
futures_list[0].set_result("test")
@@ -865,6 +905,7 @@ def test_free_reference(self):
for obj in self.executor.map(make_dummy_object, range(10)):
wr = weakref.ref(obj)
del obj
+ support.gc_collect() # For PyPy or other GCs.
self.assertIsNone(wr())
@@ -905,6 +946,47 @@ def test_idle_thread_reuse(self):
self.assertEqual(len(executor._threads), 1)
executor.shutdown(wait=True)
+ @unittest.skipUnless(hasattr(os, 'register_at_fork'), 'need os.register_at_fork')
+ def test_hang_global_shutdown_lock(self):
+ # bpo-45021: _global_shutdown_lock should be reinitialized in the child
+ # process, otherwise it will never exit
+ def submit(pool):
+ pool.submit(submit, pool)
+
+ with futures.ThreadPoolExecutor(1) as pool:
+ pool.submit(submit, pool)
+
+ for _ in range(50):
+ with futures.ProcessPoolExecutor(1, mp_context=mp.get_context('fork')) as workers:
+ workers.submit(tuple)
+
+ def test_executor_map_current_future_cancel(self):
+ stop_event = threading.Event()
+ log = []
+
+ def log_n_wait(ident):
+ log.append(f"{ident=} started")
+ try:
+ stop_event.wait()
+ finally:
+ log.append(f"{ident=} stopped")
+
+ with self.executor_type(max_workers=1) as pool:
+ # submit work to saturate the pool
+ fut = pool.submit(log_n_wait, ident="first")
+ try:
+ with contextlib.closing(
+ pool.map(log_n_wait, ["second", "third"], timeout=0)
+ ) as gen:
+ with self.assertRaises(futures.TimeoutError):
+ next(gen)
+ finally:
+ stop_event.set()
+ fut.result()
+ # ident='second' is cancelled as a result of raising a TimeoutError
+ # ident='third' is cancelled because it remained in the collection of futures
+ self.assertListEqual(log, ["ident='first' started", "ident='first' stopped"])
+
class ProcessPoolExecutorTest(ExecutorTest):
@@ -972,7 +1054,7 @@ def test_traceback(self):
def test_ressources_gced_in_workers(self):
# Ensure that argument for a job are correctly gc-ed after the job
# is finished
- mgr = get_context(self.ctx).Manager()
+ mgr = self.get_context().Manager()
obj = EventfulGCObj(mgr)
future = self.executor.submit(id, obj)
future.result()
@@ -988,36 +1070,41 @@ def test_ressources_gced_in_workers(self):
mgr.join()
def test_saturation(self):
- executor = self.executor_type(4)
- mp_context = get_context()
+ executor = self.executor
+ mp_context = self.get_context()
sem = mp_context.Semaphore(0)
job_count = 15 * executor._max_workers
- try:
- for _ in range(job_count):
- executor.submit(sem.acquire)
- self.assertEqual(len(executor._processes), executor._max_workers)
- for _ in range(job_count):
- sem.release()
- finally:
- executor.shutdown()
+ for _ in range(job_count):
+ executor.submit(sem.acquire)
+ self.assertEqual(len(executor._processes), executor._max_workers)
+ for _ in range(job_count):
+ sem.release()
def test_idle_process_reuse_one(self):
- executor = self.executor_type(4)
+ executor = self.executor
+ assert executor._max_workers >= 4
+ if self.get_context().get_start_method(allow_none=False) == "fork":
+ raise unittest.SkipTest("Incompatible with the fork start method.")
executor.submit(mul, 21, 2).result()
executor.submit(mul, 6, 7).result()
executor.submit(mul, 3, 14).result()
self.assertEqual(len(executor._processes), 1)
- executor.shutdown()
def test_idle_process_reuse_multiple(self):
- executor = self.executor_type(4)
+ executor = self.executor
+ assert executor._max_workers <= 5
+ if self.get_context().get_start_method(allow_none=False) == "fork":
+ raise unittest.SkipTest("Incompatible with the fork start method.")
executor.submit(mul, 12, 7).result()
executor.submit(mul, 33, 25)
executor.submit(mul, 25, 26).result()
executor.submit(mul, 18, 29)
- self.assertLessEqual(len(executor._processes), 2)
+ executor.submit(mul, 1, 2).result()
+ executor.submit(mul, 0, 9)
+ self.assertLessEqual(len(executor._processes), 3)
executor.shutdown()
+
create_executor_tests(ProcessPoolExecutorTest,
executor_mixins=(ProcessPoolForkMixin,
ProcessPoolForkserverMixin,
@@ -1119,7 +1206,7 @@ def _check_crash(self, error, func, *args, ignore_stderr=False):
self.executor.shutdown(wait=True)
executor = self.executor_type(
- max_workers=2, mp_context=get_context(self.ctx))
+ max_workers=2, mp_context=self.get_context())
res = executor.submit(func, *args)
if ignore_stderr:
@@ -1198,7 +1285,7 @@ def test_shutdown_deadlock(self):
# if a worker fails after the shutdown call.
self.executor.shutdown(wait=True)
with self.executor_type(max_workers=2,
- mp_context=get_context(self.ctx)) as executor:
+ mp_context=self.get_context()) as executor:
self.executor = executor # Allow clean up in fail_on_deadlock
f = executor.submit(_crash, delay=.1)
executor.shutdown(wait=True)
@@ -1211,7 +1298,7 @@ def test_shutdown_deadlock_pickle(self):
# Reported in bpo-39104.
self.executor.shutdown(wait=True)
with self.executor_type(max_workers=2,
- mp_context=get_context(self.ctx)) as executor:
+ mp_context=self.get_context()) as executor:
self.executor = executor # Allow clean up in fail_on_deadlock
# Start the executor and get the executor_manager_thread to collect
@@ -1509,16 +1596,10 @@ def test_multiple_set_exception(self):
self.assertEqual(f.exception(), e)
-_threads_key = None
-
def setUpModule():
- global _threads_key
- _threads_key = threading_helper.threading_setup()
-
-
-def tearDownModule():
- threading_helper.threading_cleanup(*_threads_key)
- multiprocessing.util._cleanup_tests()
+ unittest.addModuleCleanup(multiprocessing.util._cleanup_tests)
+ thread_info = threading_helper.threading_setup()
+ unittest.addModuleCleanup(threading_helper.threading_cleanup, *thread_info)
if __name__ == "__main__":
diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py
index 9373a62072ef47..e9b03e6c62ef14 100644
--- a/Lib/test/test_configparser.py
+++ b/Lib/test/test_configparser.py
@@ -79,6 +79,7 @@ def basic_test(self, cf):
'Spacey Bar',
'Spacey Bar From The Beginning',
'Types',
+ 'This One Has A ] In It',
]
if self.allow_no_value:
@@ -130,6 +131,7 @@ def basic_test(self, cf):
eq(cf.get('Types', 'float'), "0.44")
eq(cf.getboolean('Types', 'boolean'), False)
eq(cf.get('Types', '123'), 'strange but acceptable')
+ eq(cf.get('This One Has A ] In It', 'forks'), 'spoons')
if self.allow_no_value:
eq(cf.get('NoValue', 'option-without-value'), None)
@@ -320,6 +322,8 @@ def test_basic(self):
float {0[0]} 0.44
boolean {0[0]} NO
123 {0[1]} strange but acceptable
+[This One Has A ] In It]
+ forks {0[0]} spoons
""".format(self.delimiters, self.comment_prefixes)
if self.allow_no_value:
config_string += (
@@ -394,6 +398,9 @@ def test_basic_from_dict(self):
"boolean": False,
123: "strange but acceptable",
},
+ "This One Has A ] In It": {
+ "forks": "spoons"
+ },
}
if self.allow_no_value:
config.update({
diff --git a/Lib/test/test_context.py b/Lib/test/test_context.py
index 2d8b63a1f59581..689e3d4dc4591a 100644
--- a/Lib/test/test_context.py
+++ b/Lib/test/test_context.py
@@ -533,6 +533,41 @@ def test_hamt_collision_1(self):
self.assertEqual(len(h4), 2)
self.assertEqual(len(h5), 3)
+ def test_hamt_collision_3(self):
+ # Test that iteration works with the deepest tree possible.
+ # https://github.com/python/cpython/issues/93065
+
+ C = HashKey(0b10000000_00000000_00000000_00000000, 'C')
+ D = HashKey(0b10000000_00000000_00000000_00000000, 'D')
+
+ E = HashKey(0b00000000_00000000_00000000_00000000, 'E')
+
+ h = hamt()
+ h = h.set(C, 'C')
+ h = h.set(D, 'D')
+ h = h.set(E, 'E')
+
+ # BitmapNode(size=2 count=1 bitmap=0b1):
+ # NULL:
+ # BitmapNode(size=2 count=1 bitmap=0b1):
+ # NULL:
+ # BitmapNode(size=2 count=1 bitmap=0b1):
+ # NULL:
+ # BitmapNode(size=2 count=1 bitmap=0b1):
+ # NULL:
+ # BitmapNode(size=2 count=1 bitmap=0b1):
+ # NULL:
+ # BitmapNode(size=2 count=1 bitmap=0b1):
+ # NULL:
+ # BitmapNode(size=4 count=2 bitmap=0b101):
+ # : 'E'
+ # NULL:
+ # CollisionNode(size=4 id=0x107a24520):
+ # : 'C'
+ # : 'D'
+
+ self.assertEqual({k.name for k in h.keys()}, {'C', 'D', 'E'})
+
def test_hamt_stress(self):
COLLECTION_SIZE = 7000
TEST_ITERS_EVERY = 647
diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py
index 453ef6c9f0832f..bcedf17fc8061f 100644
--- a/Lib/test/test_contextlib.py
+++ b/Lib/test/test_contextlib.py
@@ -4,6 +4,7 @@
import sys
import tempfile
import threading
+import traceback
import unittest
from contextlib import * # Tests __all__
from test import support
@@ -86,6 +87,56 @@ def woohoo():
raise ZeroDivisionError()
self.assertEqual(state, [1, 42, 999])
+ def test_contextmanager_traceback(self):
+ @contextmanager
+ def f():
+ yield
+
+ try:
+ with f():
+ 1/0
+ except ZeroDivisionError as e:
+ frames = traceback.extract_tb(e.__traceback__)
+
+ self.assertEqual(len(frames), 1)
+ self.assertEqual(frames[0].name, 'test_contextmanager_traceback')
+ self.assertEqual(frames[0].line, '1/0')
+
+ # Repeat with RuntimeError (which goes through a different code path)
+ class RuntimeErrorSubclass(RuntimeError):
+ pass
+
+ try:
+ with f():
+ raise RuntimeErrorSubclass(42)
+ except RuntimeErrorSubclass as e:
+ frames = traceback.extract_tb(e.__traceback__)
+
+ self.assertEqual(len(frames), 1)
+ self.assertEqual(frames[0].name, 'test_contextmanager_traceback')
+ self.assertEqual(frames[0].line, 'raise RuntimeErrorSubclass(42)')
+
+ class StopIterationSubclass(StopIteration):
+ pass
+
+ for stop_exc in (
+ StopIteration('spam'),
+ StopIterationSubclass('spam'),
+ ):
+ with self.subTest(type=type(stop_exc)):
+ try:
+ with f():
+ raise stop_exc
+ except type(stop_exc) as e:
+ self.assertIs(e, stop_exc)
+ frames = traceback.extract_tb(e.__traceback__)
+ else:
+ self.fail(f'{stop_exc} was suppressed')
+
+ self.assertEqual(len(frames), 1)
+ self.assertEqual(frames[0].name, 'test_contextmanager_traceback')
+ self.assertEqual(frames[0].line, 'raise stop_exc')
+
def test_contextmanager_no_reraise(self):
@contextmanager
def whee():
@@ -126,19 +177,22 @@ def woohoo():
self.assertEqual(state, [1, 42, 999])
def test_contextmanager_except_stopiter(self):
- stop_exc = StopIteration('spam')
@contextmanager
def woohoo():
yield
- try:
- with self.assertWarnsRegex(DeprecationWarning,
- "StopIteration"):
- with woohoo():
- raise stop_exc
- except Exception as ex:
- self.assertIs(ex, stop_exc)
- else:
- self.fail('StopIteration was suppressed')
+
+ class StopIterationSubclass(StopIteration):
+ pass
+
+ for stop_exc in (StopIteration('spam'), StopIterationSubclass('spam')):
+ with self.subTest(type=type(stop_exc)):
+ try:
+ with woohoo():
+ raise stop_exc
+ except Exception as ex:
+ self.assertIs(ex, stop_exc)
+ else:
+ self.fail(f'{stop_exc} was suppressed')
def test_contextmanager_except_pep479(self):
code = """\
@@ -228,6 +282,8 @@ class A:
def woohoo(a, b):
a = weakref.ref(a)
b = weakref.ref(b)
+ # Allow test to work with a non-refcounted GC
+ support.gc_collect()
self.assertIsNone(a())
self.assertIsNone(b())
yield
@@ -696,6 +752,38 @@ def test_exit_suppress(self):
stack.push(lambda *exc: True)
1/0
+ def test_exit_exception_traceback(self):
+ # This test captures the current behavior of ExitStack so that we know
+ # if we ever unintendedly change it. It is not a statement of what the
+ # desired behavior is (for instance, we may want to remove some of the
+ # internal contextlib frames).
+
+ def raise_exc(exc):
+ raise exc
+
+ try:
+ with self.exit_stack() as stack:
+ stack.callback(raise_exc, ValueError)
+ 1/0
+ except ValueError as e:
+ exc = e
+
+ self.assertIsInstance(exc, ValueError)
+ ve_frames = traceback.extract_tb(exc.__traceback__)
+ expected = \
+ [('test_exit_exception_traceback', 'with self.exit_stack() as stack:')] + \
+ self.callback_error_internal_frames + \
+ [('_exit_wrapper', 'callback(*args, **kwds)'),
+ ('raise_exc', 'raise exc')]
+
+ self.assertEqual(
+ [(f.name, f.line) for f in ve_frames], expected)
+
+ self.assertIsInstance(exc.__context__, ZeroDivisionError)
+ zde_frames = traceback.extract_tb(exc.__context__.__traceback__)
+ self.assertEqual([(f.name, f.line) for f in zde_frames],
+ [('test_exit_exception_traceback', '1/0')])
+
def test_exit_exception_chaining_reference(self):
# Sanity check to make sure that ExitStack chaining matches
# actual nested with statements
@@ -775,6 +863,40 @@ def suppress_exc(*exc_details):
self.assertIsInstance(inner_exc, ValueError)
self.assertIsInstance(inner_exc.__context__, ZeroDivisionError)
+ def test_exit_exception_explicit_none_context(self):
+ # Ensure ExitStack chaining matches actual nested `with` statements
+ # regarding explicit __context__ = None.
+
+ class MyException(Exception):
+ pass
+
+ @contextmanager
+ def my_cm():
+ try:
+ yield
+ except BaseException:
+ exc = MyException()
+ try:
+ raise exc
+ finally:
+ exc.__context__ = None
+
+ @contextmanager
+ def my_cm_with_exit_stack():
+ with self.exit_stack() as stack:
+ stack.enter_context(my_cm())
+ yield stack
+
+ for cm in (my_cm, my_cm_with_exit_stack):
+ with self.subTest():
+ try:
+ with cm():
+ raise IndexError()
+ except MyException as exc:
+ self.assertIsNone(exc.__context__)
+ else:
+ self.fail("Expected IndexError, but no exception was raised")
+
def test_exit_exception_non_suppressing(self):
# http://bugs.python.org/issue19092
def raise_exc(exc):
@@ -929,6 +1051,10 @@ def first():
class TestExitStack(TestBaseExitStack, unittest.TestCase):
exit_stack = ExitStack
+ callback_error_internal_frames = [
+ ('__exit__', 'raise exc_details[1]'),
+ ('__exit__', 'if cb(*exc_details):'),
+ ]
class TestRedirectStream:
diff --git a/Lib/test/test_contextlib_async.py b/Lib/test/test_contextlib_async.py
index 290ef05b82a801..3d1079c205aca1 100644
--- a/Lib/test/test_contextlib_async.py
+++ b/Lib/test/test_contextlib_async.py
@@ -1,9 +1,10 @@
import asyncio
from contextlib import (
asynccontextmanager, AbstractAsyncContextManager,
- AsyncExitStack, nullcontext, aclosing)
+ AsyncExitStack, nullcontext, aclosing, contextmanager)
import functools
from test import support
+import traceback
import unittest
from test.test_contextlib import TestBaseExitStack
@@ -14,15 +15,12 @@ def _async_test(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
coro = func(*args, **kwargs)
- loop = asyncio.new_event_loop()
- asyncio.set_event_loop(loop)
- try:
- return loop.run_until_complete(coro)
- finally:
- loop.close()
- asyncio.set_event_loop_policy(None)
+ asyncio.run(coro)
return wrapper
+def tearDownModule():
+ asyncio.set_event_loop_policy(None)
+
class TestAbstractAsyncContextManager(unittest.TestCase):
@@ -127,6 +125,63 @@ async def woohoo():
raise ZeroDivisionError()
self.assertEqual(state, [1, 42, 999])
+ @_async_test
+ async def test_contextmanager_traceback(self):
+ @asynccontextmanager
+ async def f():
+ yield
+
+ try:
+ async with f():
+ 1/0
+ except ZeroDivisionError as e:
+ frames = traceback.extract_tb(e.__traceback__)
+
+ self.assertEqual(len(frames), 1)
+ self.assertEqual(frames[0].name, 'test_contextmanager_traceback')
+ self.assertEqual(frames[0].line, '1/0')
+
+ # Repeat with RuntimeError (which goes through a different code path)
+ class RuntimeErrorSubclass(RuntimeError):
+ pass
+
+ try:
+ async with f():
+ raise RuntimeErrorSubclass(42)
+ except RuntimeErrorSubclass as e:
+ frames = traceback.extract_tb(e.__traceback__)
+
+ self.assertEqual(len(frames), 1)
+ self.assertEqual(frames[0].name, 'test_contextmanager_traceback')
+ self.assertEqual(frames[0].line, 'raise RuntimeErrorSubclass(42)')
+
+ class StopIterationSubclass(StopIteration):
+ pass
+
+ class StopAsyncIterationSubclass(StopAsyncIteration):
+ pass
+
+ for stop_exc in (
+ StopIteration('spam'),
+ StopAsyncIteration('ham'),
+ StopIterationSubclass('spam'),
+ StopAsyncIterationSubclass('spam')
+ ):
+ with self.subTest(type=type(stop_exc)):
+ try:
+ async with f():
+ raise stop_exc
+ except type(stop_exc) as e:
+ self.assertIs(e, stop_exc)
+ frames = traceback.extract_tb(e.__traceback__)
+ else:
+ self.fail(f'{stop_exc} was suppressed')
+
+ self.assertEqual(len(frames), 1)
+ self.assertEqual(frames[0].name, 'test_contextmanager_traceback')
+ self.assertEqual(frames[0].line, 'raise stop_exc')
+
+
@_async_test
async def test_contextmanager_no_reraise(self):
@asynccontextmanager
@@ -209,7 +264,18 @@ async def test_contextmanager_except_stopiter(self):
async def woohoo():
yield
- for stop_exc in (StopIteration('spam'), StopAsyncIteration('ham')):
+ class StopIterationSubclass(StopIteration):
+ pass
+
+ class StopAsyncIterationSubclass(StopAsyncIteration):
+ pass
+
+ for stop_exc in (
+ StopIteration('spam'),
+ StopAsyncIteration('ham'),
+ StopIterationSubclass('spam'),
+ StopAsyncIterationSubclass('spam')
+ ):
with self.subTest(type=type(stop_exc)):
try:
async with woohoo():
@@ -346,14 +412,17 @@ async def aclose(self):
async def test_aclosing_bpo41229(self):
state = []
- class Resource:
- def __del__(self):
+ @contextmanager
+ def sync_resource():
+ try:
+ yield
+ finally:
state.append(1)
async def agenfunc():
- r = Resource()
- yield -1
- yield -2
+ with sync_resource():
+ yield -1
+ yield -2
x = agenfunc()
self.assertEqual(state, [])
@@ -398,6 +467,13 @@ def __exit__(self, *exc_details):
return self.run_coroutine(self.__aexit__(*exc_details))
exit_stack = SyncAsyncExitStack
+ callback_error_internal_frames = [
+ ('__exit__', 'return self.run_coroutine(self.__aexit__(*exc_details))'),
+ ('run_coroutine', 'raise exc'),
+ ('run_coroutine', 'raise exc'),
+ ('__aexit__', 'raise exc_details[1]'),
+ ('__aexit__', 'cb_suppress = cb(*exc_details)'),
+ ]
def setUp(self):
self.loop = asyncio.new_event_loop()
@@ -538,6 +614,41 @@ async def suppress_exc(*exc_details):
self.assertIsInstance(inner_exc, ValueError)
self.assertIsInstance(inner_exc.__context__, ZeroDivisionError)
+ @_async_test
+ async def test_async_exit_exception_explicit_none_context(self):
+ # Ensure AsyncExitStack chaining matches actual nested `with` statements
+ # regarding explicit __context__ = None.
+
+ class MyException(Exception):
+ pass
+
+ @asynccontextmanager
+ async def my_cm():
+ try:
+ yield
+ except BaseException:
+ exc = MyException()
+ try:
+ raise exc
+ finally:
+ exc.__context__ = None
+
+ @asynccontextmanager
+ async def my_cm_with_exit_stack():
+ async with self.exit_stack() as stack:
+ await stack.enter_async_context(my_cm())
+ yield stack
+
+ for cm in (my_cm, my_cm_with_exit_stack):
+ with self.subTest():
+ try:
+ async with cm():
+ raise IndexError()
+ except MyException as exc:
+ self.assertIsNone(exc.__context__)
+ else:
+ self.fail("Expected IndexError, but no exception was raised")
+
class TestAsyncNullcontext(unittest.TestCase):
@_async_test
diff --git a/Lib/test/test_copy.py b/Lib/test/test_copy.py
index ba3d233f63d1c1..7b1a927b4e5692 100644
--- a/Lib/test/test_copy.py
+++ b/Lib/test/test_copy.py
@@ -7,6 +7,7 @@
from operator import le, lt, ge, gt, eq, ne
import unittest
+from test import support
order_comparisons = le, lt, ge, gt
equality_comparisons = eq, ne
@@ -50,6 +51,9 @@ def pickle_C(obj):
self.assertRaises(TypeError, copy.copy, x)
copyreg.pickle(C, pickle_C, C)
y = copy.copy(x)
+ self.assertIsNot(x, y)
+ self.assertEqual(type(y), C)
+ self.assertEqual(y.foo, x.foo)
def test_copy_reduce_ex(self):
class C(object):
@@ -312,6 +316,9 @@ def pickle_C(obj):
self.assertRaises(TypeError, copy.deepcopy, x)
copyreg.pickle(C, pickle_C, C)
y = copy.deepcopy(x)
+ self.assertIsNot(x, y)
+ self.assertEqual(type(y), C)
+ self.assertEqual(y.foo, x.foo)
def test_deepcopy_reduce_ex(self):
class C(object):
@@ -355,8 +362,8 @@ class NewStyle(object):
pass
def f():
pass
- tests = [None, 42, 2**100, 3.14, True, False, 1j,
- "hello", "hello\u1234", f.__code__,
+ tests = [None, ..., NotImplemented, 42, 2**100, 3.14, True, False, 1j,
+ b"bytes", "hello", "hello\u1234", f.__code__,
NewStyle, range(10), Classic, max, property()]
for x in tests:
self.assertIs(copy.deepcopy(x), x)
@@ -677,6 +684,28 @@ def __eq__(self, other):
self.assertIsNot(x, y)
self.assertIsNot(x["foo"], y["foo"])
+ def test_reduce_6tuple(self):
+ def state_setter(*args, **kwargs):
+ self.fail("shouldn't call this")
+ class C:
+ def __reduce__(self):
+ return C, (), self.__dict__, None, None, state_setter
+ x = C()
+ with self.assertRaises(TypeError):
+ copy.copy(x)
+ with self.assertRaises(TypeError):
+ copy.deepcopy(x)
+
+ def test_reduce_6tuple_none(self):
+ class C:
+ def __reduce__(self):
+ return C, (), self.__dict__, None, None, None
+ x = C()
+ with self.assertRaises(TypeError):
+ copy.copy(x)
+ with self.assertRaises(TypeError):
+ copy.deepcopy(x)
+
def test_copy_slots(self):
class C(object):
__slots__ = ["foo"]
@@ -805,6 +834,7 @@ class C(object):
self.assertEqual(v[c], d)
self.assertEqual(len(v), 2)
del c, d
+ support.gc_collect() # For PyPy or other GCs.
self.assertEqual(len(v), 1)
x, y = C(), C()
# The underlying containers are decoupled
@@ -834,6 +864,7 @@ def __init__(self, i):
self.assertEqual(v[a].i, b.i)
self.assertEqual(v[c].i, d.i)
del c
+ support.gc_collect() # For PyPy or other GCs.
self.assertEqual(len(v), 1)
def test_deepcopy_weakvaluedict(self):
@@ -857,6 +888,7 @@ def __init__(self, i):
self.assertIs(t, d)
del x, y, z, t
del d
+ support.gc_collect() # For PyPy or other GCs.
self.assertEqual(len(v), 1)
def test_deepcopy_bound_method(self):
diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py
index 145adb67781701..f4c52689071875 100644
--- a/Lib/test/test_coroutines.py
+++ b/Lib/test/test_coroutines.py
@@ -4,6 +4,7 @@
import pickle
import sys
import types
+import traceback
import unittest
import warnings
from test import support
@@ -1205,41 +1206,47 @@ class CM:
def __aenter__(self):
pass
- body_executed = False
+ body_executed = None
async def foo():
+ nonlocal body_executed
+ body_executed = False
async with CM():
body_executed = True
with self.assertRaisesRegex(AttributeError, '__aexit__'):
run_async(foo())
- self.assertFalse(body_executed)
+ self.assertIs(body_executed, False)
def test_with_3(self):
class CM:
def __aexit__(self):
pass
- body_executed = False
+ body_executed = None
async def foo():
+ nonlocal body_executed
+ body_executed = False
async with CM():
body_executed = True
with self.assertRaisesRegex(AttributeError, '__aenter__'):
run_async(foo())
- self.assertFalse(body_executed)
+ self.assertIs(body_executed, False)
def test_with_4(self):
class CM:
pass
- body_executed = False
+ body_executed = None
async def foo():
+ nonlocal body_executed
+ body_executed = False
async with CM():
body_executed = True
with self.assertRaisesRegex(AttributeError, '__aenter__'):
run_async(foo())
- self.assertFalse(body_executed)
+ self.assertIs(body_executed, False)
def test_with_5(self):
# While this test doesn't make a lot of sense,
@@ -1255,7 +1262,7 @@ async def __aexit__(self, *exc):
async def func():
async with CM():
- assert (1, ) == 1
+ self.assertEqual((1, ), 1)
with self.assertRaises(AssertionError):
run_async(func())
@@ -2113,6 +2120,29 @@ async def run_gen():
return 'end'
self.assertEqual(run_async(run_gen()), ([], 'end'))
+ def test_stack_in_coroutine_throw(self):
+ # Regression test for https://github.com/python/cpython/issues/93592
+ async def a():
+ return await b()
+
+ async def b():
+ return await c()
+
+ @types.coroutine
+ def c():
+ try:
+ # traceback.print_stack()
+ yield len(traceback.extract_stack())
+ except ZeroDivisionError:
+ # traceback.print_stack()
+ yield len(traceback.extract_stack())
+
+ coro = a()
+ len_send = coro.send(None)
+ len_throw = coro.throw(ZeroDivisionError)
+ # before fixing, visible stack from throw would be shorter than from send.
+ self.assertEqual(len_send, len_throw)
+
class CoroAsyncIOCompatTest(unittest.TestCase):
@@ -2289,7 +2319,8 @@ class UnawaitedWarningDuringShutdownTest(unittest.TestCase):
def test_unawaited_warning_during_shutdown(self):
code = ("import asyncio\n"
"async def f(): pass\n"
- "asyncio.gather(f())\n")
+ "async def t(): asyncio.gather(f())\n"
+ "asyncio.run(t())\n")
assert_python_ok("-c", code)
code = ("import sys\n"
diff --git a/Lib/test/test_crypt.py b/Lib/test/test_crypt.py
index 5dc83b4ecbfa00..877c575c5534ae 100644
--- a/Lib/test/test_crypt.py
+++ b/Lib/test/test_crypt.py
@@ -1,8 +1,11 @@
import sys
import unittest
+from test.support import check_sanitizer
try:
+ if check_sanitizer(address=True, memory=True):
+ raise unittest.SkipTest("The crypt module SEGFAULTs on ASAN/MSAN builds")
import crypt
IMPORT_ERROR = None
except ImportError as ex:
diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py
index 2411a91110dd43..623a6b60b0fa55 100644
--- a/Lib/test/test_csv.py
+++ b/Lib/test/test_csv.py
@@ -157,7 +157,7 @@ def test_write_arg_valid(self):
self._write_error_test(OSError, BadIterable())
class BadList:
def __len__(self):
- return 10;
+ return 10
def __getitem__(self, i):
if i > 2:
raise OSError
@@ -335,6 +335,11 @@ def test_read_quoting(self):
['abc,3'], [[]],
quoting=csv.QUOTE_NONNUMERIC)
+ def test_read_skipinitialspace(self):
+ self._read_test(['no space, space, spaces,\ttab'],
+ [['no space', 'space', 'spaces', '\ttab']],
+ skipinitialspace=True)
+
def test_read_bigfield(self):
# This exercises the buffer realloc functionality and field size
# limits.
@@ -421,6 +426,34 @@ def test_register_kwargs(self):
self.assertEqual(csv.get_dialect(name).delimiter, ';')
self.assertEqual([['X', 'Y', 'Z']], list(csv.reader(['X;Y;Z'], name)))
+ def test_register_kwargs_override(self):
+ class mydialect(csv.Dialect):
+ delimiter = "\t"
+ quotechar = '"'
+ doublequote = True
+ skipinitialspace = False
+ lineterminator = '\r\n'
+ quoting = csv.QUOTE_MINIMAL
+
+ name = 'test_dialect'
+ csv.register_dialect(name, mydialect,
+ delimiter=';',
+ quotechar="'",
+ doublequote=False,
+ skipinitialspace=True,
+ lineterminator='\n',
+ quoting=csv.QUOTE_ALL)
+ self.addCleanup(csv.unregister_dialect, name)
+
+ # Ensure that kwargs do override attributes of a dialect class:
+ dialect = csv.get_dialect(name)
+ self.assertEqual(dialect.delimiter, ';')
+ self.assertEqual(dialect.quotechar, "'")
+ self.assertEqual(dialect.doublequote, False)
+ self.assertEqual(dialect.skipinitialspace, True)
+ self.assertEqual(dialect.lineterminator, '\n')
+ self.assertEqual(dialect.quoting, csv.QUOTE_ALL)
+
def test_incomplete_dialect(self):
class myexceltsv(csv.Dialect):
delimiter = "\t"
@@ -458,9 +491,15 @@ class testC(csv.excel):
class testUni(csv.excel):
delimiter = "\u039B"
+ class unspecified():
+ # A class to pass as dialect but with no dialect attributes.
+ pass
+
csv.register_dialect('testC', testC)
try:
self.compare_dialect_123("1,2,3\r\n")
+ self.compare_dialect_123("1,2,3\r\n", dialect=None)
+ self.compare_dialect_123("1,2,3\r\n", dialect=unspecified)
self.compare_dialect_123("1\t2\t3\r\n", testA)
self.compare_dialect_123("1:2:3\r\n", dialect=testB())
self.compare_dialect_123("1|2|3\r\n", dialect='testC')
@@ -891,7 +930,7 @@ class mydialect(csv.Dialect):
with self.assertRaises(csv.Error) as cm:
mydialect()
self.assertEqual(str(cm.exception),
- '"quotechar" must be string, not int')
+ '"quotechar" must be string or None, not int')
def test_delimiter(self):
class mydialect(csv.Dialect):
@@ -928,6 +967,35 @@ class mydialect(csv.Dialect):
self.assertEqual(str(cm.exception),
'"delimiter" must be string, not int')
+ mydialect.delimiter = None
+ with self.assertRaises(csv.Error) as cm:
+ mydialect()
+ self.assertEqual(str(cm.exception),
+ '"delimiter" must be string, not NoneType')
+
+ def test_escapechar(self):
+ class mydialect(csv.Dialect):
+ delimiter = ";"
+ escapechar = '\\'
+ doublequote = False
+ skipinitialspace = True
+ lineterminator = '\r\n'
+ quoting = csv.QUOTE_NONE
+ d = mydialect()
+ self.assertEqual(d.escapechar, "\\")
+
+ mydialect.escapechar = "**"
+ with self.assertRaisesRegex(csv.Error, '"escapechar" must be a 1-character string'):
+ mydialect()
+
+ mydialect.escapechar = b"*"
+ with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not bytes'):
+ mydialect()
+
+ mydialect.escapechar = 4
+ with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not int'):
+ mydialect()
+
def test_lineterminator(self):
class mydialect(csv.Dialect):
delimiter = ";"
@@ -1014,6 +1082,42 @@ class TestSniffer(unittest.TestCase):
'Stonecutters ''Seafood'' and Chop House'+ 'Lemont'+ 'IL'+ '12/19/02'+ 'Week Back'
"""
+ sample10 = dedent("""
+ abc,def
+ ghijkl,mno
+ ghi,jkl
+ """)
+
+ sample11 = dedent("""
+ abc,def
+ ghijkl,mnop
+ ghi,jkl
+ """)
+
+ sample12 = dedent(""""time","forces"
+ 1,1.5
+ 0.5,5+0j
+ 0,0
+ 1+1j,6
+ """)
+
+ sample13 = dedent(""""time","forces"
+ 0,0
+ 1,2
+ a,b
+ """)
+
+ def test_issue43625(self):
+ sniffer = csv.Sniffer()
+ self.assertTrue(sniffer.has_header(self.sample12))
+ self.assertFalse(sniffer.has_header(self.sample13))
+
+ def test_has_header_strings(self):
+ "More to document existing (unexpected?) behavior than anything else."
+ sniffer = csv.Sniffer()
+ self.assertFalse(sniffer.has_header(self.sample10))
+ self.assertFalse(sniffer.has_header(self.sample11))
+
def test_has_header(self):
sniffer = csv.Sniffer()
self.assertIs(sniffer.has_header(self.sample1), False)
@@ -1237,6 +1341,9 @@ def test__all__(self):
extra = {'__doc__', '__version__'}
support.check__all__(self, csv, ('csv', '_csv'), extra=extra)
+ def test_subclassable(self):
+ # issue 44089
+ class Foo(csv.Error): ...
if __name__ == '__main__':
unittest.main()
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
index 8bf48a6454d691..b550f4af555ce4 100644
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -6,7 +6,8 @@
import tempfile
import unittest
-from test.support import requires, verbose, SaveSignals, cpython_only
+from test.support import (requires, verbose, SaveSignals, cpython_only,
+ check_disallow_instantiation)
from test.support.import_helper import import_module
# Optionally test curses module. This currently requires that the
@@ -267,7 +268,12 @@ def test_output_character(self):
stdscr.echochar(b'A')
stdscr.echochar(65)
with self.assertRaises((UnicodeEncodeError, OverflowError)):
- stdscr.echochar('\u20ac')
+ # Unicode is not fully supported yet, but at least it does
+ # not crash.
+ # It is supposed to fail because either the character is
+ # not encodable with the current encoding, or it is encoded to
+ # a multibyte sequence.
+ stdscr.echochar('\u0114')
stdscr.echochar('A', curses.A_BOLD)
self.assertIs(stdscr.is_wintouched(), False)
@@ -1052,7 +1058,7 @@ def test_disallow_instantiation(self):
# Ensure that the type disallows instantiation (bpo-43916)
w = curses.newwin(10, 10)
panel = curses.panel.new_panel(w)
- self.assertRaises(TypeError, type(panel))
+ check_disallow_instantiation(self, type(panel))
@requires_curses_func('is_term_resized')
def test_is_term_resized(self):
diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index 8e645aeb4a7503..d6170eba4176d4 100644
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -5,12 +5,15 @@
from dataclasses import *
import abc
+import io
import pickle
import inspect
import builtins
+import types
+import traceback
import unittest
from unittest.mock import Mock
-from typing import ClassVar, Any, List, Union, Tuple, Dict, Generic, TypeVar, Optional
+from typing import ClassVar, Any, List, Union, Tuple, Dict, Generic, TypeVar, Optional, Protocol
from typing import get_type_hints
from collections import deque, OrderedDict, namedtuple
from functools import total_ordering
@@ -66,6 +69,24 @@ def test_field_repr(self):
self.assertEqual(repr_output, expected_output)
+ def test_field_recursive_repr(self):
+ rec_field = field()
+ rec_field.type = rec_field
+ rec_field.name = "id"
+ repr_output = repr(rec_field)
+
+ self.assertIn(",type=...,", repr_output)
+
+ def test_recursive_annotation(self):
+ class C:
+ pass
+
+ @dataclass
+ class D:
+ C: C = field()
+
+ self.assertIn(",type=...,", repr(D.__dataclass_fields__["C"]))
+
def test_named_init_params(self):
@dataclass
class C:
@@ -229,6 +250,14 @@ class C:
c = C('foo')
self.assertEqual(c.object, 'foo')
+ def test_field_named_BUILTINS_frozen(self):
+ # gh-96151
+ @dataclass(frozen=True)
+ class C:
+ BUILTINS: int
+ c = C(5)
+ self.assertEqual(c.BUILTINS, 5)
+
def test_field_named_like_builtin(self):
# Attribute names can shadow built-in names
# since code generation is used.
@@ -1126,6 +1155,10 @@ def test_init_var_preserve_type(self):
self.assertEqual(repr(InitVar[int]), 'dataclasses.InitVar[int]')
self.assertEqual(repr(InitVar[List[int]]),
'dataclasses.InitVar[typing.List[int]]')
+ self.assertEqual(repr(InitVar[list[int]]),
+ 'dataclasses.InitVar[list[int]]')
+ self.assertEqual(repr(InitVar[int|str]),
+ 'dataclasses.InitVar[int | str]')
def test_init_var_inheritance(self):
# Note that this deliberately tests that a dataclass need not
@@ -1350,6 +1383,17 @@ class B:
with self.assertRaisesRegex(TypeError, 'should be called on dataclass instances'):
replace(obj, x=0)
+ def test_is_dataclass_genericalias(self):
+ @dataclass
+ class A(types.GenericAlias):
+ origin: type
+ args: type
+ self.assertTrue(is_dataclass(A))
+ a = A(list, int)
+ self.assertTrue(is_dataclass(type(a)))
+ self.assertTrue(is_dataclass(a))
+
+
def test_helper_fields_with_class_instance(self):
# Check that we can call fields() on either a class or instance,
# and get back the same thing.
@@ -1372,6 +1416,16 @@ class C: pass
with self.assertRaisesRegex(TypeError, 'dataclass type or instance'):
fields(C())
+ def test_clean_traceback_from_fields_exception(self):
+ stdout = io.StringIO()
+ try:
+ fields(object)
+ except TypeError as exc:
+ traceback.print_exception(exc, file=stdout)
+ printed_traceback = stdout.getvalue()
+ self.assertNotIn("AttributeError", printed_traceback)
+ self.assertNotIn("__dataclass_fields__", printed_traceback)
+
def test_helper_asdict(self):
# Basic tests for asdict(), it should return a new dictionary.
@dataclass
@@ -1907,7 +1961,7 @@ class Parent(Generic[T]):
# Check MRO resolution.
self.assertEqual(Child.__mro__, (Child, Parent, Generic, object))
- def test_dataclassses_pickleable(self):
+ def test_dataclasses_pickleable(self):
global P, Q, R
@dataclass
class P:
@@ -2113,12 +2167,12 @@ class C(B):
self.assertEqual(c.z, 100)
def test_no_init(self):
- dataclass(init=False)
+ @dataclass(init=False)
class C:
i: int = 0
self.assertEqual(C().i, 0)
- dataclass(init=False)
+ @dataclass(init=False)
class C:
i: int = 2
def __init__(self):
@@ -2150,6 +2204,26 @@ def __init__(self, x):
self.x = 2 * x
self.assertEqual(C(5).x, 10)
+ def test_inherit_from_protocol(self):
+ # Dataclasses inheriting from protocol should preserve their own `__init__`.
+ # See bpo-45081.
+
+ class P(Protocol):
+ a: int
+
+ @dataclass
+ class C(P):
+ a: int
+
+ self.assertEqual(C(5).a, 5)
+
+ @dataclass
+ class D(P):
+ def __init__(self, a):
+ self.a = a * 2
+
+ self.assertEqual(D(5).a, 10)
+
class TestRepr(unittest.TestCase):
def test_repr(self):
@@ -2839,13 +2913,48 @@ class FrozenSlotsClass:
foo: str
bar: int
+ @dataclass(frozen=True)
+ class FrozenWithoutSlotsClass:
+ foo: str
+ bar: int
+
def test_frozen_pickle(self):
# bpo-43999
- assert self.FrozenSlotsClass.__slots__ == ("foo", "bar")
- p = pickle.dumps(self.FrozenSlotsClass("a", 1))
- assert pickle.loads(p) == self.FrozenSlotsClass("a", 1)
+ self.assertEqual(self.FrozenSlotsClass.__slots__, ("foo", "bar"))
+ for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+ with self.subTest(proto=proto):
+ obj = self.FrozenSlotsClass("a", 1)
+ p = pickle.loads(pickle.dumps(obj, protocol=proto))
+ self.assertIsNot(obj, p)
+ self.assertEqual(obj, p)
+
+ obj = self.FrozenWithoutSlotsClass("a", 1)
+ p = pickle.loads(pickle.dumps(obj, protocol=proto))
+ self.assertIsNot(obj, p)
+ self.assertEqual(obj, p)
+
+ def test_slots_with_default_no_init(self):
+ # Originally reported in bpo-44649.
+ @dataclass(slots=True)
+ class A:
+ a: str
+ b: str = field(default='b', init=False)
+
+ obj = A("a")
+ self.assertEqual(obj.a, 'a')
+ self.assertEqual(obj.b, 'b')
+
+ def test_slots_with_default_factory_no_init(self):
+ # Originally reported in bpo-44649.
+ @dataclass(slots=True)
+ class A:
+ a: str
+ b: str = field(default_factory=lambda:'b', init=False)
+ obj = A("a")
+ self.assertEqual(obj.a, 'a')
+ self.assertEqual(obj.b, 'b')
class TestDescriptors(unittest.TestCase):
def test_set_name(self):
@@ -2918,6 +3027,115 @@ class C:
self.assertEqual(D.__set_name__.call_count, 1)
+ def test_init_calls_set(self):
+ class D:
+ pass
+
+ D.__set__ = Mock()
+
+ @dataclass
+ class C:
+ i: D = D()
+
+ # Make sure D.__set__ is called.
+ D.__set__.reset_mock()
+ c = C(5)
+ self.assertEqual(D.__set__.call_count, 1)
+
+ def test_getting_field_calls_get(self):
+ class D:
+ pass
+
+ D.__set__ = Mock()
+ D.__get__ = Mock()
+
+ @dataclass
+ class C:
+ i: D = D()
+
+ c = C(5)
+
+ # Make sure D.__get__ is called.
+ D.__get__.reset_mock()
+ value = c.i
+ self.assertEqual(D.__get__.call_count, 1)
+
+ def test_setting_field_calls_set(self):
+ class D:
+ pass
+
+ D.__set__ = Mock()
+
+ @dataclass
+ class C:
+ i: D = D()
+
+ c = C(5)
+
+ # Make sure D.__set__ is called.
+ D.__set__.reset_mock()
+ c.i = 10
+ self.assertEqual(D.__set__.call_count, 1)
+
+ def test_setting_uninitialized_descriptor_field(self):
+ class D:
+ pass
+
+ D.__set__ = Mock()
+
+ @dataclass
+ class C:
+ i: D
+
+ # D.__set__ is not called because there's no D instance to call it on
+ D.__set__.reset_mock()
+ c = C(5)
+ self.assertEqual(D.__set__.call_count, 0)
+
+ # D.__set__ still isn't called after setting i to an instance of D
+ # because descriptors don't behave like that when stored as instance vars
+ c.i = D()
+ c.i = 5
+ self.assertEqual(D.__set__.call_count, 0)
+
+ def test_default_value(self):
+ class D:
+ def __get__(self, instance: Any, owner: object) -> int:
+ if instance is None:
+ return 100
+
+ return instance._x
+
+ def __set__(self, instance: Any, value: int) -> None:
+ instance._x = value
+
+ @dataclass
+ class C:
+ i: D = D()
+
+ c = C()
+ self.assertEqual(c.i, 100)
+
+ c = C(5)
+ self.assertEqual(c.i, 5)
+
+ def test_no_default_value(self):
+ class D:
+ def __get__(self, instance: Any, owner: object) -> int:
+ if instance is None:
+ raise AttributeError()
+
+ return instance._x
+
+ def __set__(self, instance: Any, value: int) -> None:
+ instance._x = value
+
+ @dataclass
+ class C:
+ i: D = D()
+
+ with self.assertRaisesRegex(TypeError, 'missing 1 required positional argument'):
+ c = C()
class TestStringAnnotations(unittest.TestCase):
def test_classvar(self):
@@ -3675,7 +3893,7 @@ class B:
with self.assertRaisesRegex(TypeError, msg):
B(3, 4, 5)
- # Explicitely make a field that follows KW_ONLY be non-keyword-only.
+ # Explicitly make a field that follows KW_ONLY be non-keyword-only.
@dataclass
class C:
a: int
@@ -3831,5 +4049,16 @@ class A:
c: int = 1
d: int
+ def test_make_dataclass(self):
+ A = make_dataclass("A", ['a'], kw_only=True)
+ self.assertTrue(fields(A)[0].kw_only)
+
+ B = make_dataclass("B",
+ ['a', ('b', int, field(kw_only=False))],
+ kw_only=True)
+ self.assertTrue(fields(B)[0].kw_only)
+ self.assertFalse(fields(B)[1].kw_only)
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/Lib/test/test_datetime.py b/Lib/test/test_datetime.py
index bdb9f02e5756a2..7f9094fa7bd4e6 100644
--- a/Lib/test/test_datetime.py
+++ b/Lib/test/test_datetime.py
@@ -1,59 +1,57 @@
import unittest
import sys
-from test.support import run_unittest
from test.support.import_helper import import_fresh_module
TESTS = 'test.datetimetester'
-try:
- pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'],
- blocked=['_datetime'])
- fast_tests = import_fresh_module(TESTS, fresh=['datetime',
- '_datetime', '_strptime'])
-finally:
- # XXX: import_fresh_module() is supposed to leave sys.module cache untouched,
- # XXX: but it does not, so we have to cleanup ourselves.
- for modname in ['datetime', '_datetime', '_strptime']:
- sys.modules.pop(modname, None)
-test_modules = [pure_tests, fast_tests]
-test_suffixes = ["_Pure", "_Fast"]
-# XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might
-# not believe this, but in spite of all the sys.modules trickery running a _Pure
-# test last will leave a mix of pure and native datetime stuff lying around.
-all_test_classes = []
+def load_tests(loader, tests, pattern):
+ try:
+ pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'],
+ blocked=['_datetime'])
+ fast_tests = import_fresh_module(TESTS, fresh=['datetime',
+ '_datetime', '_strptime'])
+ finally:
+ # XXX: import_fresh_module() is supposed to leave sys.module cache untouched,
+ # XXX: but it does not, so we have to cleanup ourselves.
+ for modname in ['datetime', '_datetime', '_strptime']:
+ sys.modules.pop(modname, None)
-for module, suffix in zip(test_modules, test_suffixes):
- test_classes = []
- for name, cls in module.__dict__.items():
- if not isinstance(cls, type):
- continue
- if issubclass(cls, unittest.TestCase):
- test_classes.append(cls)
- elif issubclass(cls, unittest.TestSuite):
- suit = cls()
- test_classes.extend(type(test) for test in suit)
- test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__)
- for cls in test_classes:
- cls.__name__ += suffix
- cls.__qualname__ += suffix
- @classmethod
- def setUpClass(cls_, module=module):
- cls_._save_sys_modules = sys.modules.copy()
- sys.modules[TESTS] = module
- sys.modules['datetime'] = module.datetime_module
- sys.modules['_strptime'] = module._strptime
- @classmethod
- def tearDownClass(cls_):
- sys.modules.clear()
- sys.modules.update(cls_._save_sys_modules)
- cls.setUpClass = setUpClass
- cls.tearDownClass = tearDownClass
- all_test_classes.extend(test_classes)
+ test_modules = [pure_tests, fast_tests]
+ test_suffixes = ["_Pure", "_Fast"]
+ # XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might
+ # not believe this, but in spite of all the sys.modules trickery running a _Pure
+ # test last will leave a mix of pure and native datetime stuff lying around.
+ for module, suffix in zip(test_modules, test_suffixes):
+ test_classes = []
+ for name, cls in module.__dict__.items():
+ if not isinstance(cls, type):
+ continue
+ if issubclass(cls, unittest.TestCase):
+ test_classes.append(cls)
+ elif issubclass(cls, unittest.TestSuite):
+ suit = cls()
+ test_classes.extend(type(test) for test in suit)
+ test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__)
+ for cls in test_classes:
+ cls.__name__ += suffix
+ cls.__qualname__ += suffix
+ @classmethod
+ def setUpClass(cls_, module=module):
+ cls_._save_sys_modules = sys.modules.copy()
+ sys.modules[TESTS] = module
+ sys.modules['datetime'] = module.datetime_module
+ sys.modules['_strptime'] = module._strptime
+ @classmethod
+ def tearDownClass(cls_):
+ sys.modules.clear()
+ sys.modules.update(cls_._save_sys_modules)
+ cls.setUpClass = setUpClass
+ cls.tearDownClass = tearDownClass
+ tests.addTests(loader.loadTestsFromTestCase(cls))
+ return tests
-def test_main():
- run_unittest(*all_test_classes)
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_dbm.py b/Lib/test/test_dbm.py
index e02d1e16ae3da7..16a023084e984a 100644
--- a/Lib/test/test_dbm.py
+++ b/Lib/test/test_dbm.py
@@ -1,23 +1,21 @@
"""Test script for the dbm.open function based on testdumbdbm.py"""
import unittest
-import glob
+import dbm
+import os
from test.support import import_helper
from test.support import os_helper
-# Skip tests if dbm module doesn't exist.
-dbm = import_helper.import_module('dbm')
-
try:
from dbm import ndbm
except ImportError:
ndbm = None
-_fname = os_helper.TESTFN
+dirname = os_helper.TESTFN
+_fname = os.path.join(dirname, os_helper.TESTFN)
#
-# Iterates over every database module supported by dbm currently available,
-# setting dbm to use each in turn, and yielding that module
+# Iterates over every database module supported by dbm currently available.
#
def dbm_iterator():
for name in dbm._names:
@@ -31,11 +29,12 @@ def dbm_iterator():
#
# Clean up all scratch databases we might have created during testing
#
-def delete_files():
- # we don't know the precise name the underlying database uses
- # so we use glob to locate all names
- for f in glob.glob(glob.escape(_fname) + "*"):
- os_helper.unlink(f)
+def cleaunup_test_dir():
+ os_helper.rmtree(dirname)
+
+def setup_test_dir():
+ cleaunup_test_dir()
+ os.mkdir(dirname)
class AnyDBMTestCase:
@@ -134,80 +133,67 @@ def read_helper(self, f):
for key in self._dict:
self.assertEqual(self._dict[key], f[key.encode("ascii")])
- def tearDown(self):
- delete_files()
+ def test_keys(self):
+ with dbm.open(_fname, 'c') as d:
+ self.assertEqual(d.keys(), [])
+ a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')]
+ for k, v in a:
+ d[k] = v
+ self.assertEqual(sorted(d.keys()), sorted(k for (k, v) in a))
+ for k, v in a:
+ self.assertIn(k, d)
+ self.assertEqual(d[k], v)
+ self.assertNotIn(b'xxx', d)
+ self.assertRaises(KeyError, lambda: d[b'xxx'])
def setUp(self):
+ self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod)
dbm._defaultmod = self.module
- delete_files()
+ self.addCleanup(cleaunup_test_dir)
+ setup_test_dir()
class WhichDBTestCase(unittest.TestCase):
def test_whichdb(self):
+ self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod)
for module in dbm_iterator():
# Check whether whichdb correctly guesses module name
# for databases opened with "module" module.
- # Try with empty files first
name = module.__name__
- if name == 'dbm.dumb':
- continue # whichdb can't support dbm.dumb
- delete_files()
- f = module.open(_fname, 'c')
- f.close()
+ setup_test_dir()
+ dbm._defaultmod = module
+ # Try with empty files first
+ with module.open(_fname, 'c'): pass
self.assertEqual(name, self.dbm.whichdb(_fname))
# Now add a key
- f = module.open(_fname, 'w')
- f[b"1"] = b"1"
- # and test that we can find it
- self.assertIn(b"1", f)
- # and read it
- self.assertEqual(f[b"1"], b"1")
- f.close()
+ with module.open(_fname, 'w') as f:
+ f[b"1"] = b"1"
+ # and test that we can find it
+ self.assertIn(b"1", f)
+ # and read it
+ self.assertEqual(f[b"1"], b"1")
self.assertEqual(name, self.dbm.whichdb(_fname))
@unittest.skipUnless(ndbm, reason='Test requires ndbm')
def test_whichdb_ndbm(self):
# Issue 17198: check that ndbm which is referenced in whichdb is defined
- db_file = '{}_ndbm.db'.format(_fname)
- with open(db_file, 'w'):
- self.addCleanup(os_helper.unlink, db_file)
- self.assertIsNone(self.dbm.whichdb(db_file[:-3]))
-
- def tearDown(self):
- delete_files()
+ with open(_fname + '.db', 'wb'): pass
+ self.assertIsNone(self.dbm.whichdb(_fname))
def setUp(self):
- delete_files()
- self.filename = os_helper.TESTFN
- self.d = dbm.open(self.filename, 'c')
- self.d.close()
+ self.addCleanup(cleaunup_test_dir)
+ setup_test_dir()
self.dbm = import_helper.import_fresh_module('dbm')
- def test_keys(self):
- self.d = dbm.open(self.filename, 'c')
- self.assertEqual(self.d.keys(), [])
- a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')]
- for k, v in a:
- self.d[k] = v
- self.assertEqual(sorted(self.d.keys()), sorted(k for (k, v) in a))
- for k, v in a:
- self.assertIn(k, self.d)
- self.assertEqual(self.d[k], v)
- self.assertNotIn(b'xxx', self.d)
- self.assertRaises(KeyError, lambda: self.d[b'xxx'])
- self.d.close()
-
-
-def load_tests(loader, tests, pattern):
- classes = []
- for mod in dbm_iterator():
- classes.append(type("TestCase-" + mod.__name__,
- (AnyDBMTestCase, unittest.TestCase),
- {'module': mod}))
- suites = [unittest.makeSuite(c) for c in classes]
-
- tests.addTests(suites)
- return tests
+
+for mod in dbm_iterator():
+ assert mod.__name__.startswith('dbm.')
+ suffix = mod.__name__[4:]
+ testname = f'TestCase_{suffix}'
+ globals()[testname] = type(testname,
+ (AnyDBMTestCase, unittest.TestCase),
+ {'module': mod})
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/test_dbm_gnu.py b/Lib/test/test_dbm_gnu.py
index b3e55728c8e70d..f39b0029373485 100644
--- a/Lib/test/test_dbm_gnu.py
+++ b/Lib/test/test_dbm_gnu.py
@@ -31,8 +31,7 @@ def tearDown(self):
def test_disallow_instantiation(self):
# Ensure that the type disallows instantiation (bpo-43916)
self.g = gdbm.open(filename, 'c')
- tp = type(self.g)
- self.assertRaises(TypeError, tp)
+ support.check_disallow_instantiation(self, type(self.g))
def test_key_methods(self):
self.g = gdbm.open(filename, 'c')
diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py
index 179a9ea7042fbf..172d5cada5fa71 100644
--- a/Lib/test/test_decimal.py
+++ b/Lib/test/test_decimal.py
@@ -34,9 +34,10 @@
import locale
from test.support import (run_unittest, run_doctest, is_resource_enabled,
requires_IEEE_754, requires_docstrings,
- requires_legacy_unicode_capi)
+ requires_legacy_unicode_capi, check_sanitizer)
from test.support import (TestFailed,
- run_with_locale, cpython_only)
+ run_with_locale, cpython_only,
+ darwin_malloc_err_warning)
from test.support.import_helper import import_fresh_module
from test.support import warnings_helper
import random
@@ -44,9 +45,13 @@
import threading
+if sys.platform == 'darwin':
+ darwin_malloc_err_warning('test_decimal')
+
+
C = import_fresh_module('decimal', fresh=['_decimal'])
P = import_fresh_module('decimal', blocked=['_decimal'])
-orig_sys_decimal = sys.modules['decimal']
+import decimal as orig_sys_decimal
# fractions module must import the correct decimal module.
cfractions = import_fresh_module('fractions', fresh=['fractions'])
@@ -1814,13 +1819,7 @@ def hashit(d):
# check that hash(d) == hash(int(d)) for integral values
for value in test_values:
- self.assertEqual(hashit(value), hashit(int(value)))
-
- #the same hash that to an int
- self.assertEqual(hashit(Decimal(23)), hashit(23))
- self.assertRaises(TypeError, hash, Decimal('sNaN'))
- self.assertTrue(hashit(Decimal('Inf')))
- self.assertTrue(hashit(Decimal('-Inf')))
+ self.assertEqual(hashit(value), hash(int(value)))
# check that the hashes of a Decimal float match when they
# represent exactly the same values
@@ -1829,7 +1828,7 @@ def hashit(d):
for s in test_strings:
f = float(s)
d = Decimal(s)
- self.assertEqual(hashit(f), hashit(d))
+ self.assertEqual(hashit(d), hash(f))
with localcontext() as c:
# check that the value of the hash doesn't depend on the
@@ -1850,6 +1849,19 @@ def hashit(d):
x = 1100 ** 1248
self.assertEqual(hashit(Decimal(x)), hashit(x))
+ def test_hash_method_nan(self):
+ Decimal = self.decimal.Decimal
+ self.assertRaises(TypeError, hash, Decimal('sNaN'))
+ value = Decimal('NaN')
+ self.assertEqual(hash(value), object.__hash__(value))
+ class H:
+ def __hash__(self):
+ return 42
+ class D(Decimal, H):
+ pass
+ value = D('NaN')
+ self.assertEqual(hash(value), object.__hash__(value))
+
def test_min_and_max_methods(self):
Decimal = self.decimal.Decimal
@@ -2451,6 +2463,15 @@ class CUsabilityTest(UsabilityTest):
class PyUsabilityTest(UsabilityTest):
decimal = P
+ def setUp(self):
+ super().setUp()
+ self._previous_int_limit = sys.get_int_max_str_digits()
+ sys.set_int_max_str_digits(7000)
+
+ def tearDown(self):
+ sys.set_int_max_str_digits(self._previous_int_limit)
+ super().tearDown()
+
class PythonAPItests(unittest.TestCase):
def test_abc(self):
@@ -4510,6 +4531,15 @@ class CCoverage(Coverage):
class PyCoverage(Coverage):
decimal = P
+ def setUp(self):
+ super().setUp()
+ self._previous_int_limit = sys.get_int_max_str_digits()
+ sys.set_int_max_str_digits(7000)
+
+ def tearDown(self):
+ sys.set_int_max_str_digits(self._previous_int_limit)
+ super().tearDown()
+
class PyFunctionality(unittest.TestCase):
"""Extra functionality in decimal.py"""
@@ -5488,6 +5518,9 @@ def __abs__(self):
# Issue 41540:
@unittest.skipIf(sys.platform.startswith("aix"),
"AIX: default ulimit: test is flaky because of extreme over-allocation")
+ @unittest.skipIf(check_sanitizer(address=True, memory=True),
+ "ASAN/MSAN sanitizer defaults to crashing "
+ "instead of returning NULL for malloc failure.")
def test_maxcontext_exact_arith(self):
# Make sure that exact operations do not raise MemoryError due
diff --git a/Lib/test/test_decorators.py b/Lib/test/test_decorators.py
index d4353457933f3d..57a741ffd29742 100644
--- a/Lib/test/test_decorators.py
+++ b/Lib/test/test_decorators.py
@@ -1,5 +1,6 @@
from test import support
import unittest
+from types import MethodType
def funcattrs(**kwds):
def decorate(func):
@@ -329,6 +330,91 @@ def outer(cls):
self.assertEqual(Class().inner(), 'spam')
self.assertEqual(Class().outer(), 'eggs')
+ def test_wrapped_classmethod_inside_classmethod(self):
+ class MyClassMethod1:
+ def __init__(self, func):
+ self.func = func
+
+ def __call__(self, cls):
+ if hasattr(self.func, '__get__'):
+ return self.func.__get__(cls, cls)()
+ return self.func(cls)
+
+ def __get__(self, instance, owner=None):
+ if owner is None:
+ owner = type(instance)
+ return MethodType(self, owner)
+
+ class MyClassMethod2:
+ def __init__(self, func):
+ if isinstance(func, classmethod):
+ func = func.__func__
+ self.func = func
+
+ def __call__(self, cls):
+ return self.func(cls)
+
+ def __get__(self, instance, owner=None):
+ if owner is None:
+ owner = type(instance)
+ return MethodType(self, owner)
+
+ for myclassmethod in [MyClassMethod1, MyClassMethod2]:
+ class A:
+ @myclassmethod
+ def f1(cls):
+ return cls
+
+ @classmethod
+ @myclassmethod
+ def f2(cls):
+ return cls
+
+ @myclassmethod
+ @classmethod
+ def f3(cls):
+ return cls
+
+ @classmethod
+ @classmethod
+ def f4(cls):
+ return cls
+
+ @myclassmethod
+ @MyClassMethod1
+ def f5(cls):
+ return cls
+
+ @myclassmethod
+ @MyClassMethod2
+ def f6(cls):
+ return cls
+
+ self.assertIs(A.f1(), A)
+ self.assertIs(A.f2(), A)
+ self.assertIs(A.f3(), A)
+ self.assertIs(A.f4(), A)
+ self.assertIs(A.f5(), A)
+ self.assertIs(A.f6(), A)
+ a = A()
+ self.assertIs(a.f1(), A)
+ self.assertIs(a.f2(), A)
+ self.assertIs(a.f3(), A)
+ self.assertIs(a.f4(), A)
+ self.assertIs(a.f5(), A)
+ self.assertIs(a.f6(), A)
+
+ def f(cls):
+ return cls
+
+ self.assertIs(myclassmethod(f).__get__(a)(), A)
+ self.assertIs(myclassmethod(f).__get__(a, A)(), A)
+ self.assertIs(myclassmethod(f).__get__(A, A)(), A)
+ self.assertIs(myclassmethod(f).__get__(A)(), type(A))
+ self.assertIs(classmethod(f).__get__(a)(), A)
+ self.assertIs(classmethod(f).__get__(a, A)(), A)
+ self.assertIs(classmethod(f).__get__(A, A)(), A)
+ self.assertIs(classmethod(f).__get__(A)(), type(A))
class TestClassDecorators(unittest.TestCase):
diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py
index 93cc6ca4f44ecb..7886bbfea1f0d7 100644
--- a/Lib/test/test_deque.py
+++ b/Lib/test/test_deque.py
@@ -129,7 +129,8 @@ def __eq__(self, other):
self.assertEqual(d.count(None), 16)
def test_comparisons(self):
- d = deque('xabc'); d.popleft()
+ d = deque('xabc')
+ d.popleft()
for e in [d, deque('abc'), deque('ab'), deque(), list(d)]:
self.assertEqual(d==e, type(d)==type(e) and list(d)==list(e))
self.assertEqual(d!=e, not(type(d)==type(e) and list(d)==list(e)))
@@ -529,8 +530,8 @@ def test_repr(self):
self.assertEqual(repr(d)[-20:], '7, 198, 199, [...]])')
def test_init(self):
- self.assertRaises(TypeError, deque, 'abc', 2, 3);
- self.assertRaises(TypeError, deque, 1);
+ self.assertRaises(TypeError, deque, 'abc', 2, 3)
+ self.assertRaises(TypeError, deque, 1)
def test_hash(self):
self.assertRaises(TypeError, hash, deque('abc'))
@@ -868,6 +869,7 @@ def test_weakref(self):
p = weakref.proxy(d)
self.assertEqual(str(p), str(d))
d = None
+ support.gc_collect() # For PyPy or other GCs.
self.assertRaises(ReferenceError, str, p)
def test_strange_subclass(self):
diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py
index 3cb923ed0520a3..b174e7168f7db8 100644
--- a/Lib/test/test_descr.py
+++ b/Lib/test/test_descr.py
@@ -1303,6 +1303,12 @@ class X(object):
with self.assertRaises(AttributeError):
del X().a
+ # Inherit from object on purpose to check some backwards compatibility paths
+ class X(object):
+ __slots__ = "a"
+ with self.assertRaisesRegex(AttributeError, "'X' object has no attribute 'a'"):
+ X().a
+
def test_slots_special(self):
# Testing __dict__ and __weakref__ in __slots__...
class D(object):
@@ -2539,10 +2545,8 @@ def getdict(self):
m2instance.b = 2
m2instance.a = 1
self.assertEqual(m2instance.__dict__, "Not a dict!")
- try:
+ with self.assertRaises(TypeError):
dir(m2instance)
- except TypeError:
- pass
# Two essentially featureless objects, (Ellipsis just inherits stuff
# from object.
@@ -4056,7 +4060,7 @@ class D(C):
except TypeError:
pass
else:
- assert 0, "best_base calculation found wanting"
+ self.fail("best_base calculation found wanting")
def test_unsubclassable_types(self):
with self.assertRaises(TypeError):
@@ -4442,6 +4446,8 @@ def __getattr__(self, attr):
print("Oops!")
except RuntimeError:
pass
+ else:
+ self.fail("Didn't raise RuntimeError")
finally:
sys.stdout = test_stdout
@@ -4990,8 +4996,11 @@ def test_repr(self):
self.assertIn('{!r}: {!r}'.format(k, v), r)
-class PTypesLongInitTest(unittest.TestCase):
+class AAAPTypesLongInitTest(unittest.TestCase):
# This is in its own TestCase so that it can be run before any other tests.
+ # (Hence the 'AAA' in the test class name: to make it the first
+ # item in a list sorted by name, like
+ # unittest.TestLoader.getTestCaseNames() does.)
def test_pytype_long_ready(self):
# Testing SF bug 551412 ...
@@ -5714,7 +5723,7 @@ class A(metaclass=M):
def test_incomplete_super(self):
"""
- Attrubute lookup on a super object must be aware that
+ Attribute lookup on a super object must be aware that
its target type can be uninitialized (type->tp_mro == NULL).
"""
class M(DebugHelperMeta):
@@ -5728,13 +5737,23 @@ def mro(cls):
class A(metaclass=M):
pass
+ def test_disappearing_custom_mro(self):
+ """
+ gh-92112: A custom mro() returning a result conflicting with
+ __bases__ and deleting itself caused a double free.
+ """
+ class B:
+ pass
+
+ class M(DebugHelperMeta):
+ def mro(cls):
+ del M.mro
+ return (B,)
+
+ with self.assertRaises(TypeError):
+ class A(metaclass=M):
+ pass
-def test_main():
- # Run all local test cases, with PTypesLongInitTest first.
- support.run_unittest(PTypesLongInitTest, OperatorsTest,
- ClassPropertiesAndMethods, DictProxyTests,
- MiscTests, PicklingTests, SharedKeyTests,
- MroTest)
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_descrtut.py b/Lib/test/test_descrtut.py
index 8e25f58d7aa20e..c5fbcba2d6aea5 100644
--- a/Lib/test/test_descrtut.py
+++ b/Lib/test/test_descrtut.py
@@ -1,7 +1,7 @@
# This contains most of the executable examples from Guido's descr
# tutorial, once at
#
-# http://www.python.org/2.2/descrintro.html
+# https://www.python.org/download/releases/2.2.3/descrintro/
#
# A few examples left implicit in the writeup were fleshed out, a few were
# skipped due to lack of interest (e.g., faking super() by hand isn't
diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py
index 110c006862737b..85e0accb611b1d 100644
--- a/Lib/test/test_devpoll.py
+++ b/Lib/test/test_devpoll.py
@@ -6,7 +6,7 @@
import random
import select
import unittest
-from test.support import run_unittest, cpython_only
+from test.support import cpython_only
if not hasattr(select, 'devpoll') :
raise unittest.SkipTest('test works only on Solaris OS family')
@@ -138,8 +138,5 @@ def test_events_mask_overflow_c_limits(self):
self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1)
-def test_main():
- run_unittest(DevPollTests)
-
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py
index 666cd81e68d81a..8aad4a5967b34e 100644
--- a/Lib/test/test_dict.py
+++ b/Lib/test/test_dict.py
@@ -1051,7 +1051,7 @@ def test_splittable_pop(self):
@support.cpython_only
def test_splittable_pop_pending(self):
- """pop a pending key in a splitted table should not crash"""
+ """pop a pending key in a split table should not crash"""
a, b = self.make_shared_key_dict(2)
a['a'] = 4
@@ -1398,7 +1398,7 @@ def test_reversed(self):
self.assertRaises(StopIteration, next, r)
def test_reverse_iterator_for_empty_dict(self):
- # bpo-38525: revered iterator should work properly
+ # bpo-38525: reversed iterator should work properly
# empty dict is directly used for reference count test
self.assertEqual(list(reversed({})), [])
@@ -1465,7 +1465,7 @@ def test_dict_items_result_gc(self):
self.assertTrue(gc.is_tracked(next(it)))
@support.cpython_only
- def test_dict_items_result_gc(self):
+ def test_dict_items_result_gc_reversed(self):
# Same as test_dict_items_result_gc above, but reversed.
it = reversed({None: []}.items())
gc.collect()
diff --git a/Lib/test/test_dict_version.py b/Lib/test/test_dict_version.py
index 8cdccad0d79ab6..243084c75c42bc 100644
--- a/Lib/test/test_dict_version.py
+++ b/Lib/test/test_dict_version.py
@@ -1,5 +1,5 @@
"""
-Test implementation of the PEP 509: dictionary versionning.
+Test implementation of the PEP 509: dictionary versioning.
"""
import unittest
from test.support import import_helper
diff --git a/Lib/test/test_dictviews.py b/Lib/test/test_dictviews.py
index be271bebaaf1ff..dae93740d46c7c 100644
--- a/Lib/test/test_dictviews.py
+++ b/Lib/test/test_dictviews.py
@@ -320,6 +320,9 @@ def test_abc_registry(self):
self.assertIsInstance(d.values(), collections.abc.ValuesView)
self.assertIsInstance(d.values(), collections.abc.MappingView)
self.assertIsInstance(d.values(), collections.abc.Sized)
+ self.assertIsInstance(d.values(), collections.abc.Collection)
+ self.assertIsInstance(d.values(), collections.abc.Iterable)
+ self.assertIsInstance(d.values(), collections.abc.Container)
self.assertIsInstance(d.items(), collections.abc.ItemsView)
self.assertIsInstance(d.items(), collections.abc.MappingView)
diff --git a/Lib/test/test_difflib.py b/Lib/test/test_difflib.py
index 9a24e00f64b4a4..6afd90af8442ad 100644
--- a/Lib/test/test_difflib.py
+++ b/Lib/test/test_difflib.py
@@ -1,5 +1,5 @@
import difflib
-from test.support import run_unittest, findfile
+from test.support import findfile
import unittest
import doctest
import sys
@@ -547,12 +547,14 @@ def test_longest_match_with_popular_chars(self):
self.assertFalse(self.longer_match_exists(a, b, match.size))
-def test_main():
+def setUpModule():
difflib.HtmlDiff._default_prefix = 0
- Doctests = doctest.DocTestSuite(difflib)
- run_unittest(
- TestWithAscii, TestAutojunk, TestSFpatches, TestSFbugs,
- TestOutputFormat, TestBytes, TestJunkAPIs, TestFindLongest, Doctests)
+
+
+def load_tests(loader, tests, pattern):
+ tests.addTest(doctest.DocTestSuite(difflib))
+ return tests
+
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py
index 21c68d488c33d3..69f7db7b923b9f 100644
--- a/Lib/test/test_dis.py
+++ b/Lib/test/test_dis.py
@@ -180,6 +180,23 @@ def bug42562():
2 RETURN_VALUE
"""
+# Extended arg followed by NOP
+code_bug_45757 = bytes([
+ 0x90, 0x01, # EXTENDED_ARG 0x01
+ 0x09, 0xFF, # NOP 0xFF
+ 0x90, 0x01, # EXTENDED_ARG 0x01
+ 0x64, 0x29, # LOAD_CONST 0x29
+ 0x53, 0x00, # RETURN_VALUE 0x00
+ ])
+
+dis_bug_45757 = """\
+ 0 EXTENDED_ARG 1
+ 2 NOP
+ 4 EXTENDED_ARG 1
+ 6 LOAD_CONST 297 (297)
+ 8 RETURN_VALUE
+"""
+
_BIG_LINENO_FORMAT = """\
%3d 0 LOAD_GLOBAL 0 (spam)
2 POP_TOP
@@ -534,6 +551,10 @@ def test_bug_1333982(self):
def test_bug_42562(self):
self.do_disassembly_test(bug42562, dis_bug42562)
+ def test_bug_45757(self):
+ # Extended arg followed by NOP
+ self.do_disassembly_test(code_bug_45757, dis_bug_45757)
+
def test_big_linenos(self):
def func(count):
namespace = {}
@@ -686,13 +707,10 @@ def get_disassembly(self, func, lasti=-1, wrapper=True, **kwargs):
return output.getvalue()
-if sys.flags.optimize:
+if dis.code_info.__doc__ is None:
code_info_consts = "0: None"
else:
- code_info_consts = (
- """0: 'Formatted details of methods, functions, or code.'
- 1: None"""
-)
+ code_info_consts = "0: 'Formatted details of methods, functions, or code.'"
code_info_code_info = f"""\
Name: code_info
@@ -816,7 +834,6 @@ def f(c=c):
Constants:
0: 0
1: 1
- 2: None
Names:
0: x"""
@@ -1027,7 +1044,7 @@ def jumpy():
Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', argrepr='>', offset=34, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=21, argval=42, argrepr='to 42', offset=36, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=38, starts_line=8, is_jump_target=False),
- Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=26, argval=52, argrepr='to 52', offset=40, starts_line=None, is_jump_target=False),
+ Instruction(opname='JUMP_FORWARD', opcode=110, arg=5, argval=52, argrepr='to 52', offset=40, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=4, argval=8, argrepr='to 8', offset=42, starts_line=7, is_jump_target=True),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=44, starts_line=10, is_jump_target=True),
Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval='I can haz else clause?', argrepr="'I can haz else clause?'", offset=46, starts_line=None, is_jump_target=False),
@@ -1052,24 +1069,24 @@ def jumpy():
Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, argrepr='4', offset=84, starts_line=None, is_jump_target=False),
Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', argrepr='<', offset=86, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=46, argval=92, argrepr='to 92', offset=88, starts_line=None, is_jump_target=False),
- Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=52, argval=104, argrepr='to 104', offset=90, starts_line=17, is_jump_target=False),
+ Instruction(opname='JUMP_FORWARD', opcode=110, arg=6, argval=104, argrepr='to 104', offset=90, starts_line=17, is_jump_target=False),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=92, starts_line=11, is_jump_target=True),
Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=28, argval=56, argrepr='to 56', offset=94, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=96, starts_line=19, is_jump_target=True),
Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval='Who let lolcatz into this test suite?', argrepr="'Who let lolcatz into this test suite?'", offset=98, starts_line=None, is_jump_target=False),
Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=100, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=102, starts_line=None, is_jump_target=False),
- Instruction(opname='SETUP_FINALLY', opcode=122, arg=48, argval=202, argrepr='to 202', offset=104, starts_line=20, is_jump_target=True),
+ Instruction(opname='SETUP_FINALLY', opcode=122, arg=63, argval=232, argrepr='to 232', offset=104, starts_line=20, is_jump_target=True),
Instruction(opname='SETUP_FINALLY', opcode=122, arg=6, argval=120, argrepr='to 120', offset=106, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1, argrepr='1', offset=108, starts_line=21, is_jump_target=False),
Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval=0, argrepr='0', offset=110, starts_line=None, is_jump_target=False),
Instruction(opname='BINARY_TRUE_DIVIDE', opcode=27, arg=None, argval=None, argrepr='', offset=112, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=114, starts_line=None, is_jump_target=False),
Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=116, starts_line=None, is_jump_target=False),
- Instruction(opname='JUMP_FORWARD', opcode=110, arg=12, argval=144, argrepr='to 144', offset=118, starts_line=None, is_jump_target=False),
+ Instruction(opname='JUMP_FORWARD', opcode=110, arg=13, argval=146, argrepr='to 146', offset=118, starts_line=None, is_jump_target=False),
Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, argrepr='', offset=120, starts_line=22, is_jump_target=True),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=2, argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=122, starts_line=None, is_jump_target=False),
- Instruction(opname='JUMP_IF_NOT_EXC_MATCH', opcode=121, arg=106, argval=212, argrepr='to 212', offset=124, starts_line=None, is_jump_target=False),
+ Instruction(opname='JUMP_IF_NOT_EXC_MATCH', opcode=121, arg=72, argval=144, argrepr='to 144', offset=124, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=126, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=128, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=130, starts_line=None, is_jump_target=False),
@@ -1078,42 +1095,56 @@ def jumpy():
Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=136, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=138, starts_line=None, is_jump_target=False),
Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=140, starts_line=None, is_jump_target=False),
- Instruction(opname='JUMP_FORWARD', opcode=110, arg=22, argval=188, argrepr='to 188', offset=142, starts_line=None, is_jump_target=False),
- Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=144, starts_line=25, is_jump_target=True),
- Instruction(opname='SETUP_WITH', opcode=143, arg=12, argval=172, argrepr='to 172', offset=146, starts_line=None, is_jump_target=False),
- Instruction(opname='STORE_FAST', opcode=125, arg=1, argval='dodgy', argrepr='dodgy', offset=148, starts_line=None, is_jump_target=False),
- Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=150, starts_line=26, is_jump_target=False),
- Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Never reach this', argrepr="'Never reach this'", offset=152, starts_line=None, is_jump_target=False),
- Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=154, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=156, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=158, starts_line=None, is_jump_target=False),
- Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=160, starts_line=25, is_jump_target=False),
- Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, argrepr='', offset=162, starts_line=None, is_jump_target=False),
+ Instruction(opname='JUMP_FORWARD', opcode=110, arg=30, argval=204, argrepr='to 204', offset=142, starts_line=None, is_jump_target=False),
+ Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=144, starts_line=22, is_jump_target=True),
+ Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=146, starts_line=25, is_jump_target=True),
+ Instruction(opname='SETUP_WITH', opcode=143, arg=12, argval=174, argrepr='to 174', offset=148, starts_line=None, is_jump_target=False),
+ Instruction(opname='STORE_FAST', opcode=125, arg=1, argval='dodgy', argrepr='dodgy', offset=150, starts_line=None, is_jump_target=False),
+ Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=152, starts_line=26, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Never reach this', argrepr="'Never reach this'", offset=154, starts_line=None, is_jump_target=False),
+ Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=156, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=158, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=160, starts_line=None, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=162, starts_line=25, is_jump_target=False),
Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, argrepr='', offset=164, starts_line=None, is_jump_target=False),
- Instruction(opname='CALL_FUNCTION', opcode=131, arg=3, argval=3, argrepr='', offset=166, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=168, starts_line=None, is_jump_target=False),
- Instruction(opname='JUMP_FORWARD', opcode=110, arg=8, argval=188, argrepr='to 188', offset=170, starts_line=None, is_jump_target=False),
- Instruction(opname='WITH_EXCEPT_START', opcode=49, arg=None, argval=None, argrepr='', offset=172, starts_line=None, is_jump_target=True),
- Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=89, argval=178, argrepr='to 178', offset=174, starts_line=None, is_jump_target=False),
- Instruction(opname='RERAISE', opcode=119, arg=1, argval=1, argrepr='', offset=176, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=178, starts_line=None, is_jump_target=True),
- Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=180, starts_line=None, is_jump_target=False),
+ Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, argrepr='', offset=166, starts_line=None, is_jump_target=False),
+ Instruction(opname='CALL_FUNCTION', opcode=131, arg=3, argval=3, argrepr='', offset=168, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=170, starts_line=None, is_jump_target=False),
+ Instruction(opname='JUMP_FORWARD', opcode=110, arg=22, argval=218, argrepr='to 218', offset=172, starts_line=None, is_jump_target=False),
+ Instruction(opname='WITH_EXCEPT_START', opcode=49, arg=None, argval=None, argrepr='', offset=174, starts_line=None, is_jump_target=True),
+ Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=90, argval=180, argrepr='to 180', offset=176, starts_line=None, is_jump_target=False),
+ Instruction(opname='RERAISE', opcode=119, arg=1, argval=1, argrepr='', offset=178, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=180, starts_line=None, is_jump_target=True),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=182, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=184, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=186, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=188, starts_line=None, is_jump_target=True),
- Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=190, starts_line=28, is_jump_target=False),
- Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=192, starts_line=None, is_jump_target=False),
- Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=194, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=196, starts_line=None, is_jump_target=False),
- Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=198, starts_line=None, is_jump_target=False),
- Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=200, starts_line=None, is_jump_target=False),
- Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=202, starts_line=None, is_jump_target=True),
- Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=204, starts_line=None, is_jump_target=False),
- Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=206, starts_line=None, is_jump_target=False),
- Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=208, starts_line=None, is_jump_target=False),
- Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=210, starts_line=None, is_jump_target=False),
- Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=212, starts_line=22, is_jump_target=True),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=184, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=186, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=188, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=190, starts_line=None, is_jump_target=False),
+ Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=192, starts_line=28, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=194, starts_line=None, is_jump_target=False),
+ Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=196, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=198, starts_line=None, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=200, starts_line=None, is_jump_target=False),
+ Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=202, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=204, starts_line=23, is_jump_target=True),
+ Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=206, starts_line=28, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=208, starts_line=None, is_jump_target=False),
+ Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=210, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=212, starts_line=None, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=214, starts_line=None, is_jump_target=False),
+ Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=216, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=218, starts_line=25, is_jump_target=True),
+ Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=220, starts_line=28, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=222, starts_line=None, is_jump_target=False),
+ Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=224, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=226, starts_line=None, is_jump_target=False),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=228, starts_line=None, is_jump_target=False),
+ Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=230, starts_line=None, is_jump_target=False),
+ Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=232, starts_line=None, is_jump_target=True),
+ Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=234, starts_line=None, is_jump_target=False),
+ Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=236, starts_line=None, is_jump_target=False),
+ Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=238, starts_line=None, is_jump_target=False),
+ Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=240, starts_line=None, is_jump_target=False),
]
# One last piece of inspect fodder to check the default line number handling
@@ -1234,5 +1265,46 @@ def test_assert_not_in_with_arg_in_bytecode(self):
with self.assertRaises(AssertionError):
self.assertNotInBytecode(code, "LOAD_CONST", 1)
+
+class TestDisTraceback(unittest.TestCase):
+ def setUp(self) -> None:
+ try: # We need to clean up existing tracebacks
+ del sys.last_traceback
+ except AttributeError:
+ pass
+ return super().setUp()
+
+ def get_disassembly(self, tb):
+ output = io.StringIO()
+ with contextlib.redirect_stdout(output):
+ dis.distb(tb)
+ return output.getvalue()
+
+ def test_distb_empty(self):
+ with self.assertRaises(RuntimeError):
+ dis.distb()
+
+ def test_distb_last_traceback(self):
+ # We need to have an existing last traceback in `sys`:
+ tb = get_tb()
+ sys.last_traceback = tb
+
+ self.assertEqual(self.get_disassembly(None), dis_traceback)
+
+ def test_distb_explicit_arg(self):
+ tb = get_tb()
+
+ self.assertEqual(self.get_disassembly(tb), dis_traceback)
+
+
+class TestDisTracebackWithFile(TestDisTraceback):
+ # Run the `distb` tests again, using the file arg instead of print
+ def get_disassembly(self, tb):
+ output = io.StringIO()
+ with contextlib.redirect_stdout(output):
+ dis.distb(tb, file=output)
+ return output.getvalue()
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/Lib/test/test_distutils.py b/Lib/test/test_distutils.py
index 90bfc70a133ae1..d82d2b6423433e 100644
--- a/Lib/test/test_distutils.py
+++ b/Lib/test/test_distutils.py
@@ -5,7 +5,7 @@
be run.
"""
-import warnings
+import unittest
from test import support
from test.support import warnings_helper
@@ -15,16 +15,14 @@
import distutils.tests
-def test_main():
- # used by regrtest
- support.run_unittest(distutils.tests.test_suite())
- support.reap_children()
-
-
def load_tests(*_):
# used by unittest
return distutils.tests.test_suite()
+def tearDownModule():
+ support.reap_children()
+
+
if __name__ == "__main__":
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py
index 6f51b1bc4f0d1f..ebd4ad91924cfb 100644
--- a/Lib/test/test_doctest.py
+++ b/Lib/test/test_doctest.py
@@ -15,10 +15,12 @@
import unittest
import tempfile
import shutil
+import types
import contextlib
# NOTE: There are some additional tests relating to interaction with
# zipimport in the test_zipimport_support test module.
+# There are also related tests in `test_doctest2` module.
######################################################################
## Sample Objects (used by test cases)
@@ -95,6 +97,17 @@ def a_classmethod(cls, v):
22
""")
+ a_class_attribute = 42
+
+ @classmethod
+ @property
+ def a_classmethod_property(cls):
+ """
+ >>> print(SampleClass.a_classmethod_property)
+ 42
+ """
+ return cls.a_class_attribute
+
class NestedClass:
"""
>>> x = SampleClass.NestedClass(5)
@@ -443,7 +456,7 @@ def basics(): r"""
>>> tests = finder.find(sample_func)
>>> print(tests) # doctest: +ELLIPSIS
- []
+ []
The exact name depends on how test_doctest was invoked, so allow for
leading path components.
@@ -500,6 +513,7 @@ def basics(): r"""
1 SampleClass.NestedClass.__init__
1 SampleClass.__init__
2 SampleClass.a_classmethod
+ 1 SampleClass.a_classmethod_property
1 SampleClass.a_property
1 SampleClass.a_staticmethod
1 SampleClass.double
@@ -555,6 +569,7 @@ def basics(): r"""
1 some_module.SampleClass.NestedClass.__init__
1 some_module.SampleClass.__init__
2 some_module.SampleClass.a_classmethod
+ 1 some_module.SampleClass.a_classmethod_property
1 some_module.SampleClass.a_property
1 some_module.SampleClass.a_staticmethod
1 some_module.SampleClass.double
@@ -596,6 +611,7 @@ def basics(): r"""
1 SampleClass.NestedClass.__init__
1 SampleClass.__init__
2 SampleClass.a_classmethod
+ 1 SampleClass.a_classmethod_property
1 SampleClass.a_property
1 SampleClass.a_staticmethod
1 SampleClass.double
@@ -616,11 +632,32 @@ def basics(): r"""
0 SampleClass.NestedClass.square
1 SampleClass.__init__
2 SampleClass.a_classmethod
+ 1 SampleClass.a_classmethod_property
1 SampleClass.a_property
1 SampleClass.a_staticmethod
1 SampleClass.double
1 SampleClass.get
+When used with `exclude_empty=False` we are also interested in line numbers
+of doctests that are empty.
+It used to be broken for quite some time until `bpo-28249`.
+
+ >>> from test import doctest_lineno
+ >>> tests = doctest.DocTestFinder(exclude_empty=False).find(doctest_lineno)
+ >>> for t in tests:
+ ... print('%5s %s' % (t.lineno, t.name))
+ None test.doctest_lineno
+ 22 test.doctest_lineno.ClassWithDocstring
+ 30 test.doctest_lineno.ClassWithDoctest
+ None test.doctest_lineno.ClassWithoutDocstring
+ None test.doctest_lineno.MethodWrapper
+ 39 test.doctest_lineno.MethodWrapper.method_with_docstring
+ 45 test.doctest_lineno.MethodWrapper.method_with_doctest
+ None test.doctest_lineno.MethodWrapper.method_without_docstring
+ 4 test.doctest_lineno.func_with_docstring
+ 12 test.doctest_lineno.func_with_doctest
+ None test.doctest_lineno.func_without_docstring
+
Turning off Recursion
~~~~~~~~~~~~~~~~~~~~~
DocTestFinder can be told not to look for tests in contained objects
@@ -698,6 +735,18 @@ def non_Python_modules(): r"""
class TestDocTestFinder(unittest.TestCase):
+ def test_issue35753(self):
+ # This import of `call` should trigger issue35753 when
+ # `support.run_doctest` is called due to unwrap failing,
+ # however with a patched doctest this should succeed.
+ from unittest.mock import call
+ dummy_module = types.ModuleType("dummy")
+ dummy_module.__dict__['inject_call'] = call
+ try:
+ support.run_doctest(dummy_module, verbosity=True)
+ except ValueError as e:
+ raise support.TestFailed("Doctest unwrap failed") from e
+
def test_empty_namespace_package(self):
pkg_name = 'doctest_empty_pkg'
with tempfile.TemporaryDirectory() as parent_dir:
@@ -3097,20 +3146,27 @@ def test_no_trailing_whitespace_stripping():
patches that contain trailing whitespace. More info on Issue 24746.
"""
-######################################################################
-## Main
-######################################################################
-def test_main():
- # Check the doctest cases in doctest itself:
- ret = support.run_doctest(doctest, verbosity=True)
+def test_run_doctestsuite_multiple_times():
+ """
+ It was not possible to run the same DocTestSuite multiple times
+ http://bugs.python.org/issue2604
+ http://bugs.python.org/issue9736
+
+ >>> import unittest
+ >>> import test.sample_doctest
+ >>> suite = doctest.DocTestSuite(test.sample_doctest)
+ >>> suite.run(unittest.TestResult())
+
+ >>> suite.run(unittest.TestResult())
+
+ """
- # Check the doctest cases defined here:
- from test import test_doctest
- support.run_doctest(test_doctest, verbosity=True)
- # Run unittests
- support.run_unittest(__name__)
+def load_tests(loader, tests, pattern):
+ tests.addTest(doctest.DocTestSuite(doctest))
+ tests.addTest(doctest.DocTestSuite())
+ return tests
def test_coverage(coverdir):
@@ -3123,8 +3179,9 @@ def test_coverage(coverdir):
r.write_results(show_missing=True, summary=True,
coverdir=coverdir)
+
if __name__ == '__main__':
if '-c' in sys.argv:
test_coverage('/tmp/doctest.cover')
else:
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_doctest2.py b/Lib/test/test_doctest2.py
index 347a143641071d..ab8a0696736e23 100644
--- a/Lib/test/test_doctest2.py
+++ b/Lib/test/test_doctest2.py
@@ -13,7 +13,6 @@
import sys
import unittest
-from test import support
if sys.flags.optimize >= 2:
raise unittest.SkipTest("Cannot test docstrings with -O2")
@@ -107,17 +106,21 @@ def clsm(cls, val):
"""
return val
-def test_main():
- from test import test_doctest2
- EXPECTED = 19
- f, t = support.run_doctest(test_doctest2)
- if t != EXPECTED:
- raise support.TestFailed("expected %d tests to run, not %d" %
- (EXPECTED, t))
+
+class Test(unittest.TestCase):
+ def test_testmod(self):
+ import doctest, sys
+ EXPECTED = 19
+ f, t = doctest.testmod(sys.modules[__name__])
+ if f:
+ self.fail("%d of %d doctests failed" % (f, t))
+ if t != EXPECTED:
+ self.fail("expected %d tests to run, not %d" % (EXPECTED, t))
+
# Pollute the namespace with a bunch of imported functions and classes,
# to make sure they don't get tested.
from doctest import *
if __name__ == '__main__':
- test_main()
+ unittest.main()
diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py
index 7d3e30cbee964a..77252502883112 100644
--- a/Lib/test/test_docxmlrpc.py
+++ b/Lib/test/test_docxmlrpc.py
@@ -133,7 +133,7 @@ def test_autolinking(self):
self.assertIn(
(b'
