From 4ed423723c5024d1f8f8964ee09496ac7ffba63d Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Thu, 27 Feb 2025 12:09:20 -0800 Subject: [PATCH 01/15] gh-80050: Update BufferedReader.read around non-blocking Synchronize `io.BufferedReader` and `io.BufferedIOBase` documentation with the current implementation. Focused on behavior around non-blocking streams. --- Doc/library/io.rst | 58 +++++++++++++++++++++------------------------- 1 file changed, 27 insertions(+), 31 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 0d8cc5171d5476..b72e0d5d66535f 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -568,15 +568,19 @@ I/O Base Classes .. method:: read(size=-1, /) - Read and return up to *size* bytes. If the argument is omitted, ``None``, - or negative, data is read and returned until EOF is reached. An empty - :class:`bytes` object is returned if the stream is already at EOF. + Read and return up to *size* bytes. If the argument is omitted, ``None``, + or negative read as much as possible. - If the argument is positive, and the underlying raw stream is not - interactive, multiple raw reads may be issued to satisfy the byte count - (unless EOF is reached first). But for interactive raw streams, at most - one raw read will be issued, and a short result does not imply that EOF is - imminent. + Reading as much as possible will use :meth:`~raw.readall` if available, + otherwise will read in a loop until read returns ``None``, a size-zero + ``bytes``, or a non-retryable error. For most streams this is to EOF, but + for non-blocking streams more data may become available. + + Less bytes may be returned than requested. An empty :class:`bytes` object + is returned if the stream is already at EOF. More than one read may be + made, and calls may be retried if specific errors are encountered, see + :meth:`os.read` and :pep:`475` for more details. Less than size bytes + being returned does not imply that EOF is imminent. A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. @@ -592,6 +596,11 @@ I/O Base Classes If *size* is ``-1`` (the default), an arbitrary number of bytes are returned (more than zero unless EOF is reached). + .. note:: + + When the underlying raw stream is non-blocking, a :exc:`BlockingIOError` + may be raised if a read operation cannot be completed immediately. + .. method:: readinto(b, /) Read bytes into a pre-allocated, writable @@ -737,14 +746,14 @@ than raw I/O does. .. method:: read1(size=-1, /) - In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. + In :class:`BytesIO`, this is the same as :meth:`io.BufferedIOBase.read`. .. versionchanged:: 3.7 The *size* argument is now optional. .. method:: readinto1(b, /) - In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.readinto`. + In :class:`BytesIO`, this is the same as :meth:`io.BufferedIOBase.readinto`. .. versionadded:: 3.5 @@ -767,34 +776,21 @@ than raw I/O does. .. method:: peek(size=0, /) - Return bytes from the stream without advancing the position. At most one - single read on the raw stream is done to satisfy the call. The number of - bytes returned may be less or more than requested. + Return bytes from the stream without advancing the position. The number of + bytes returned may be less or more than requested. If the underlying raw + stream is non-blocking and the operation would block, returns empty bytes. .. method:: read(size=-1, /) - Read and return *size* bytes, or if *size* is not given or negative, until - EOF or if the read call would block in non-blocking mode. - - .. note:: - - When the underlying raw stream is non-blocking, a :exc:`BlockingIOError` - may be raised if a read operation cannot be completed immediately. + In :class:`BufferedReader` this is the same as :meth:`io.BufferedIOBase.read` .. method:: read1(size=-1, /) - Read and return up to *size* bytes with only one call on the raw stream. - If at least one byte is buffered, only buffered bytes are returned. - Otherwise, one raw stream read call is made. + In :class:`BufferedReader` this is the same as :meth:`io.BufferedIOBase.read1` .. versionchanged:: 3.7 The *size* argument is now optional. - .. note:: - - When the underlying raw stream is non-blocking, a :exc:`BlockingIOError` - may be raised if a read operation cannot be completed immediately. - .. class:: BufferedWriter(raw, buffer_size=DEFAULT_BUFFER_SIZE) A buffered binary stream providing higher-level access to a writeable, non @@ -856,7 +852,7 @@ than raw I/O does. :data:`DEFAULT_BUFFER_SIZE`. :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods - except for :meth:`~BufferedIOBase.detach`, which raises + except for :meth:`io.BufferedIOBase.detach`, which raises :exc:`UnsupportedOperation`. .. warning:: @@ -1006,8 +1002,8 @@ Text I/O If *line_buffering* is ``True``, :meth:`~IOBase.flush` is implied when a call to write contains a newline character or a carriage return. - If *write_through* is ``True``, calls to :meth:`~BufferedIOBase.write` are guaranteed - not to be buffered: any data written on the :class:`TextIOWrapper` + If *write_through* is ``True``, calls to :meth:`io.BufferedIOBase.write` are + guaranteed not to be buffered: any data written on the :class:`TextIOWrapper` object is immediately handled to its underlying binary *buffer*. .. versionchanged:: 3.3 From b87870b84027d7bc24d7078616fb78fb00adc419 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Thu, 27 Feb 2025 12:09:20 -0800 Subject: [PATCH 02/15] Tweaks around raw.readall --- Doc/library/io.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index b72e0d5d66535f..2763e1dbf62fe4 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -571,7 +571,8 @@ I/O Base Classes Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative read as much as possible. - Reading as much as possible will use :meth:`~raw.readall` if available, + The default implementation will use :py:func:`raw.readall` if available ( + which should implement :meth:`RawIOBase.readall`), otherwise will read in a loop until read returns ``None``, a size-zero ``bytes``, or a non-retryable error. For most streams this is to EOF, but for non-blocking streams more data may become available. From 2119f7836e255af33a1519c2c5d059c58d8a5a63 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Thu, 27 Feb 2025 12:57:03 -0800 Subject: [PATCH 03/15] tweak readall, hopefully passes --- Doc/library/io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 2763e1dbf62fe4..3cd33ae270e825 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -571,7 +571,7 @@ I/O Base Classes Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative read as much as possible. - The default implementation will use :py:func:`raw.readall` if available ( + The default implementation will use ``raw.readall`` if available ( which should implement :meth:`RawIOBase.readall`), otherwise will read in a loop until read returns ``None``, a size-zero ``bytes``, or a non-retryable error. For most streams this is to EOF, but From ec7f02395ab42fe3dbea64cc8eb414e1d7246680 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Thu, 27 Feb 2025 14:15:47 -0800 Subject: [PATCH 04/15] tweaks --- Doc/library/io.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 3cd33ae270e825..7591574829356d 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -571,18 +571,19 @@ I/O Base Classes Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative read as much as possible. - The default implementation will use ``raw.readall`` if available ( - which should implement :meth:`RawIOBase.readall`), - otherwise will read in a loop until read returns ``None``, a size-zero - ``bytes``, or a non-retryable error. For most streams this is to EOF, but - for non-blocking streams more data may become available. - Less bytes may be returned than requested. An empty :class:`bytes` object is returned if the stream is already at EOF. More than one read may be made, and calls may be retried if specific errors are encountered, see :meth:`os.read` and :pep:`475` for more details. Less than size bytes being returned does not imply that EOF is imminent. + When reading as much as possible the default implementation will use + ``raw.readall`` if available (which should implement + :meth:`RawIOBase.readall`), otherwise will read in a loop until read + returns ``None``, a size-zero :clss:`bytes`, or a non-retryable error. For + most streams this is to EOF, but for non-blocking streams more data may + become available. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. From 11b754d0a2ab9a3a1e77e0d4ef590da6ee40dca4 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Thu, 27 Feb 2025 16:43:47 -0800 Subject: [PATCH 05/15] clss -> class --- Doc/library/io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 7591574829356d..fd847a1601f061 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -580,7 +580,7 @@ I/O Base Classes When reading as much as possible the default implementation will use ``raw.readall`` if available (which should implement :meth:`RawIOBase.readall`), otherwise will read in a loop until read - returns ``None``, a size-zero :clss:`bytes`, or a non-retryable error. For + returns ``None``, a size-zero :class:`bytes`, or a non-retryable error. For most streams this is to EOF, but for non-blocking streams more data may become available. From b6d319bbbb1957a1568caa29342c7c5b5e053fd2 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Thu, 27 Feb 2025 21:41:52 -0800 Subject: [PATCH 06/15] grammatical tweaking --- Doc/library/io.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index fd847a1601f061..8a08a523d3d684 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -573,14 +573,14 @@ I/O Base Classes Less bytes may be returned than requested. An empty :class:`bytes` object is returned if the stream is already at EOF. More than one read may be - made, and calls may be retried if specific errors are encountered, see + made and calls may be retried if specific errors are encountered, see :meth:`os.read` and :pep:`475` for more details. Less than size bytes being returned does not imply that EOF is imminent. When reading as much as possible the default implementation will use ``raw.readall`` if available (which should implement :meth:`RawIOBase.readall`), otherwise will read in a loop until read - returns ``None``, a size-zero :class:`bytes`, or a non-retryable error. For + returns ``None``, an empty :class:`bytes`, or a non-retryable error. For most streams this is to EOF, but for non-blocking streams more data may become available. From 59bd2a1cccc7b92ecd62799a290b309c15840a78 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Tue, 11 Mar 2025 16:37:28 -0700 Subject: [PATCH 07/15] Update to include behavior around None BufferedReader at least just forwards `None` from the underlying stream. 1. `BufferedReader.read(size=10)` `raw.read(size=10)` returns None 2. `BufferedReader.read()` without size, raw implements `readall()` which returns None 3. `BufferedReader.read()` without size, raw doesn't have readall so uses a loop that calls `raw.read()` which returns None. --- Doc/library/io.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 8a08a523d3d684..3a5e08642bf04d 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -584,8 +584,9 @@ I/O Base Classes most streams this is to EOF, but for non-blocking streams more data may become available. - A :exc:`BlockingIOError` is raised if the underlying raw stream is in - non blocking-mode, and has no data available at the moment. + The standard library :class:`io.BufferedReader` returns ``None`` if + reading would block and no data is available. Other implementations may + return ``None`` or raise :exc:`BlockingIOError`. .. method:: read1(size=-1, /) From fbbc382bd2183dc40a9d30d5711ee95bf2934fe8 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Tue, 11 Mar 2025 16:42:55 -0700 Subject: [PATCH 08/15] tweak --- Doc/library/io.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 3a5e08642bf04d..8ffdcb3c47176f 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -584,8 +584,8 @@ I/O Base Classes most streams this is to EOF, but for non-blocking streams more data may become available. - The standard library :class:`io.BufferedReader` returns ``None`` if - reading would block and no data is available. Other implementations may + The standard library :class:`io.BufferedReader` returns ``None`` if the + stream is non-blocking and no data is available. Other implementations may return ``None`` or raise :exc:`BlockingIOError`. .. method:: read1(size=-1, /) From dea91abf619def3403b06959d1412d66566d0845 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Wed, 12 Mar 2025 21:03:51 -0700 Subject: [PATCH 09/15] Tweak read and read1 None and BlockingIOError comments --- Doc/library/io.rst | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 8ffdcb3c47176f..fdeddbd7aa021d 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -584,25 +584,25 @@ I/O Base Classes most streams this is to EOF, but for non-blocking streams more data may become available. - The standard library :class:`io.BufferedReader` returns ``None`` if the - stream is non-blocking and no data is available. Other implementations may - return ``None`` or raise :exc:`BlockingIOError`. + .. note:: + + When the underlying raw stream is non-blocking, implementations may + either raise :exc:`BlockingIOError` or return ``None`` if no data is + available. :mod:`io` implementations return ``None``. .. method:: read1(size=-1, /) Read and return up to *size* bytes, with at most one call to the underlying raw stream's :meth:`~RawIOBase.read` (or - :meth:`~RawIOBase.readinto`) method. This can be useful if you are - implementing your own buffering on top of a :class:`BufferedIOBase` - object. - - If *size* is ``-1`` (the default), an arbitrary number of bytes are - returned (more than zero unless EOF is reached). + :meth:`~RawIOBase.readinto`) method. If *size* is ``-1`` or not provided, + the implementation will decide a size to use and at most one call will be + made. .. note:: - When the underlying raw stream is non-blocking, a :exc:`BlockingIOError` - may be raised if a read operation cannot be completed immediately. + When the underlying raw stream is non-blocking, implementations may + either raise :exc:`BlockingIOError` or return ``None`` if no data is + available. :mod:`io` implementations return ``None``. .. method:: readinto(b, /) From 7c4ef7ea360e4b5025d50fde024383151e0872da Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Wed, 12 Mar 2025 21:20:30 -0700 Subject: [PATCH 10/15] in read1 refer to EINTR retrying --- Doc/library/io.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index fdeddbd7aa021d..491eabf19b606d 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -592,11 +592,10 @@ I/O Base Classes .. method:: read1(size=-1, /) - Read and return up to *size* bytes, with at most one call to the - underlying raw stream's :meth:`~RawIOBase.read` (or - :meth:`~RawIOBase.readinto`) method. If *size* is ``-1`` or not provided, - the implementation will decide a size to use and at most one call will be - made. + Read and return up to *size* bytes, calilng :meth:`~raw.readinto`, + retrying if :py:const:`~errno.EINTR` is encountered per :pep:`475`. If + *size* is ``-1`` or not provided, the implementation will choose an + arbitrary value for *size*. .. note:: From c6374022dd7316bfcc42a179fa2bf893a48595e7 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Wed, 12 Mar 2025 21:26:22 -0700 Subject: [PATCH 11/15] fix doc lint around raw.readinto --- Doc/library/io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 491eabf19b606d..2d5e6c19013509 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -592,7 +592,7 @@ I/O Base Classes .. method:: read1(size=-1, /) - Read and return up to *size* bytes, calilng :meth:`~raw.readinto`, + Read and return up to *size* bytes, calilng ``raw.readinto``, retrying if :py:const:`~errno.EINTR` is encountered per :pep:`475`. If *size* is ``-1`` or not provided, the implementation will choose an arbitrary value for *size*. From 22bc9aaf347552706b73386772f4254c73e914d5 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Tue, 13 May 2025 12:19:44 -0700 Subject: [PATCH 12/15] personal review fixes --- Doc/library/io.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 2d5e6c19013509..28dfe4ad2e629c 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -592,10 +592,10 @@ I/O Base Classes .. method:: read1(size=-1, /) - Read and return up to *size* bytes, calilng ``raw.readinto``, - retrying if :py:const:`~errno.EINTR` is encountered per :pep:`475`. If - *size* is ``-1`` or not provided, the implementation will choose an - arbitrary value for *size*. + Read and return up to *size* bytes, calling :meth:`~RawIOBase.readinto` + which may retry if :py:const:`~errno.EINTR` is encountered per + :pep:`475`. If *size* is ``-1`` or not provided, the implementation will + choose an arbitrary value for *size*. .. note:: @@ -748,14 +748,14 @@ than raw I/O does. .. method:: read1(size=-1, /) - In :class:`BytesIO`, this is the same as :meth:`io.BufferedIOBase.read`. + In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. .. versionchanged:: 3.7 The *size* argument is now optional. .. method:: readinto1(b, /) - In :class:`BytesIO`, this is the same as :meth:`io.BufferedIOBase.readinto`. + In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.readinto`. .. versionadded:: 3.5 @@ -854,7 +854,7 @@ than raw I/O does. :data:`DEFAULT_BUFFER_SIZE`. :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods - except for :meth:`io.BufferedIOBase.detach`, which raises + except for :meth:`~BufferedIOBase.detach`, which raises :exc:`UnsupportedOperation`. .. warning:: @@ -1004,8 +1004,8 @@ Text I/O If *line_buffering* is ``True``, :meth:`~IOBase.flush` is implied when a call to write contains a newline character or a carriage return. - If *write_through* is ``True``, calls to :meth:`io.BufferedIOBase.write` are - guaranteed not to be buffered: any data written on the :class:`TextIOWrapper` + If *write_through* is ``True``, calls to :meth:`~BufferedIOBase.write` are guaranteed + not to be buffered: any data written on the :class:`TextIOWrapper` object is immediately handled to its underlying binary *buffer*. .. versionchanged:: 3.3 From 1c4a09cef8a0d9f3cde40301a4277da888809cb9 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Mon, 19 May 2025 17:57:17 -0400 Subject: [PATCH 13/15] Update BufferedIOBase class doc for non-blocking --- Doc/library/io.rst | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 28dfe4ad2e629c..ff4db2569fc84d 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -528,14 +528,13 @@ I/O Base Classes It inherits from :class:`IOBase`. The main difference with :class:`RawIOBase` is that methods :meth:`read`, - :meth:`readinto` and :meth:`write` will try (respectively) to read as much - input as requested or to consume all given output, at the expense of - making perhaps more than one system call. - - In addition, those methods can raise :exc:`BlockingIOError` if the - underlying raw stream is in non-blocking mode and cannot take or give - enough data; unlike their :class:`RawIOBase` counterparts, they will - never return ``None``. + :meth:`readinto` and :meth:`write` will try (respectively) to read + as much input as requested or to emit all provided data. + + In addition, if the underlying raw stream is in non-blocking mode, when the + system returns would block :meth:`write` will raise :exc:`BlockingIOError` + with :attr:`BlockingIOError.characters_written` and :meth:`read` will return + data read so far or ``None`` if no data is available. Besides, the :meth:`read` method does not have a default implementation that defers to :meth:`readinto`. From 8ea45f3ab08147b8faf66dedd21e2bc7c3404915 Mon Sep 17 00:00:00 2001 From: Cody Maloney Date: Mon, 19 May 2025 18:03:26 -0400 Subject: [PATCH 14/15] Include characters_written in BlockingReader --- Doc/library/io.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index ff4db2569fc84d..64199a2fa289f2 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -823,8 +823,8 @@ than raw I/O does. Write the :term:`bytes-like object`, *b*, and return the number of bytes written. When in non-blocking mode, a - :exc:`BlockingIOError` is raised if the buffer needs to be written out but - the raw stream blocks. + :exc:`BlockingIOError` with :attr:`BlockingIOError.characters_written` set + is raised if the buffer needs to be written out but the raw stream blocks. .. class:: BufferedRandom(raw, buffer_size=DEFAULT_BUFFER_SIZE) From 45bb9124238a5df4b82c6430641f97a1adeb3fe3 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Wed, 21 May 2025 12:00:57 -0400 Subject: [PATCH 15/15] Fewer rather than Less --- Doc/library/io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 64199a2fa289f2..0e305d9be058b1 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -570,7 +570,7 @@ I/O Base Classes Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative read as much as possible. - Less bytes may be returned than requested. An empty :class:`bytes` object + Fewer bytes may be returned than requested. An empty :class:`bytes` object is returned if the stream is already at EOF. More than one read may be made and calls may be retried if specific errors are encountered, see :meth:`os.read` and :pep:`475` for more details. Less than size bytes