Skip to content

asyncio ProactorEventLoop: wait_closed() can raise ConnectionResetError #83037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
1st1 opened this issue Nov 19, 2019 · 15 comments
Open

asyncio ProactorEventLoop: wait_closed() can raise ConnectionResetError #83037

1st1 opened this issue Nov 19, 2019 · 15 comments
Labels
3.8 (EOL) end of life 3.9 only security fixes OS-windows topic-asyncio

Comments

@1st1
Copy link
Member

1st1 commented Nov 19, 2019

BPO 38856
Nosy @pfmoore, @tjguk, @asvetlov, @ambv, @zware, @1st1, @zooba, @cmeyer, @basnijholt, @aeros
PRs
  • gh-83037: Fix StreamWriter.wait_closed() ConnectionResetError for ProactorEventLoop #18199
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/asvetlov'
    closed_at = None
    created_at = <Date 2019-11-19.22:06:26.615>
    labels = ['3.8', '3.9', 'OS-windows', 'expert-asyncio']
    title = 'asyncio ProactorEventLoop: wait_closed() can raise ConnectionResetError'
    updated_at = <Date 2021-08-12.21:30:37.215>
    user = 'https://github.com/1st1'

    bugs.python.org fields:

    activity = <Date 2021-08-12.21:30:37.215>
    actor = 'seer'
    assignee = 'asvetlov'
    closed = False
    closed_date = None
    closer = None
    components = ['Windows', 'asyncio']
    creation = <Date 2019-11-19.22:06:26.615>
    creator = 'yselivanov'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 38856
    keywords = ['patch']
    message_count = 10.0
    messages = ['356996', '358097', '358109', '360410', '360742', '362633', '364242', '364250', '373755', '399483']
    nosy_count = 11.0
    nosy_names = ['paul.moore', 'tim.golden', 'asvetlov', 'lukasz.langa', 'zach.ware', 'yselivanov', 'steve.dower', 'cmeyer', 'basnijholt', 'aeros', 'seer']
    pr_nums = ['18199']
    priority = 'high'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue38856'
    versions = ['Python 3.8', 'Python 3.9']

    @1st1
    Copy link
    Member Author

    1st1 commented Nov 19, 2019

    The exception should probably be just ignored. Andrew, thoughts?

    Here's an example error traceback:

          Traceback (most recent call last):
            File "c:\projects\asyncpg\asyncpg\connection.py", line 1227, in _cancel
              await w.wait_closed()
            File "C:\Python38\lib\asyncio\streams.py", line 376, in wait_closed
              await self._protocol._get_close_waiter(self)
            File "c:\projects\asyncpg\asyncpg\connection.py", line 1202, in _cancel
              await r.read()  # Wait until EOF
            File "C:\Python38\lib\asyncio\streams.py", line 694, in read
              block = await self.read(self._limit)
            File "C:\Python38\lib\asyncio\streams.py", line 701, in read
              await self._wait_for_data('read')
            File "C:\Python38\lib\asyncio\streams.py", line 534, in _wait_for_data
              await self._waiter
            File "C:\Python38\lib\asyncio\proactor_events.py", line 280, in _loop_reading
              data = fut.result()
            File "C:\Python38\lib\asyncio\windows_events.py", line 808, in _poll
              value = callback(transferred, key, ov)
            File "C:\Python38\lib\asyncio\windows_events.py", line 457, in finish_recv
              raise ConnectionResetError(*exc.args)
          ConnectionResetError: [WinError 64] The specified network name is no longer available

    @ambv
    Copy link
    Contributor

    ambv commented Dec 9, 2019

    Note: this is going to miss Python 3.8.1 as I'm releasing 3.8.1rc1 right now. Please address this before 3.8.2 (due in February).

    @asvetlov
    Copy link
    Contributor

    asvetlov commented Dec 9, 2019

    Sorry, I've missed this issue.
    I'll address it in a while.

    @asvetlov asvetlov self-assigned this Dec 9, 2019
    @vstinner
    Copy link
    Member

    Why is this issue marked as a release blocker? Is it a Python 3.8 regression, or just a regular bug?

    @vstinner vstinner changed the title wait_closed() can raise ConnectionResetError asyncio ProactorEventLoop: wait_closed() can raise ConnectionResetError Jan 21, 2020
    @aeros
    Copy link
    Contributor

    aeros commented Jan 27, 2020

    The exception should probably be just ignored.

    It looks like the "ERROR_NETNAME_DELETED" (win32 error 64) exception is already ignored in multiprocessing's connection.wait():

    if err not in _ready_errors:
    . Would it be appropriate to ignore it here as well?

    I think that we should still retain the ConnectionResetError for "ERROR_OPERATION_ABORTED", but just specifically ignore "ERROR_NETNAME_DELETED". Ignoring "ERROR_OPERATION_ABORTED" doesn't seem correct here since it's relied on for cancelling file I/O (see https://docs.microsoft.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations).

    The patch should be straightforward. I'll test it locally and then open a PR, unless Andrew is currently working on it.

    @ambv
    Copy link
    Contributor

    ambv commented Feb 25, 2020

    Downgrading priority on this, agreed with Victor.

    @idomic
    Copy link
    Mannequin

    idomic mannequin commented Mar 15, 2020

    Is this open for a PR?

    @aeros
    Copy link
    Contributor

    aeros commented Mar 15, 2020

    Is this open for a PR?

    I opened a PR for this issue a bit ago (applies Yury's suggestion of suppressing the exception), just waiting on review from Andrew when he finds the time to do so.

    @basnijholt
    Copy link
    Mannequin

    basnijholt mannequin commented Jul 16, 2020

    I have noticed the following on Linux too:

    Traceback (most recent call last):
      File "/config/custom_components/kef_custom/aiokef.py", line 327, in _disconnect
        await self._writer.wait_closed()
      File "/usr/local/lib/python3.7/asyncio/streams.py", line 323, in wait_closed
        await self._protocol._closed
      File "/config/custom_components/kef_custom/aiokef.py", line 299, in _send_message
        await self._writer.drain()
      File "/usr/local/lib/python3.7/asyncio/streams.py", line 339, in drain
        raise exc
      File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 814, in _read_ready__data_received
        data = self._sock.recv(self.max_size)
    ConnectionResetError: [Errno 104] Connection reset by peer
    

    after which every invocation of await asyncio.open_connection fails with ConnectionRefusedError until I restart the entire Python process.

    IIUC, just ignoring the exception will not be enough.

    This issue is about the ProactorEventLoop for Windows specifically, however, my process uses the default event loop on Linux.

    @seer
    Copy link
    Mannequin

    seer mannequin commented Aug 12, 2021

    Python 3.9.6 Linux same issue.

    @seer seer mannequin added the 3.9 only security fixes label Aug 12, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @ezio-melotti ezio-melotti moved this to Todo in asyncio Jul 17, 2022
    @iUnknwn
    Copy link
    Contributor

    iUnknwn commented Feb 27, 2023

    It looks like there's additional information here: #93821 - including a proposed patch.

    While this is under investigation, are there any known workarounds for this issue at present?

    @gvanrossum
    Copy link
    Member

    A PR with the proposed fix is gh-18199 (found in gh-93821 -- thanks for the link, @iUnknwn). This changes the special handling of ERROR_NETNAME_DELETED in recv() and friends (though not in send() and friends). Alas, that PR is out of date (the recv() code was refactored somewhat) and someone would have to volunteer an updated version.

    From internal sources (via @zooba) I understand that the general approach to this error is to retry (assuming it is intermittent or eventually a different error will occur) but there is currently no reproducer, which makes it hard to do create a test. I'm not sure if the proposed patch actually does that -- it seems as if it would cause recv() to return None, which I suspect most callers don't expect. So maybe the retry would have to be coded differently.

    Anyway, @iUnknwn, from the logs you sent me privately it seems your problem is actually in accept(), which we should discuss in gh-93821.

    @gvanrossum
    Copy link
    Member

    Comments by @eryksun on gh-93821 suggest that this error code indicates a disconnect, so that suggests that recv() and friends are right in raising ConnectionResetError, although perhaps just returning b'' would also be acceptable -- that's what is returned for BrokenPipeError.

    So I can think of two fixes:

    • either we return b'' from recv() and friends (along the lines of the original discussion, @1st1 and @aeros);
    • or we keep recv() etc. unchanged but catch, log and ignore errors in wait_closed().

    @gvanrossum
    Copy link
    Member

    Still not sure what to do, but I've rebased the PR so at least it can be tested.

    @gvanrossum
    Copy link
    Member

    @eryksun Do you have an opinion on the proposed PR (gh-93821)? Should we merge it and consider this fixed, or should we close the PR and take another approach to fix the issue?

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 (EOL) end of life 3.9 only security fixes OS-windows topic-asyncio
    Projects
    Status: Todo
    Development

    No branches or pull requests

    7 participants