Skip to content

Add a --rerun option to the iOS testbed. #122992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 14, 2024

Conversation

freakboy3742
Copy link
Contributor

Adds a --rerun option to the iOS testbed runner.

We've seen a small number of transient failures in the iOS buildbot - mostly in socket-related tests, due to socket availability. These tests pass on the next test, and can't be easily reproduced. The test runner has a --rerun option that may help with the resilience of these tests, and is enabled in the --slow/fast-ci configurations; this PR enables that option for the iOS test runner.

@freakboy3742
Copy link
Contributor Author

!buildbot iOS

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit f9cca3e 🤖

The command will test the builders whose names match following regular expression: iOS

The builders matched are:

  • iOS ARM64 Simulator PR

Copy link
Member

@ned-deily ned-deily left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good option to use

@freakboy3742 freakboy3742 merged commit ce39d3c into python:main Aug 14, 2024
40 checks passed
@miss-islington-app
Copy link

Thanks @freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 14, 2024
Enable --rerun when running tests on iOS.
(cherry picked from commit ce39d3c)

Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
@bedevere-app
Copy link

bedevere-app bot commented Aug 14, 2024

GH-122993 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Aug 14, 2024
@freakboy3742 freakboy3742 deleted the ios-rerun branch August 14, 2024 00:23
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.x has failed when building commit ce39d3c.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#builders/1380/builds/1068) and take a look at the build logs.
  4. Check if the failure is related to this commit (ce39d3c) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#builders/1380/builds/1068

Failed tests:

  • test_urllib2net

Failed subtests:

  • test_ftp_timeout - test.test_urllib2net.TimeoutTest.test_ftp_timeout
  • test_ftp - test.test_urllib2net.OtherNetworkTests.test_ftp
  • test_ftp_no_timeout - test.test_urllib2net.TimeoutTest.test_ftp_no_timeout
  • test_ftp_basic - test.test_urllib2net.TimeoutTest.test_ftp_basic
  • test_ftp_default_timeout - test.test_urllib2net.TimeoutTest.test_ftp_default_timeout

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 373, in test_ftp_timeout
    u = _urlopen_with_retry(self.FTP_HOST, timeout=60)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
Warning --   File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/libregrtest/run_workers.py", line 399, in run
Warning --     mp_result = self._runtest(test_name)
Warning --   File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/libregrtest/run_workers.py", line 359, in _runtest
Warning --     retcode, tmp_files = self.run_tmp_files(worker_runtests,
Warning --                          ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
Warning --                                             stdout_file.fileno())
Warning --                                             ^^^^^^^^^^^^^^^^^^^^^
Warning --   File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/libregrtest/run_workers.py", line 295, in run_tmp_files
Warning --     retcode = self._run_process(worker_runtests,
Warning --                                 stdout_fd, tmp_dir)
Warning --   File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/libregrtest/run_workers.py", line 177, in _run_process
Warning --     popen = create_worker_process(runtests, output_fd, tmp_dir)
Warning --   File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/libregrtest/worker.py", line 64, in create_worker_process
Warning --     return subprocess.Popen(cmd, **kwargs)
Warning --            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
Warning --   File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/subprocess.py", line 824, in __init__
Warning --     raise OSError(
Warning --         errno.ENOTSUP, f"{sys.platform} does not support processes."
Warning --     )
Warning -- OSError: [Errno 45] ios does not support processes.


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 1548, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
                  ~~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 2443, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
                    ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 353, in ntransfercmd
    host, port = self.makepasv()
                 ~~~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 327, in makepasv
    untrusted_host, port = parse227(self.sendcmd('PASV'))
                                    ~~~~~~~~~~~~^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 281, in sendcmd
    return self.getresp()
           ~~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 500 OOPS: vsf_sysutil_bind


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 1548, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
                  ~~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 2426, in retrfile
    raise URLError(f'ftp error: {reason}') from reason
urllib.error.URLError: <urlopen error ftp error: 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 364, in test_ftp_no_timeout
    u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 372, in test_ftp_timeout
    with socket_helper.transient_internet(self.FTP_HOST):
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not a container or iterable


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 261, in _test_urls
    f = urlopen(url, req, support.INTERNET_TIMEOUT)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error <urlopen error ftp error: 500 OOPS: vsf_sysutil_bind>>


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 343, in test_ftp_basic
    u = _urlopen_with_retry(self.FTP_HOST)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 352, in test_ftp_default_timeout
    u = _urlopen_with_retry(self.FTP_HOST)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/urllib/request.py", line 2423, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
                    ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 353, in ntransfercmd
    host, port = self.makepasv()
                 ~~~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 327, in makepasv
    untrusted_host, port = parse227(self.sendcmd('PASV'))
                                    ~~~~~~~~~~~~^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 281, in sendcmd
    return self.getresp()
           ~~~~~~~~~~~~^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 500 OOPS: vsf_sysutil_bind


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 349, in test_ftp_default_timeout
    with socket_helper.transient_internet(self.FTP_HOST):
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not a container or iterable


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 342, in test_ftp_basic
    with socket_helper.transient_internet(self.FTP_HOST, timeout=None):
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not a container or iterable


Traceback (most recent call last):
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/test_urllib2net.py", line 361, in test_ftp_no_timeout
    with socket_helper.transient_internet(self.FTP_HOST):
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/Users/buildbot/Library/Developer/XCTestDevices/DEF0AAE3-414C-4A0D-A16C-D561C37F8D16/data/Containers/Bundle/Application/028C1970-6491-46B7-A9E9-5B8076EA3A32/iOSTestbed.app/python/lib/python3.14/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not a container or iterable

freakboy3742 added a commit that referenced this pull request Aug 14, 2024
Enable --rerun when running tests on iOS.
(cherry picked from commit ce39d3c)

Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
@ned-deily
Copy link
Member

Oops, it looks like regrtest by default uses a subprocess for the reruns. https://buildbot.python.org/#builders/1380/builds/1068

@freakboy3742
Copy link
Contributor Author

Whelp... I guess we need --single-process as well. Unfortunately, that option doesn't exist in 3.13, so we've either got to backport #119728, or live without --rerun on the 3.13 branch. Any preferences @ned-deily?

@ned-deily
Copy link
Member

It would be a RM call; to me, backporting a feature in the release candidate phase of 3.13 seems a stretch. Possibly in a maintenance release. But I think the right thing to do now is to live without it for 3.13.0.

It does look like there is some useful info in the buildbot traceback: 500 OOPS: vsf_sysutil_bind.

@freakboy3742
Copy link
Contributor Author

Makes sense. I've opened #122994 to add the --single-process option on main; I guess the soon-to-be-answered question is whether the problem is just a very inconvenient, new form of transient error with FTP tests, or an iOS problem to be fixed...

@freakboy3742
Copy link
Contributor Author

Looks like the vsf_sysutil_bind error is transient... but it can also trigger on macOS

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL7 LTO + PGO 3.13 has failed when building commit bfec6f3.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#builders/1461/builds/212) and take a look at the build logs.
  4. Check if the failure is related to this commit (bfec6f3) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#builders/1461/builds/212

Failed tests:

  • test_urllib2net

Failed subtests:

  • test_ftp_timeout - test.test_urllib2net.TimeoutTest.test_ftp_timeout
  • test_ftp - test.test_urllib2net.OtherNetworkTests.test_ftp
  • test_ftp_no_timeout - test.test_urllib2net.TimeoutTest.test_ftp_no_timeout
  • test_ftp_basic - test.test_urllib2net.TimeoutTest.test_ftp_basic
  • test_ftp_default_timeout - test.test_urllib2net.TimeoutTest.test_ftp_default_timeout

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 372, in test_ftp_timeout
    with socket_helper.transient_internet(self.FTP_HOST):
    ...<2 lines>...
        self.assertEqual(u.fp.fp.raw._sock.gettimeout(), 60)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not iterable


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 349, in test_ftp_default_timeout
    with socket_helper.transient_internet(self.FTP_HOST):
    ...<6 lines>...
        self.assertEqual(u.fp.fp.raw._sock.gettimeout(), 60)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not iterable


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 1548, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
                  ~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 2426, in retrfile
    raise URLError(f'ftp error: {reason}') from reason
urllib.error.URLError: <urlopen error ftp error: 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 364, in test_ftp_no_timeout
    u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 361, in test_ftp_no_timeout
    with socket_helper.transient_internet(self.FTP_HOST):
    ...<6 lines>...
        self.assertIsNone(u.fp.fp.raw._sock.gettimeout())
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not iterable


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 1548, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
                  ~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 2443, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
                    ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 353, in ntransfercmd
    host, port = self.makepasv()
                 ~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 327, in makepasv
    untrusted_host, port = parse227(self.sendcmd('PASV'))
                                    ~~~~~~~~~~~~^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 281, in sendcmd
    return self.getresp()
           ~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 500 OOPS: vsf_sysutil_bind


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 261, in _test_urls
    f = urlopen(url, req, support.INTERNET_TIMEOUT)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error <urlopen error ftp error: 500 OOPS: vsf_sysutil_bind>>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 373, in test_ftp_timeout
    u = _urlopen_with_retry(self.FTP_HOST, timeout=60)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 352, in test_ftp_default_timeout
    u = _urlopen_with_retry(self.FTP_HOST)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 342, in test_ftp_basic
    with socket_helper.transient_internet(self.FTP_HOST, timeout=None):
    ...<2 lines>...
        self.assertIsNone(u.fp.fp.raw._sock.gettimeout())
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 264, in transient_internet
    filter_error(err)
    ~~~~~~~~~~~~^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 237, in filter_error
    (("ConnectionRefusedError" in err.reason) or
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'error_perm' is not iterable


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/support/socket_helper.py", line 249, in transient_internet
    yield
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 343, in test_ftp_basic
    u = _urlopen_with_retry(self.FTP_HOST)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 29, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 25, in _retry_thrice
    raise last_exc
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/test/test_urllib2net.py", line 21, in _retry_thrice
    return func(*args, **kwargs)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 489, in open
    response = self._open(req, data)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 506, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
                              '_open', req)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 466, in _call_chain
    result = func(*args)
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 1558, in ftp_open
    raise URLError(exp) from exp
urllib.error.URLError: <urlopen error 500 OOPS: vsf_sysutil_bind>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/urllib/request.py", line 2423, in retrfile
    conn, retrlen = self.ftp.ntransfercmd(cmd)
                    ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 353, in ntransfercmd
    host, port = self.makepasv()
                 ~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 327, in makepasv
    untrusted_host, port = parse227(self.sendcmd('PASV'))
                                    ~~~~~~~~~~~~^^^^^^^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 281, in sendcmd
    return self.getresp()
           ~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.13.cstratak-RHEL7-ppc64le.lto-pgo/build/Lib/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 500 OOPS: vsf_sysutil_bind

@ned-deily
Copy link
Member

These errors may be related to the recent OS upgrade on the PSF pythontest.net test server.

@mhsmith
Copy link
Member

mhsmith commented Aug 14, 2024

I had been planning to use --slow-ci in the Android buildbot for consistency with the other platforms, but that won't be possible unless the --single-process option is available as well. So I'll submit a backport of that option, even if it does have to wait until 3.13.1.

blhsing pushed a commit to blhsing/cpython that referenced this pull request Aug 22, 2024
Enable --rerun when running tests on iOS.
@freakboy3742 freakboy3742 added the needs backport to 3.13 bugs and security fixes label Aug 27, 2024
@miss-islington-app
Copy link

Thanks @freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 27, 2024
Enable --rerun when running tests on iOS.
(cherry picked from commit ce39d3c)

Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
@bedevere-app
Copy link

bedevere-app bot commented Aug 27, 2024

GH-123367 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Aug 27, 2024
freakboy3742 added a commit that referenced this pull request Aug 27, 2024
Add a --rerun option to the iOS testbed. (GH-122992)

Enable --rerun when running tests on iOS.
(cherry picked from commit ce39d3c)

Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants