bpo-41129: Fix check for macOS SDK paths when building Python #25785

nedbat · 2021-05-01T16:09:55Z

Catalina puts files in /System/Volumes/Data, so the CPython sources can
be there. setup.py shouldn't assume /System means SDK files.

https://bugs.python.org/issue41129

ned-deily

Thanks for the PR. AFAIK, this is a pretty rare problem probably depending on some unusual file system configurations; I have never seen it on Catalina or Big Sur. But it should be fixed.

bedevere-bot · 2021-05-01T16:58:21Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

ned-deily · 2021-05-01T16:58:46Z

setup.py

@@ -230,7 +230,7 @@ def is_macosx_sdk_path(path):
    Returns True if 'path' can be located in an OSX SDK
    """
    return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
-                or path.startswith('/System/')
+                or (path.startswith('/System/') and not path.startswith('/System/Volumes/Data'))


As I suggested in an earlier comment on the bug tracker for this, all we are interested in here are files in the SDK. For macOS 10.15 and 11 under /System, there are /System/Library and, for completeness, /System/iOSSupport. The test should be explicitly for those.

Catalina puts files in /System/Volumes/Data, so the CPython sources can be there. setup.py shouldn't assume /System means SDK files. The fix is Ned Deily's suggestion from the bpo.

nedbat · 2021-05-01T18:06:04Z

I have made the requested changes; please review again.

bedevere-bot · 2021-05-01T18:06:07Z

Thanks for making the requested changes!

@ned-deily: please review the changes made to this pull request.

miss-islington · 2021-05-03T03:02:48Z

Thanks @nedbat for the PR, and @ned-deily for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

…GH-25785) Narrow search to match contents of SDKs, namely only files in ``/System/Library``, ``/System/IOSSupport``, and ``/usr`` other than ``/usr/local``. Previously, anything under ``/System`` was assumed to be in an SDK which causes problems with the new file system layout in 10.15+ where user file systems may appear to be mounted under ``/System``. Paths in ``/Library`` were also incorrectly treated as SDK locations. Co-authored-by: Ned Deily <nad@python.org> (cherry picked from commit d52bbde) Co-authored-by: Ned Batchelder <ned@nedbatchelder.com>

bedevere-bot · 2021-05-03T03:03:02Z

GH-25830 is a backport of this pull request to the 3.9 branch.

…) (GH-25830) Narrow search to match contents of SDKs, namely only files in ``/System/Library``, ``/System/IOSSupport``, and ``/usr`` other than ``/usr/local``. Previously, anything under ``/System`` was assumed to be in an SDK which causes problems with the new file system layout in 10.15+ where user file systems may appear to be mounted under ``/System``. Paths in ``/Library`` were also incorrectly treated as SDK locations. Co-authored-by: Ned Deily <nad@python.org> (cherry picked from commit d52bbde) Co-authored-by: Ned Batchelder <ned@nedbatchelder.com>

bedevere-bot · 2021-05-03T04:10:45Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.9 has failed when building commit 98035ec.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/108/builds/31) and take a look at the build logs.
Check if the failure is related to this commit (98035ec) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/108/builds/31

Failed tests:

test_importlib

Failed subtests:

test_multiprocessing_pool_circular_import - test.test_importlib.test_threaded_import.ThreadedImportTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

406 tests OK.

10 slowest tests:

test_concurrent_futures: 4 min 21 sec
test_tokenize: 3 min 20 sec
test_unparse: 3 min 17 sec
test_multiprocessing_spawn: 2 min 49 sec
test_lib2to3: 2 min 19 sec
test_multiprocessing_forkserver: 2 min 6 sec
test_unicodedata: 1 min 59 sec
test_asyncio: 1 min 47 sec
test_capi: 1 min 42 sec
test_io: 1 min 6 sec

1 test failed:
test_importlib

18 tests skipped:
test_devpoll test_epoll test_gdb test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_smtpnet test_spwd
test_ssl test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_importlib

Total duration: 27 min 7 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 54, in _launch
    child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 106 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-96sb31va': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-827db9oi': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-30bew1da': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-8wcosp6t': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-lm0nm2jv': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-c1bsiprl': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-cqxpgr6g': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-8i_rynzw': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-iaaoj6uz': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-x00y3jgb': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-_0p9it19': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/pool.py", line 196, in __init__
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/queues.py", line 341, in __init__
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/connection.py", line 532, in Pipe
    fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 113 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-_0hs0ec6': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-wyfmoikl': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-fnnu8a3g': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-8swcsu_w': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/mp-gv4d4fro': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 257, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/support/script_helper.py", line 156, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/support/script_helper.py", line 142, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/support/script_helper.py", line 70, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.9.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

bedevere-bot added the awaiting review label May 1, 2021

the-knights-who-say-ni added the CLA signed label May 1, 2021

ned-deily requested changes May 1, 2021

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels May 1, 2021

ned-deily requested changes May 1, 2021

View reviewed changes

nedbat force-pushed the nedbat/bpo41129 branch from f920892 to e8ceeb8 Compare May 1, 2021 17:10

bpo-41129: fix Mac Catalina filesystem confusion during build

3e65c28

Catalina puts files in /System/Volumes/Data, so the CPython sources can be there. setup.py shouldn't assume /System means SDK files. The fix is Ned Deily's suggestion from the bpo.

nedbat force-pushed the nedbat/bpo41129 branch from e8ceeb8 to 3e65c28 Compare May 1, 2021 17:32

bedevere-bot added awaiting change review and removed awaiting changes labels May 1, 2021

bedevere-bot requested a review from ned-deily May 1, 2021 18:06

ned-deily self-assigned this May 1, 2021

Refine the test somewhat and add NEWS entry

e16e9c6

ned-deily merged commit d52bbde into python:master May 3, 2021

bedevere-bot removed the awaiting change review label May 3, 2021

ned-deily changed the title ~~bpo-41129: fix Mac Catalina filesystem confusion during build~~ bpo-41129: Fix check for macOS SDK paths when building Python May 3, 2021

ned-deily added the needs backport to 3.9 only security fixes label May 3, 2021

bedevere-bot removed the needs backport to 3.9 only security fixes label May 3, 2021

nedbat deleted the nedbat/bpo41129 branch May 4, 2021 00:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-41129: Fix check for macOS SDK paths when building Python #25785

bpo-41129: Fix check for macOS SDK paths when building Python #25785

nedbat commented May 1, 2021 •

edited

Loading

ned-deily left a comment

bedevere-bot commented May 1, 2021

ned-deily May 1, 2021 •

edited

Loading

nedbat May 1, 2021

nedbat commented May 1, 2021

bedevere-bot commented May 1, 2021

miss-islington commented May 3, 2021

bedevere-bot commented May 3, 2021

bedevere-bot commented May 3, 2021

bpo-41129: Fix check for macOS SDK paths when building Python #25785

bpo-41129: Fix check for macOS SDK paths when building Python #25785

Conversation

nedbat commented May 1, 2021 • edited Loading

ned-deily left a comment

Choose a reason for hiding this comment

bedevere-bot commented May 1, 2021

ned-deily May 1, 2021 • edited Loading

Choose a reason for hiding this comment

nedbat May 1, 2021

Choose a reason for hiding this comment

nedbat commented May 1, 2021

bedevere-bot commented May 1, 2021

miss-islington commented May 3, 2021

bedevere-bot commented May 3, 2021

bedevere-bot commented May 3, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

nedbat commented May 1, 2021 •

edited

Loading

ned-deily May 1, 2021 •

edited

Loading