Build failure for `python@3.12.1` on mac with non-system version of ncurses #118873

s-sajid-ali · 2024-05-09T23:45:51Z

Bug report

Bug description:

Using the spack package manager to install python@3.12.1 on darwin-sonoma-m1 with the following command:

sasyed@MAC-140753 ~> spack install --keep-stage python@3.12.1+optimizations%clang@17.0.6 ^libffi@3.4.6

results in the following build error:

Build error

...
==> Installing python-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il [21/21]
==> No binary for python-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il found: installing from source
==> Fetching https://www.python.org/ftp/python/3.12.1/Python-3.12.1.tgz
==> Ran patch() for python
==> python: Executing phase: 'configure'
==> python: Executing phase: 'build'
==> python: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j1' 'install'
...
     1763    ld: warning: reexported library with install name '@rpath/libunwind.1.dylib' found at '/Users/sasyed/Documents/packages/spack/opt/spa
             ck/darwin-sonoma-m1/apple-clang-15.0.0/llvm-17.0.6-uzrfikeehxbzxp3x3twl6wjnh53hpvmr/lib/libunwind.1.0.dylib' couldn't be matched with
              any parent library and will be linked directly
     1764    DYLD_LIBRARY_PATH=/private/var/folders/mb/zd6pgmt100q77s5fhn5xf4jw0000gq/T/sasyed/spack-stage/spack-stage-python-3.12.1-frh24vuhc6b5e
             yy346wng65mlwm3h3il/spack-src ./python.exe -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(
             ), *sys.version_info[:2]))' >platform
  >> 1765    [ERROR] _curses failed to import: dlopen(/private/var/folders/mb/zd6pgmt100q77s5fhn5xf4jw0000gq/T/sasyed/spack-stage/spack-stage-pyth
             on-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il/spack-src/build/lib.macosx-14.0-arm64-3.12/_curses.cpython-312-darwin.so, 0x0002): symbol 
             not found in flat namespace '_mvwadd_wch'
  >> 1766    [ERROR] _curses_panel failed to import: PyCapsule_Import could not import module "_curses"
  ...

The associated log files with this build error are:

spack-build-01-configure-out.txt
spack-build-02-build-out.txt
spack-build-03-install-out.txt

No build error when building python@3.11.7. The associated log files are:
spack-build-01-configure-out.txt
spack-build-02-build-out.txt
spack-build-03-install-out.txt

Note that the following flags are added by spack when optimizations are turned on:

        if "+optimizations" in spec:
            config_args.append("--enable-optimizations")
            # Prefer thin LTO for faster compilation times.
            if "@3.11.0: %clang@3.9:" in spec or "@3.11.0: %apple-clang@8:" in spec:
                config_args.append("--with-lto=thin")
            else:
                config_args.append("--with-lto")
            config_args.append("--with-computed-gotos")

https://github.com/spack/spack/blob/b79761b7eb5eb2e2d6b8174c3043524462772bf6/var/spack/repos/builtin/packages/python/package.py#L574-L581

CPython versions tested on:

3.12

Operating systems tested on:

macOS

The text was updated successfully, but these errors were encountered:

ned-deily · 2024-05-10T04:06:25Z

You should take this issue up first with the spack project. I'm not familiar with spack but a quick glance at the logs you provided shows a rather complicated and non-standard build process. A web search shows that at least one other person has seen this particular build failure using spack. The failure seems to be that that the built _curses module fails to import:

[ERROR] _curses failed to import: dlopen(/private/var/folders/mb/zd6pgmt100q77s5fhn5xf4jw0000gq/T/sasyed/spack-stage/spack-stage-python-3.12.1-frh24vuhc6b5eyy346wng65mlwm3h3il/spack-src/build/lib.macosx-14.0-arm64-3.12/_curses.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_mvwadd_wch'

It could be due to a mismatch between ncurses headers and libs when building.

Also, be aware that Python 3.12.3 is current so 3.12.1 is no longer supported.

s-sajid-ali · 2024-05-11T15:47:55Z

Thanks for the response @ned-deily!

The underlying issue seems to be from this bit of configure.ac:

cpython/configure.ac

Lines 6572 to 6588 in a019347

    
           AS_VAR_IF([ac_cv_header_ncurses_h], [yes], [ 
        
             if test "$ac_sys_system" != "Darwin"; then 
        
               dnl On macOS, there is no separate /usr/lib/libncursesw nor libpanelw. 
        
               PKG_CHECK_MODULES([CURSES], [ncursesw], [ 
        
                 AC_DEFINE([HAVE_NCURSESW], [1]) 
        
                 have_curses=ncursesw 
        
               ], [ 
        
                 WITH_SAVE_ENV([ 
        
                   AC_CHECK_LIB([ncursesw], [initscr], [ 
        
                     AC_DEFINE([HAVE_NCURSESW], [1]) 
        
                     have_curses=ncursesw 
        
                     CURSES_CFLAGS=${CURSES_CFLAGS-""} 
        
                     CURSES_LIBS=${CURSES_LIBS-"-lncursesw"} 
        
                   ]) 
        
                 ]) 
        
               ]) 
        
             fi

which makes the following assumption:

 dnl On macOS, there is no separate /usr/lib/libncursesw nor libpanelw.

However, for the build environment I'm using, with a custom ncurses the above assumption breaks:

sasyed@MAC-140753 ~> diff (pkg-config --libs ncurses | psub) (pkg-config --libs ncursesw | psub)
1c1
< -L/PATH_TO_NCURSES_6.5/lib -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
 -Wl,-search_paths_first -lncurses -ltinfo
---
> -L/PATH_TO_NCURSES_6.5/lib -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
 -Wl,-search_paths_first -lncursesw -ltinfow
sasyed@MAC-140753 ~ [1]>

Would it be possible to add the -lncursesw via CURSES_LIBS in spack or does the configure.ac need to be patched with something like the following?

--- a/configure.ac	2024-05-10 23:12:33
+++ b/configure.ac	2024-05-10 23:25:10
@@ -6418,8 +6418,8 @@
 AC_CHECK_HEADERS([curses.h ncurses.h])
 
 AS_VAR_IF([ac_cv_header_ncurses_h], [yes], [
-  if test "$ac_sys_system" != "Darwin"; then
-    dnl On macOS, there is no separate /usr/lib/libncursesw nor libpanelw.
+  dnl man diff: Exit status is 0 if inputs are the same, 1 if different!
+  AS_IF([test "diff <($PKG_CONFIG --libs ncurses) <($PKG_CONFIG --libs ncursesw)" != 0 ], [
     PKG_CHECK_MODULES([CURSES], [ncursesw], [
       AC_DEFINE([HAVE_NCURSESW], [1])
       have_curses=ncursesw
@@ -6433,7 +6433,7 @@
         ])
       ])
     ])
-  fi
+  ])
 
   AS_VAR_IF([have_curses], [no], [
     PKG_CHECK_MODULES([CURSES], [ncurses], [

ned-deily · 2024-05-13T21:55:56Z

@erlend-aasland, what do you think? This might also be related to #113565.

erlend-aasland · 2024-06-02T12:38:50Z

Could you check if #119816 works for you?

s-sajid-ali · 2024-06-03T18:20:10Z

@erlend-aasland : The current spack recipe for Python does not support building from git branches, though it is possible to do so. I'm not sure if it is worth the effort to add that to test this PR.

On the other hand, if I use system compilers with the newer ncurses in the path, configure seems to pick the system built ncurses as in #113565.

For the build that failed, I'd used spack to build Python and harnessed spack built llvm. Note that spack comes with compiler wrappers that were used during the build that failed. Using the spack built compilers directly leads to unrelated errors:

checking for long double... yes
checking size of long double... configure: error: in '/Users/sasyed/Documents/packages/temp/bugfix/cpython-head':
configure: error: cannot compute sizeof (long double)
See 'config.log' for more details

erlend-aasland · 2024-06-03T21:33:04Z

@s-sajid-ali, got it. Looks like it should be fairly easy to hack that script to download a 3.13 beta release, though. If we decide to backport #119816 to 3.13, you'd should be able to test it relatively soon.

s-sajid-ali · 2024-06-04T15:51:49Z

@erlend-aasland : Yes, building the 3.13-beta2 release is straightforward. I'll be on the lookout for the beta2 release and let you know if the patch fixes the issue.

erlend-aasland · 2024-06-04T19:22:38Z

I think you'll have to wait for beta3; it's a large PR, and I don't want to rush it. Feel free to take a stab at reviewing it, if you feel like it :)

s-sajid-ali · 2024-06-12T19:39:20Z

Thanks for the update and asking me to review it, but I don't really know enough about autotools/autoconf to give useful feedback. It does look good on a high level and I'd be happy to test it out when the beta3 release comes out.

Thanks again for fixing this!

erlend-aasland · 2024-07-01T12:19:28Z

@s-sajid-ali, you'll find the 3.13 backport of #119816, #121202, in the upcoming 3.13-beta4 release, scheduled to be released in a couple of weeks time. There has been no decision regarding a 3.12 backport yet.

s-sajid-ali · 2024-07-03T20:49:59Z

Thanks for the update @erlend-aasland! I'll close this issue since the fix has been merged.

s-sajid-ali added the type-bug An unexpected behavior, bug, or error label May 9, 2024

ned-deily closed this as not planned Won't fix, can't repro, duplicate, stale May 10, 2024

ned-deily reopened this May 13, 2024

s-sajid-ali changed the title ~~Build failure for python@3.12.1 on M1 mac with optimizations enabled.~~ Build failure for python@3.12.1 on mac with non-system version of ncurses May 16, 2024

s-sajid-ali mentioned this issue May 16, 2024

python: fix ncurses linking on macOS, libffi dependency settings spack/spack#44226

Open

erlend-aasland mentioned this issue Jul 1, 2024

[3.12] gh-113565: Improve and harden detection of curses dependencies (#119816) #121222

Closed

s-sajid-ali closed this as completed Jul 3, 2024

adamjstewart mentioned this issue Aug 13, 2024

python: add v3.11.9, v3.12.5, default to latest version spack/spack#45712

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build failure for `python@3.12.1` on mac with non-system version of ncurses #118873

Build failure for `python@3.12.1` on mac with non-system version of ncurses #118873

s-sajid-ali commented May 9, 2024 •

edited

Loading

ned-deily commented May 10, 2024 •

edited

Loading

s-sajid-ali commented May 11, 2024

ned-deily commented May 13, 2024

erlend-aasland commented Jun 2, 2024

s-sajid-ali commented Jun 3, 2024

erlend-aasland commented Jun 3, 2024 •

edited

Loading

s-sajid-ali commented Jun 4, 2024 •

edited

Loading

erlend-aasland commented Jun 4, 2024

s-sajid-ali commented Jun 12, 2024

erlend-aasland commented Jul 1, 2024

s-sajid-ali commented Jul 3, 2024

Build failure for python@3.12.1 on mac with non-system version of ncurses #118873

Build failure for python@3.12.1 on mac with non-system version of ncurses #118873

Comments

s-sajid-ali commented May 9, 2024 • edited Loading

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

ned-deily commented May 10, 2024 • edited Loading

s-sajid-ali commented May 11, 2024

ned-deily commented May 13, 2024

erlend-aasland commented Jun 2, 2024

s-sajid-ali commented Jun 3, 2024

erlend-aasland commented Jun 3, 2024 • edited Loading

s-sajid-ali commented Jun 4, 2024 • edited Loading

erlend-aasland commented Jun 4, 2024

s-sajid-ali commented Jun 12, 2024

erlend-aasland commented Jul 1, 2024

s-sajid-ali commented Jul 3, 2024

Build failure for `python@3.12.1` on mac with non-system version of ncurses #118873

Build failure for `python@3.12.1` on mac with non-system version of ncurses #118873

s-sajid-ali commented May 9, 2024 •

edited

Loading

ned-deily commented May 10, 2024 •

edited

Loading

erlend-aasland commented Jun 3, 2024 •

edited

Loading

s-sajid-ali commented Jun 4, 2024 •

edited

Loading