BLD: updates to the Meson build #23740

rgommers · 2023-05-09T14:36:46Z

Mostly synced from SciPy, and also addresses some warnings that show up with Meson 1.1.0

mattip · 2023-05-09T21:54:41Z

The CI failure is in the meson build on macos_arm64.

rgommers · 2023-05-10T10:35:22Z

Hmm, I tested that locally, but in a conda env. Shouldn't be hard to figure out. The error was:

Found CMake: /opt/homebrew/bin/cmake (3.24.1)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)

numpy/meson.build:53:9: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig, framework and cmake

also something wrong with CMake in Homebrew it looks like.

rgommers · 2023-05-11T09:56:00Z

It's good that macos_arm64 failed, because it reminded me about dealing with lapack_lite. It turns out that in this job, OpenBLAS was installed but not found. From another passing CI log from today:

Found CMake: /opt/homebrew/bin/cmake (3.24.1)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)

I think we want to keep this unchanged, meaning that if no BLAS/LAPACK library was found, we use the fallback library and continue. We should at least emit a clear warning though, because for many users this continues to be a performance footgun.

mattip · 2023-05-11T14:28:06Z

OpenBLAS was installed but not found

Do you need help debugging this? The wheel builder job

seems to be working

The numpy-1.25.0.dev0+1377.g276cc995c-cp39-cp39-macosx_11_0_arm64.whl built 4 days ago and uploaded to https://anaconda.org/scipy-wheels-nightly/numpy/files does have a 22 MB file named libopenblas64_.0.dylib in the .dylibs file.

rgommers · 2023-05-11T14:33:55Z

Not really, this is pretty routine by now, the problem should be that the .pc file of the OpenBLAS used there is not on pkg-config's search path.

I am bandwidth-constrained though and may not touch this for several days, so if you want to fix it, I won't complain:)

charris · 2023-05-13T00:47:33Z

close/reopen

mattip · 2023-05-18T08:32:56Z

meson.build

@@ -34,10 +35,11 @@ elif cc.get_id() == 'msvc'
          'when building with MSVC')
  endif
 endif
+if not cy.version().version_compare('>=0.29.34')
+  error('SciPy requires Cython >= 0.29.34')


Is there a "project_name" macro we could use here instead?

Suggested change

error('SciPy requires Cython >= 0.29.34')

error('NumPy requires Cython >= 0.29.34')

Good catch, thanks. I'll see if the current CI run passes, then will push this with a few more tweaks.

Is there a "project_name" macro we could use here instead?

Not that I know of - but seems more complexity than it's worth anyway, and makes it harder to grep for the error string, so I'd prefer not to do that.

rgommers · 2023-05-18T08:41:40Z

It looks like we should start moving some more CI jobs over to Meson soon. Also, the automatic wheel build trigger is a massive waste of jobs.

This looks better now, I'll keep it like this and will add a comment that OpenBLAS is not used:

Library m found: YES
Found CMake: /opt/homebrew/bin/cmake (3.24.1)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)

EDIT and the warning I wanted to see:

numpy/linalg/meson.build:17: WARNING: LAPACK was not found, NumPy is using an unoptimized, naive build from sources!

Mostly synced from SciPy, and also addresses some warnings that show up with Meson 1.1.0 [skip circle] [skip azp]

rgommers · 2023-05-18T09:14:18Z

@andyfaff FYI I make some minor tweaks to cirrus_macosx_arm64.yml. NumPy does not actually require a BLAS/LAPACK, and use of CFLAGS in CI jobs should be avoided, since it tends to mask problems that then will show up for users. With distutils there was no better (easy) way, but Meson has a compiler.get_supported_arguments method so that any needed compile flags can be added in a portable manner.

rgommers · 2023-05-18T09:33:00Z

A fun new error with -Werror in Python 3.11.3 that I cannot reproduce locally:

FAILED: numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so.p/src_umath_override.c.o 
cc -Inumpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so.p -Inumpy/core -I../numpy/core -Inumpy/core/include -I../numpy/core/include -I../numpy/core/src/common -I../numpy/core/src/multiarray -I../numpy/core/src/npymath -I../numpy/core/src/umath -I/usr/include/x86_64-linux-gnu/openblas-serial/ -I/opt/hostedtoolcache/Python/3.11.3/x64/include/python3.11 -fvisibility=hidden -fdiagnostics-color=always -Wall -Winvalid-pch -Werror -std=c99 -O2 -g -fno-strict-aliasing -DHAVE_CBLAS -fPIC -DNPY_INTERNAL_BUILD -DHAVE_NPY_CONFIG_H -DNPY_DISABLE_OPTIMIZATION -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -MD -MQ numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so.p/src_umath_override.c.o -MF numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so.p/src_umath_override.c.o.d -o numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so.p/src_umath_override.c.o -c ../numpy/core/src/umath/override.c
In file included from /opt/hostedtoolcache/Python/3.11.3/x64/include/python3.11/Python.h:44,
                 from ../numpy/core/include/numpy/npy_3kcompat.h:15,
                 from ../numpy/core/src/common/npy_pycompat.h:4,
                 from ../numpy/core/src/umath/override.c:4:
../numpy/core/src/umath/override.c: In function ‘PyUFunc_CheckOverride’:
/opt/hostedtoolcache/Python/3.11.3/x64/include/python3.11/object.h:538:9: error: ‘override_array_ufunc’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  538 |         _Py_Dealloc(op);
      |         ^~~~~~~~~~~~~~~
../numpy/core/src/umath/override.c:319:19: note: ‘override_array_ufunc’ was declared here
  319 |         PyObject *override_array_ufunc;
      |                   ^~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors

seberg · 2023-05-18T10:21:52Z

I think I saw it before at some point and I don't think the compiler is right, just guessing on the "this may happen" side of things. Anyway, initializing to silence the warning is OK.

rgommers · 2023-05-18T11:17:50Z

I don't think the compiler is right, just guessing on the "this may happen" side of things

Indeed, the code wasn't broken but the compiler cannot know. I think using an uninitialized variable that is then used only within an if-statement controlled by some other not-directly-the-same-pointer-condition should always be avoided.

rgommers · 2023-05-18T11:18:05Z

This is good to go now.

mattip · 2023-05-18T11:22:33Z

Thanks @rgommers

rgommers added 03 - Maintenance Meson Items related to the introduction of Meson as the new build system for NumPy labels May 9, 2023

github-actions bot added the 36 - Build Build related PR label May 9, 2023

charris closed this May 13, 2023

charris reopened this May 13, 2023

rgommers force-pushed the meson-updates branch from 98a8b72 to c235dd5 Compare May 18, 2023 08:22

rgommers removed the 36 - Build Build related PR label May 18, 2023

mattip reviewed May 18, 2023

View reviewed changes

BLD: updates to the Meson build

941bc5b

Mostly synced from SciPy, and also addresses some warnings that show up with Meson 1.1.0 [skip circle] [skip azp]

rgommers force-pushed the meson-updates branch from c235dd5 to 941bc5b Compare May 18, 2023 09:08

MAINT: add missing NULL pointer initialization

bf703b6

mattip merged commit bfe5f7a into numpy:main May 18, 2023

rgommers deleted the meson-updates branch May 18, 2023 11:31

seberg mentioned this pull request Aug 1, 2023

MAINT: Massive update of files from main for numpy 1.26 #24308

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BLD: updates to the Meson build #23740

BLD: updates to the Meson build #23740

rgommers commented May 9, 2023

mattip commented May 9, 2023

rgommers commented May 10, 2023

rgommers commented May 11, 2023

mattip commented May 11, 2023

rgommers commented May 11, 2023

charris commented May 13, 2023

mattip May 18, 2023

rgommers May 18, 2023

rgommers commented May 18, 2023 •

edited

Loading

rgommers commented May 18, 2023

rgommers commented May 18, 2023

seberg commented May 18, 2023

rgommers commented May 18, 2023

rgommers commented May 18, 2023

mattip commented May 18, 2023

	error('SciPy requires Cython >= 0.29.34')
	error('NumPy requires Cython >= 0.29.34')

BLD: updates to the Meson build #23740

BLD: updates to the Meson build #23740

Conversation

rgommers commented May 9, 2023

mattip commented May 9, 2023

rgommers commented May 10, 2023

rgommers commented May 11, 2023

mattip commented May 11, 2023

rgommers commented May 11, 2023

charris commented May 13, 2023

mattip May 18, 2023

Choose a reason for hiding this comment

rgommers May 18, 2023

Choose a reason for hiding this comment

rgommers commented May 18, 2023 • edited Loading

rgommers commented May 18, 2023

rgommers commented May 18, 2023

seberg commented May 18, 2023

rgommers commented May 18, 2023

rgommers commented May 18, 2023

mattip commented May 18, 2023

rgommers commented May 18, 2023 •

edited

Loading