I think recent versions of rtools are preinstalled in GitHub actions. I suspect it is possible to split windows builds into separate cases for 32 and 64 bits and use the appropriate version of rtools. Of course, we could also just drop 32 bit builds for 2.0.

Of course, we could also just drop 32 bit builds for 2.0.

This one:) xref gh-23717.

Another option is to use MSVC on windows. We need 2 things from rtools: mingw (for gcc) and gfortran. We used to require mingw in order to mangle the openblas.a file to create an import library, but no longer need to do that, so mingw is no longer required. gfortran is needed for fortran, but that is only needed in the f2py tests. We could skip them on win32 (if gforfrtran is not found).

A few different failures left:

..\..\meson.build:30:4: ERROR: Problem encountered: NumPy requires GCC >= 8.4

Should not be too difficult to resolve, we need the right version from Rtools. @mattip will try to fix that up next week once he has access to his Windows machine.

FAILED ..\build-install\Lib\site-packages\numpy\tests\test_public_api.py::test_array_api_entry_point

The .dist-info directory of the wheel does seem to contain the right entry_points.txt entry, but the entry point doesn't get installed for some reason. I'll look into this.

numpy/core/_multiarray_umath.cpython-310-aarch64-linux-gnu.so: undefined symbol: cblas_cdotc_sub

This is due to the wheel being built against ILP64 OpenBLAS, and the symbol suffix handling for that needs wiring up.

meson-python can handle setting up windows to use MSVC via a setup setting that will be ignored on non-windows.

[tool.meson-python.args]
setup = ['--vsenv']

I think that is also the path to adding 64-bit interfaces via a compile = ['-DBLAS_SYMBOL_SUFFIX=64_'] setting.

I will rebase this off main to clear the conflict and push a commit to add the tool.meson-python.args settings.

hmm, maybe the compile = ['-DBLAS_SYMBOL_SUFFIX=64_'] option is too much: it will also use 64-bit interfaces on 32-bit builds.

Getting somewhere.

The manylinux wheel builds are failing with

FAILED core/tests/test_umath.py::TestComplexFunctions::test_loss_of_precision[complex64]
FAILED core/tests/test_umath.py::TestComplexFunctions::test_loss_of_precision[complex128]
FAILED core/tests/test_umath.py::TestComplexFunctions::test_loss_of_precision[clongdouble]

The musllinux wheel builds have lots of DeprecationWarnings in tests, and fails python /project/tools/wheels/check_license.py

MacOS wheels are failing some einsum tests

FAILED core/tests/test_einsum.py::TestEinsum::test_einsum_sums_int8 - Asserti...
FAILED core/tests/test_einsum.py::TestEinsum::test_einsum_sums_uint8 - Assert...

Windows wheels is failing to find OpenBLAS, the PKG_CONFIG_PATH is probably not propagating properly.

PyPy does not build, it seems the include paths are not right?

meson-python can handle setting up windows to use MSVC via a setup setting that will be ignored on non-windows.

This seems a bit too agressive, it's better to make it specific to cibuildwheel to avoid forcing MSVC usage on everyone who builds from source. Probably this is the way:

[tool.cibuildwheel.windows]
config-settings = {"setup-args=--vsenv"}

or alternatively as an env var:

CIBW_CONFIG_SETTINGS_WINDOWS: "setup-args=--vsenv"

PyPy does not build, it seems the include paths are not right?

I thought that was fixed for SciPy so it should work - good idea to disable that build config like you did and do that separately though.

MacOS wheels are failing some einsum tests

That looks odd. Also there's a test_new_policy failure in the macOS jobs.

It'd be useful to make the cibuildwheel jobs non-verbose, the CI logs are hard to navigate now - the log is 23,900 lines long. It's the verbose=2 usage in tools/wheels/cibw_test_command.sh`. Do you see a reason to keep them verbose?

The musllinux wheel builds have lots of DeprecationWarnings in tests, and fails python /project/tools/wheels/check_license.py

I am unsure why other jobs are not failing on that, but the LICENSE.txt file is not installed into site-packages/numpy on purpose. It belongs in .dist-info not inside the source tree.

   + python /project/tools/wheels/check_license.py
  Traceback (most recent call last):
    File "/project/tools/wheels/check_license.py", line 54, in <module>
      main()
    File "/project/tools/wheels/check_license.py", line 38, in main
      with open(license_txt, encoding="utf-8") as f:
  FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp.MGkClm/venv/lib/python3.10/site-packages/numpy/LICENSE.txt

So it's the test that needs tweaking here.

The manylinux wheel builds are failing with

FAILED core/tests/test_umath.py::TestComplexFunctions::test_loss_of_precision[complex64]

Those seem to be gone, at least I checked two jobs and it only shows the same test_new_policy failure as on macOS.

It's the verbose=2 usage in tools/wheels/cibw_test_command.sh`.

I will reduce it

make it specific to cibuildwheel to avoid forcing MSVC usage on everyone

I will move the directive

So it's the test that needs tweaking here.

I will try to fix that

The wheels are building with meson setup ... -Dbuildtype=release -Db_ndebug=if-release which sets the compiler optimization to -O3. I think this is the source of some of the test failures. The -Ddbuildtype=release is a meson-python default, where the non-wheel builds are using meson without meson-python. I think this should be a general default overrid for meson-python, we don't want users getting failed tests out of the box. At some point we should fix the failures.

@mattip do we want to default to -O2, like distutils does? Seems fine to me, I don't have a strong preference here. We actually have different defaults hardcoded per compiler in numpy.distutils, e.g. the Intel compilers are using -O1. Overall that's a bit of a mess, so I think having a single uniform default and then treating the rest as either numpy bugs or compiler bugs seems cleaner.

Using

[tool.meson-python.args]
setup = ['-Dbuildtype=custom','-Doptimization=0', '-Db_ndebug=true']

gives me

   User defined options
      Native files: /project/.mesonpy-fjiq3ch_/build/meson-python-native-file.ini
      buildtype   : custom
      optimization: 0
      b_ndebug    : true
      b_vscrt     : md

but the three tests still fail. I will commit what I have so far so others can try. I am not sure what is the delta to the other meson builds or to th distutil builds.

There are also DeprecationWarnings Conversion of an array with ndim > 0 to a scalar is deprecated in tests.

There is still a problem with the license file. I think the syntax here

# Using https://peps.python.org/pep-0639/ which is still in draft
license = {text = "BSD-3-Clause"}
license-files.paths = [
    "LICENSE.txt",
    "LICENSES_bundles.txt"
]

is different from the syntax in the meson-python test

[project]
name = 'license-file'
version = '1.0.0'
license = { file = 'something/LICENSE.custom' }

There is still a problem with the license file. I think the syntax here

Ah right, yes I had counted on PEP 639 moving forward by now. Let me have a look at this one right now.

I think you can add CIBW_ARCHS_WINDOWS=auto64 around here in the wheel.yml file

Ah thanks, that explains it! The skip's in pyproject.toml seem entirely unused then, the jobs are indeed controlled in wheels.yml.

The 32-bit long double detection hack isn't quite working: undefined symbol: Dragon4_PrintFloat_Intel_extended128. May have to punt on that.

The macOS arm64 on Cirrus CI are still failing with:

_multiarray_umath.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace (_cblas_caxpy)

That's related to forcing NPY_USE_BLAS_ILP64="1" but not using -DBLAS_SYMBOL_SUFFIX=64_ -DHAVE_BLAS_ILP64 in CFLAGS unlike on GHA for x86-64.

Other than that, this is getting close. Trying to get it green by tonight.

Okay, this is ready to go in, phew:) All the follow-ups to this PR are listed in gh-23981.

This is still green after merging in the long double format detection fix from main. The commit history is clean enough I'd say and useful, so I'll prefer to merge it as is.

I'll go ahead and merge this to keep the ball rolling. Thanks for the reviews and help @mattip!