Ugh:

tart isolation failed: failed to create VM cloned from "ghcr.io/cirruslabs/macos-sonoma-xcode:15": tart command returned non-zero exit code: "Error: The file “15” couldn’t be saved in the folder “macos-sonoma-xcode” because a file with the same name already exists."

I failed to get Cirrus CI installed on my own fork - guess I'll have to try harder.

Hmm, there are a large amount of f2py errors and failures in the wheel builds:( I'll have to have a look at the open PRs with fixes and recent changes I guess.

Good news is that the macOS 14 + XCode 15 image is now running. Bad news is that it needs more tweaks, it dies quickly when talking to Homebrew:

See: https://docs.brew.sh/Homebrew-and-Python
ln -s python3 /opt/homebrew/opt/python@3.10/bin/python
which python
����

Something about gfortran in CI is not working:

gfortran: warning: could not understand version �14.0�
ld: warning: ignoring duplicate libraries: '-lemutls_w', '-lgcc', '-lgfortran'
ld: library 'bundle1.o' not found
collect2: error: ld returned 1 exit status
error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle /var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp56jwuv5g/var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp56jwuv5g/src.macosx-11.0-arm64-3.10/_test_value_attrspec_TestValueAttr_ext_modulemodule.o /var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp56jwuv5g/var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp56jwuv5g/src.macosx-11.0-arm64-3.10/fortranobject.o /var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp56jwuv5g/var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp8ye9twj6/gh21665.o /var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp56jwuv5g/var/folders/c2/b2kmn__12wq57m82ltz0hl1w0000gn/T/tmp56jwuv5g/src.macosx-11.0-arm64-3.10/_test_value_attrspec_TestValueAttr_ext_module-f2pywrappers2.o -L/opt/gfortran-darwin-arm64-native/bin/../lib/gcc/arm64-apple-darwin20.0.0/11.3.0 -L/opt/gfortran-darwin-arm64-native/bin/../lib/gcc/arm64-apple-darwin20.0.0/11.3.0/../../.. -L/opt/gfortran-darwin-arm64-native/bin/../lib/gcc/arm64-apple-darwin20.0.0/11.3.0/../../.. -lgfortran -o ./_test_value_attrspec_TestValueAttr_ext_module.cpython-310-darwin.so" failed with exit status 1
6 failed, 40769 passed, 374 skipped, 48 xfailed, 4 xpassed, 35 warnings, 215 errors in 85.86s (0:01:25)

bundle1.o should be found under SDKROOT.

On the 11.0 deployment target job, getting:

ld: library not found for -lm

for f2py tests is also a sign of some env var like LD_LIBRARY_PATH not being set the way gfortran wants it.

This is not the cause, since I see the same message on a run that just passed on main:

Auto-upgraded from ghcr.io/cirruslabs/macos-monterey-xcode:14 to ghcr.io/cirruslabs/macos-monterey-xcode:latest

I seem to remember something about needing the sdk-path in the environment

        # if [ "macos-11" == "${{ matrix.os }}" ]; then
        #   Use xcrun --sdk macosx --show-sdk-path instead of hardcoding the path
        #   echo "LDFLAGS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/lib" >> $GITHUB_ENV;
        #   echo "LIBRARY_PATH=-L/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/lib" >> $GITHUB_ENV;

Ahhh, CI debugging!

I think this might be a call for some @isuruf magic. From memory when we see errors like ld: library not found for -lm, that usually means theres something gone wrong with the gfortran toolchain.

Having said that, the wheel build seems to be working, but it's the test that is failing. I'm wondering if the subprocess.Popen call is getting the same set of environment variables that are supplied to the original build. For example, if SDKROOT is not set properly, then I think you'll see those errors. Can you check what environment variables are being given to/inherited by the call? I wouldn't be surprised if cibuildwheel doesn't supply them in the test run.

Yeah, some paths are wrong. I think @isuruf's suggestion may be right; there may be other things wrong. The problem is that the setup is so brittle that it's almost impossible to work with:

• Homebrew Python is used, needs symlinking and a hardcoded PATH
• OpenBLAS needs a bunch of manual copying around and LD_LIBRARY_PATH
• Gfortran needs LD_LIBRARY_PATH and SDKROOT hardcoded
• There's also a DYLD_LIBRARY_PATH floating around inside one of the shell scripts for one of the above
• It's not possible to dynamically determine things like SDKROOT, that doesn't carry over well between cibuildwheel steps, so it needs hardcoding in CIBW_ENVIRONMENT_MACOS - and then good luck determining if a new OS image changed it or not

I'm going to have to start over and change less in the shell scripts. I am also going to drop gfortran completely from the Accelerate jobs (and possibly the OpenBLAS wheel build jobs too); we don't need any Fortran compiler really - the f2py tests don't need to run in every job, nothing can be wrong if those pass in regular CI, since no compiled code inside the numpy wheel is involved (any errors are going to be CI config, not f2py bugs).

Things can definitely be made more robust, but to be fair it's not the easiest to work in CI directly. I wonder if it'd be easier to install something like miniconda, and use that for python, compiler (gfortran) provision. This would hopefully get rid of the SDKROOT issues.

GHA is better in this regard because there are pre-written Actions that do all this setup for us.

Yes, agreed - definitely not easy to work with this setup. I think that's the right plan (probably micromamba rather than miniconda, faster and more self-contained). Since the gfortran we use is repackaged from conda-forge, I think we should be able to use it directly from conda-forge instead. @isuruf does that sound right?

For OpenBLAS we should stay with our own builds I believe - but of course we're moving to the openblas-scipy64 wheel, which will avoid all the hardcoded paths and copying around as well.

Since the gfortran we use is repackaged from conda-forge, I think we should be able to use it directly from conda-forge instead. @isuruf does that sound right?

conda-forge toolchain also requires this variable. Let me send a PR to fix this more robustly.

See rgommers#59

I'm going to investigate https://tart.run/quick-start/ as a way of running a macos container locally.

I've had good success with using tart to run a macOS container locally, such that would be used on CirrusCI (the ghcr.io/cirruslabs/macos-sonoma-xcode:latest image). @rgommers, if you have a mac it's going to be much faster to use a local container than try and fix things in CI.

Rough Steps:

1. brew install cirruslabs/cli/tart on your local machine
2. tart clone ghcr.io/cirruslabs/macos-sonoma-xcode:latest sonoma-xcode (this is a large download).
3. tart run sonoma-xcode &. This will start a tart window with a Sonoma GUI in it.
4. you can connect to the container using ssh admin@$(tart ip sonoma-xcode) from the host computer.
5. In order to get cibuildwheel to work you'll need to use the tart UI window to install a CPython. This is because when you eventually run cibuildwheel in the container it needs the official CPython install. Only when cibuildwheel is run in CI will it use other python installs.
6. Navigate to Applications/CPython and run the install certificates command file. Otherwise you won't be able to use pip to install anything.
7. Go back to the ssh window to the container from step 4 (alternately start up a terminal window in the tart UI window). You'll have to use git to clone the repo.

This whole setup gives you a container that is identical to that used to build wheels on cirrusci, but on your local machine.
From the ssh connection I did: brew install micromamba, created a conda env, activated that conda env, installed the conda-forge compilers package and python3.11 from conda-forge, pip installed cibuildwheel, set the appropriate CIBW env vars from the numpy wheel build configuration, then ran cibuildwheel --platforms macos. The whole build and test worked without having to set SDKROOT, LD_LIBRARY_PATH, or any of the other variables.

Thanks for the fix @isuruf and for investigating Tart @andyfaff! That local testing setup looks really useful. I'm traveling and on a slow connection right now, but I'll try it out next week.

Actually you can install the cirrus-cli, brew install cirruslabs/cli/cirrus, once you've installed tart. This will enable you to do a cirrus run from the numpy directory, which will run a .cirrus.yml file. You have to create that file, I just copied the relevant files that I was trying to modify.

Noting that #25255 is an alternate to this PR

The commits from this PR not directly related to Accelerate wheel builds are now merged. The rest of this is superseded by gh-25255, where @andyfaff got the build to work. So I'll close this issue. Thanks all!