@seiko2plus, cibuildwheel has an example CI setup that seems to cover what you're trying to do here,https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation. It uses a GH Action to setup the emulation. It's probably worth investigating that path before iterating too much here.

It may be easier to iterate CI configs on your personal fork, it saves a lot of CI triggering on the main repo.

cibuildwheel has an example CI setup that seems to cover what you're trying to do

I couldn't find a guide for native cross-compiling. What I'm trying to do is use binaries cross-compiled on the host system, which includes the "ninja" tool, while building. This mixes binaries from emulated architectures with the host's x86_64 architecture via binfmt.

It may be easier to iterate CI configs on your personal fork, it saves a lot of CI triggering on the main repo.

My bad, I have moved the tests to my local machine

Meson's Python module doesn't support crosscompiling

We've built macosx_arm64 wheels for scipy on an intel build machine using cibuildwheel+meson. So I'm not sure if this statement is true.

Hmm. Do you mean that you have a cross compiler that builds on x86_64, but produces output for an armhf host? If so, then it's not clear why you have to use QEMU.

If you're using QEMU aren't you using emulation to run a native armhf compiler to build for an armhf host? If this is the case then why not use https://github.com/docker/setup-qemu-action to start the docker container?

https://github.com/scipy/scipy/pull/17580/files shows the changeset for macosx cross compilation with scipy.

macOS x86-64 to arm64 cross-compilation support is a special case, because Apple makes it extraordinarily easy. Beyond that, cibuildwheel has no support. Meson does, but there are still problems to resolve that are specific to the python module in Meson mostly (see mesonbuild/meson-python#321 (comment)).

I think this binfmt approach works for now, however we should aim to replace it with a regular Meson cross file once the Python-specific cross build issue is resolved. And then we'd like the workflow to be to build a wheel for armhf & co on x86-64 with something like python -m build --wnx -Csetup-args="--cross-file=x86toarmhf.txt". And then still install and run tests for that in a similar way as in this PR with qemu-user-static.

Also: it'd be great to get this to work, so we can retire Travis CI usage. Thanks for working on it @seiko2plus.

All green, the runtime tests only covers "test_kind, test_multiarray, test_simd, test_umath and test_ufunc" which is good enough I suppose. The full test takes up to 25m without container cache involved, not sure how to test the cache yet.

I made a manual re-run for armhf with debug enabled but still missed the cache:

 ##[debug]Resolved Keys:
##[debug]["container-Linux-arm-linux-gnueabihf-arm32v7/ubuntu:22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6"]
##[debug]Checking zstd --quiet --version
##[debug]1.5.5
##[debug]zstd version: 1.5.5
##[debug]Resource Url: https://acghubeus2.actions.githubusercontent.com/sH05vHzoTO8FskJ61ZFM2de2fOtiqhFE8SSrZeKfiH7OXqclo7/_apis/artifactcache/cache?keys=container-Linux-arm-linux-gnueabihf-arm32v7%2Fubuntu%3A22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6&version=82cace457337bec689d5d2c52d6ee729c5a67a454f53c4372afbe7269f7d18f6
##[debug]Resource Url: https://acghubeus2.actions.githubusercontent.com/sH05vHzoTO8FskJ61ZFM2de2fOtiqhFE8SSrZeKfiH7OXqclo7/_apis/artifactcache/caches?key=container-Linux-arm-linux-gnueabihf-arm32v7%2Fubuntu%3A22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6
##[debug]Failed to delete archive: Error: ENOENT: no such file or directory, unlink ''
Cache not found for input keys: container-Linux-arm-linux-gnueabihf-arm32v7/ubuntu:22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6
##[debug]Node Action run completed with exit code 0
##[debug]Save intra-action state CACHE_KEY = container-Linux-arm-linux-gnueabihf-arm32v7/ubuntu:22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6
##[debug]Finishing: Cache docker container

I made a manual re-run for armhf with debug enabled but still missed the cache:

The Post Cache docker container step contains:

Warning: Path Validation Error: Path(s) specified in the action for caching do(es) not exist, hence no cache is being saved.

The path it's looking for is:

path: /docker_s390x-linux-gnu

and it's being saved to:

  mkdir -p "~/docker_${TOOLCHAIN_NAME}"
  docker save -o "~/docker_${TOOLCHAIN_NAME}/the_container.tar" the_container

with TOOLCHAIN_NAME: s390x-linux-gnu. So it mostly looks right, but is the ~/docker vs /docker correct here?

@rgommers, oh nice catch you made my day!

Another thing we have a limit of 10GB cache currently about 8.5GB is been used, is there away to increase it?

No, it's 10 GB per repository (https://github.blog/changelog/2021-11-23-github-actions-cache-size-is-now-increased-to-10gb-per-repository/) and you can't even pay to increase it.

How much more do we need with these new jobs? Or is it 8.5 GB including these jobs, and you're worried about cache evictions on re-runs?

How much more do we need with these new jobs?

Currently three containers one for each architecture, all of them may exceed 1.5GB and we may need to adds support for risc-v, and ppc64(big-endian).

Or is it 8.5 GB including these jobs,

without including these jobs.

you're worried about cache evictions on re-runs?

yes, saving about 8 minutes on each run may allow us to extend the current runtime tests to cover more unites.

Can we cache in a third party location, akin to nightly wheels? E g. Dockerhub. An account for OSS projects can have up to 200 image pulls every six hours.

Can we cache in a third party location, akin to nightly wheels?

Yes, we can since the cache hit can be detected to enable/disable steps.

E g. Dockerhub. An account for OSS projects can have up to 200 image pulls every six hours.

LGTM, lets just see first how github manages the cache storage once its exceed 10GB.

Can we cache in a third party location, akin to nightly wheels? E g. Dockerhub. An account for OSS projects can have up to 200 image pulls every six hours.

It's possible, but caching outside of GitHub is way slower. We used Docker Hub for the Gitpod images, which were also ~1.5GB and those took a couple of minutes to transfer and load. Plus we got rid of a lot of images after Docker Hub made changes to their free team plan. So I'm not very eager to look at it now.

LGTM, lets just see first how github manages the cache storage once its exceed 10GB.

+1, we can reassess after merging this.

I made a re-run but it doesn't work, and it seems the cache will never hit, so after checking the debugging URLs, I just realized that its seem to be only allowed through the default branch maybe due to security concerns:

{"$id":"1","innerException":null,"message":"The user 'System:PublicAccess;aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is not authorized to access this resource.","typeName":"Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server","typeKey":"UnauthorizedRequestException","errorCode":0,"eventId":3000}

The user 'System:PublicAccess;aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is not authorized to access this resource

So it seem there's no way to handle it through github.

According to the github doc https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache, there's still chance to works after the merge in order to gain the write access.

Let's put this in and iterate on the caching setup as needed.

Thanks @seiko2plus

Awesome! @seiko2plus can we delete .travis.yml now, or do you still need it for something?

The cache page seems to show we have only one cached image: a 390MB one named 3.1.32-linux-x64-master that is built quite often.

That's from the Emscripten job, and it's because the mymindstorm action caching is broken:

Run mymindstorm/setup-emsdk@ab889da2abbcbb280f91ec4c215d3bb4f3a8f775
Warning: No cached files found at path "/home/runner/work/numpy/numpy/emsdk-cache" - downloading and caching emsdk.

We have nothing else that uses a cache right now, so we should be fine space-wise.

can we delete .travis.yml now, or do you still need it for something?

I just left it for testing distutils, no other reason for keeping it.