@tylerjereddy Do you need this PR for scipy to start migrating to cibuildwheel? I'm a little busy right now but I can try to find some time to finish up the last bits of this PR if its needed urgently.
This should be the last piece of the cibuildwheel migration for numpy.

Nope, I was just curious and reading the code, for now. I suspect there be a bit of a "wait and see" before adopting in SciPy after NumPy embraces it.

Nice. I like it that we can get the artifacts from the github action summary.

The new sdist has a LICENSES_bundled.txt file and a numpy.egg-info/ directory that are not found in the sdist uploaded to PyPI.

In both cases, the sdist includes the same unprocessed LICENSE.txt file, I think the missing file in the PyPI sdist is a mistake.

As for the numpy.egg-info directory, it seems to be a build artifact and I think should not be included in the sdist.

In both cases, the sdist includes the same unprocessed LICENSE.txt file, I think the missing file in the PyPI sdist is a mistake.

LICENSES_bundled.txt should be concatenated into LICENSE.txt, so that it's not included separately in the sdist seems right to me. The explanation is probably that setuptools is imported too early, and the MANIFEST.in file is ignored. It contains:

# Exclude license file that we append to the main license when running
# `python setup.py sdist`.  And exclude generated files in repo root.
exclude LICENSES_bundled.txt

My mistake, the LICENSE.txt in both sdists is the processed one.

Thanks for the reviews. I think the main issue right now is that the PEP 517 build is going through setuptools implicitly. This is fine, but it means that there are some changes.

I think I got the license file issue resolved(it was because everything that matched LICENSE was included if not explicitly declared, and license files can't be excluded by setuptools(only distutils), from MANIFEST.in).

Unfortunately, there is no way to remove .egg-info during the build, setuptools needs it to do their version of the MANIFEST thing. Maybe this is fine? (after all the wheels have .dist-info in them, and I'm pretty sure pip will regenerate wheel-info upon install of a sdist without one) Right now, it's proving to be a little bit of a pain to delete the folder from the tarball.

I am willing to hack my way through the rest of this, but I just wanted to first get an opinion on using setuptools. IIUC, numpy is moving away from numpy.distutils/distutils, but I'm not sure whether the conversation on what build system numpy is moving to in the future got resolved.

If setuptools is going to be the preferred way numpy is going to be built in the future, I think it would make sense for me to continue down this path. Otherwise, I think I'll table this until a new build system is decided upon and integrated.

Regardless of build system choice, I think we should probably stick to the PEP 517 way of building sdists, as build isolation/clean environment of CI is probably the main reason for moving sdists building into the CI.

Ah that explains it, thanks @lithomas1. We are going to move away from setuptools, and the setuptools way of generating an sdist is known to be inferior to what distutils does. I'd recommend to just stay with:

git clean -xdf
python setup.py sdist

We've done this since forever, and it's reliable. I don't see a problem keeping it like that for another release or two.

@lithomas1 Off topic, but I will ask here. Do all the github wheel builds get tested? The OS X multibuild wheels have started failing on Python 3.8 and 3.9 (#21169), while all the github builds are getting uploaded to the nightly repo. I'm wondering is this is due to a lack of testing, or if the github builds are working better.

They do get tested(running the "full" tests), but I'm not sure why they would only fail on Azure. I guess something in the configs is different? It might be an issue with the Azure builds as the regular CI here isn't failing.

gentle ping @rgommers @mattip. It would be nice to get this merged.

I'll have a look now, thanks for the ping @lithomas1

Upload to the Anaconda bucket will only happen from main, so will verify that now with a manual trigger.

The manual trigger doesn't seem to work, is this known @lithomas1, or am I doing something wrong by pressing the trigger on the merge commit?

See https://github.com/numpy/numpy/actions/runs/1993651328

Ah, my bad! There was an unexpected delay after pressing the trigger, and right at that moment my merge commit showed up. So the manual trigger did show up and is running fine now, just ~20 minutes later than expected:

The upload logic here doesn't seem quite right though, the wheels are getting uploaded while the sdist didn't:

See https://github.com/numpy/numpy/runs/5572823596?check_suite_focus=true

This feature got added to cibuildwheel:

• feat: SDist builds pypa/cibuildwheel#1096