Please do not bump the numpy version, we still support back to np1.16

I've updated to use numpy==1.16 for x86 and numpy==1.19 for aarch64 wheels. Will that work?

Can't you just cache the numpy wheels? That way you compute it once but not for future ones, but still have compatibility for all supported numpy versions.

@ksunden I haven't used cache with Github Actions and cibuildwheel. Most examples I see are for when the dependency has its own step. With cibuildwheel, the dependency is with CIBW_BEFORE_BUILD. Not sure how to cache that. Any suggestions?

@janaknat (First of all a disclaimer that I am no expert on cibw)

You can have a build numpy wheel stage that runs first and just populates a cache using pip wheel numpy==1.16.0 (if invalid, otherwise just finish immediately)
(See https://github.com/actions/cache, https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows#using-the-cache-action)

The install step for numpy is a CIBW_BEFORE_SCRIPT that just calls into pip. Instead of going out to pypi, it can just install the wheel directly from the cache.

There may need to be some additional variables to build up the path for the wheel, but nothing too crazy.

Alternatively it may "just work" to just add ~/.cache/pip to the github actions cache, no need to introduce additional steps or explicitly cache the wheel separately... I'd have to play with that a bit, but it should work.

This doesn't really need to go in the rc, but would be nice to have by the time 3.4 is finished.

@ksunden I tried with caching ~/.cache/pip.

Build log: https://github.com/janaknat/matplotlib/runs/1834849484?check_suite_focus=true
Caching: https://github.com/janaknat/matplotlib/blob/7a04819ff0d81d2ef8b9a509f21f781be387c563/.github/workflows/cibuildwheel.yml#L40

Numpy is still getting built. These dependencies are being built inside the manylinux2014-aarch64 container under QEMU as part of cibuildwheel's operation. Not sure if numpy can be cached outside the container with a separate step since that'll pull in the x86 numpy wheel.

For a more finegrained level of control however, the root of the host file system is mounted as /host, allowing for example to access shared files, caches, etc. on the host file system.

from https://cibuildwheel.readthedocs.io/en/stable/faq/

So you can just copy the wheel file onto the host in a cached folder, then copy it back in the before script, if it exists/

Another potential option: making a docker image with numpy preinstalled, I see that cibuildwheel has env vars to specify the docker image (on an architecture specific basis) this would require an outside maintenance step though, so not sure its the best solution long term.

Wouldn't it be simpler/easier to just use this in the pyproject.toml (or as a manual pip install in CIBW_BEFORE_BUILD/ALL, if you really have to): https://github.com/scipy/oldest-supported-numpy ?

I see there's no pyproject.toml. :'( Okay, before build it is.

Also, NumPy dropped manylinux1 wheels for 3.9. With an older pinned NumPy, it should be fine, but something to keep in mind.

@henryiii I've made the changes. oldest-supported-numpy defaults to 1.14.3 for python 3.7 (x86/i386) and 1.17.3 for python 3.8 (x86/i386). aarch64 goes with 1.19.2.

@ksunden I've added a step to build and stash numpy=1.16 aarch64 wheels.

Please do not bump the numpy version, we still support back to np1.16

@tacaswell I am confused about what NEP 29 means (assuming Matplotlib is following NEP 29). On the lead up to NumPy 1.20, there was a debate about whether or not to support Python 3.6. During that discussion, I asked for clarification about what the purpose of NEP 29 was. I said that I thought it was a mechanism to get projects to drop support for old versions of Python (and NumPy); but that in practice it looked more like it was actually intended to keep projects from dropping versions of Python (and NumPy) too soon. In that discussion, Ralf said he thought it was intended to get projects to drop old version and not to keep them from dropping version too soon.

If it is the case the NEP 29 was intended to provide projects with a schedule for dropping old versions, shouldn't Matplotlib 3.4 drop support for NumPy 1.16 since 3.4 will be released after Jan 13, 2021?

Consensus on the call is that we are going to leave the setup.py requirement, runtime checks and the tests on the 3.4.x branch at 1.16, but build the wheels with 1.17 (assuming that there is enough meta-data available in wheels for that to be possible).

Won’t that end up with broken wheels if a user has NumPy 1.16, rather than upgrading NumPy or showing a proper error? Lying about requirements is dangerous?

The other way round would make more sense, wouldn’t it?

I agree we should not lie about versions!

Are wheels / pip smart enough to fall back to compiling from source if numpy1.16 is installed? On a bit more consideration I suspect not.

This is an annoying edge case. Because we planned to release before the deadline the code support np1.16 and is tested as such. We could bump the min version to 1.17 which would match the letter-of-law for NEP29 but there would be no other benefit (in terms of being able to have made use of new numpy features) and a cost of making it uninstallable for people on numpy 1.16 despite the code actually being able to run.

Would bumping to 1.17 make getting arm wheels to build any easier?

Would bumping to 1.17 make getting arm wheels to build any easier?

Arm wheel support started with numpy==1.19. So, no change in build ease with numpy==1.17.

@tacaswell NEP 29 is currently self-contradicting: it tries to specify both minimum support required and minimum support allowed. The latter is what we should specify so that everyone can get in sync.

My thought would be to leave this at 1.16, then bump master to 1.17 and look for things that can be simplified, etc. I think this appears to be too far along to make any changes that would actually take advantage of 1.17, so I think it should be declared this release was targeting an older date, which is why it's 1.16. Though it could be bumped to 1.17, it’s just sad that it can’t be taken advantage of. L

Quoting from https://github.com/scipy/oldest-supported-numpy#can-i-use-this-if-my-package-requires-a-recent-version-of-numpy:

In many cases, even though your package may require a version of Numpy that is more recent than the pinned versions here, this is often a runtime requirement, i.e. for running (rather than building) your package. In many cases, unless you use recent features of the Numpy C API, you will still be able to build your package with an older version of Numpy and therefore you will still be able to use oldest-supported-numpy. You can still impose a more recent Numpy requirement in install_requires.

I would still strongly recommend using oldest-supported-numpy, as they cover a lot of cases that would have to be manually listed and maintained here: https://github.com/scipy/oldest-supported-numpy/blob/master/setup.cfg

If you want to ensure a build time requirement that is higher, you can always override just a few lines of that with conditionals. Or ask for a NEP 29 version of oldest-supported-numpy. :)

The latter is what we should specify so that everyone can get in sync.

I'm not sure what this means. There is no benefit to matching minimum versions of NumPy, and all the NEP 29 libraries are not released in sync anyway, so there always will be mismatches. As long as it's not more than a version it shouldn't be a problem, I’d think.

The latter is what we should specify so that everyone can get in sync.

I'm not sure what this means. There is no benefit to matching minimum versions of NumPy, and all the NEP 29 libraries are not released in sync anyway, so there always will be mismatches. As long as it's not more than a version it shouldn't be a problem, I’d think.

The problem is that the community are hesitant to move on, so they keep supporting older versions (AKA burning developer and CI cycles). It would be good for the community to agree at a pace that all the projects will move forward at.

I would repeat: As long as it's not more than a version it shouldn't be a problem, I’d think.

I think the libraries that agreed on NEP 29 have been dropping support for older versions. The only problem here is that 3.4 was supposed to come out before NumPy 1.16 was to be dropped but got delayed. I think it's quite fine just to keep 1.16 for the release, and drop 1.16 for master, where actual improvements (like __array_function__ or something) possibly could be taken advantage of. Having to change in the middle of release cycle just because it passes an arbitrary date is not reasonable.

Yes, sorry if it wasn't clear: I was arguing for improving the NEP wording to state its intent more clearly, not for changing the version of NumPy being used here.

On a bit more consideration, getting things set up so that we can build the wheels with np117, but get it to fallback to the tarball when the user has np116 installed seems like it will be someplace on the a-lot-of-work to impossible scale.

Contrary to my previous comment, I think we should just keep going with np116 for mpl34. I think it is still in the spirit of NEP29 (we meant to release before the cut off, we started the final release work before the cut off, etc).

I agree with @stefanv that we should make the NEP29 wording clearer on the intent. I think the thing we should make clearer is that you projects should look at when they plan to release, look at the calendar and then make the supported version decisions very early in the development cycle (now is the right time to make those decisions for mpl35) and then stick with those versions through the release. The longer time horizon of the planning is how you get the smoothing effect on the support windows so you can avoid exactly this conversation.

I pushed two fixes here, to move the version into a variable used more often, as I said, and to fix the cache key, as it was previously doing nothing.

Thank you for all of your work and persistence on this @janaknat !

Hopefully the next release we will have aarch weels up :)

@tacaswell Thanks a bunch.

@tacaswell Any ideas on when the next release will be?

Before Jan 13, 2021 last I heard. :-P

@meeseeksdev backport to v3.4.x