Needs a release note. If this might affect backward compatibility that should be mentioned.

How big is the diff? If it is small might be worth posting so we can see what is different.

How big is the diff? If it is small might be worth posting so we can see what is different.

Do you mean the second commit? Or what diff are you referring to?

Missing a line in setup.py to add this to the wheel, so we don't get a repeat of that PR. Also should be added to MANIFEST.in next to __init__.pxd so it shows up in the sdist. Otherwise LGTM, thanks.

We need the check_size ignore, see the long discussion cython/cython#2221.

For those who want some background what all this is, see the details.

I think this should be backported to 1.19.2 since people who use -D NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION will not be able to use Cython 3.0 without it.

I tried building pandas HEAD with this PR and it seems to work, unless I did something wrong.

Also should be added to MANIFEST.in next to __init__.pxd so it shows up in the sdist.

I changed include numpy/__init__.pxd to include numpy/*.pxd. Likewise for setup.py.

We need the check_size ignore

I was merely asking whether we need it for all types or just ndarray. Cython only had it set on ndarray previously. I can certainly add it to all extension types, if that's what you want.

Since this only regards Cython 3.0, I don't really see an issue with backwards compatibility.

I'll leave the decision to you whether to expose the numpy.broadcast attributes/properties or not. If you want to add the properties on your side, I'd be happy to receive a PR for Cython that copies the change back.

If you want to add the properties on your side, I'd be happy to receive a PR for Cython that copies the change back.

I'm a bit confused - why are both numpy and cython distributing this file, and how do users specify which they intend to use?

why are both numpy and cython distributing this file

The file was originally distributed by Cython, until we realised that a representation of NumPy's C-API is better maintained by NumPy in a version specific way. So we created the infrastructure for that, which now allows NumPy to distribute it. However, older NumPy versions do not have it, which is why Cython still ships a copy.

how do users specify which they intend to use?

Cython only uses its own version if it doesn't find one on the Python import path. If NumPy is installed, its version will be preferred. Note that this PR only adds an improved version of the (existing) declarations that requires features from Cython 3.0+. Thus the file suffix that prevents older Cython versions from picking it up. They will continue to use the existing numpy/__init__.pxd file or their own shipped version.

I also found an unused private utility function in the files that was only used by the legacy __getbuffer__ implementation, which was already removed previously. I removed the function in both files in separate commits, in case you want to keep it in the old one (for whatever reason).

LGTM, I will give other reviewers some time to take a look.

Please squash the PR when merging to facilitate the backport.

only used by the legacy getbuffer implementation

@scoder Was the legacy implementation in NumPy or Cython? I ask because this will be backported to NumPy 1.19.x.

Was the legacy implementation in NumPy or Cython?

The code was only in Cython, never in NumPy. The latter only accidentally inherited the unused utility function. It should have been removed before adding the file to NumPy (1.19).

Thanks @scoder

For some reason, I can no longer use np.broadcast in Cython 3.0+ in GIL releasing code or functions that expect C types. I get errors like Cannot convert Python object to 'npy_intp *' and Truth-testing Python object not allowed without gil when using the struct properties. Is there a way to fix this on my side?

You will need to be more precise. cnp.broadcast requires the GIL, its methods probably don't (genrally!). Truth-testing is used where exactly/how?
I could imagine Cython 3 is just better at figuring out that your code was never really allowed to release the GIL.

You will need to be more precise. cnp.broadcast requires the GIL, its methods probably don't (genrally!). Truth-testing is used where exactly/how? I could imagine Cython 3 is just better at figuring out that your code was never really allowed to release the GIL.

Basically, I cannot access the struct properties outlined here: https://numpy.org/doc/stable/reference/c-api/types-and-structures.html#c.PyArrayMultiIterObject . This is only if I upgrade cython to 3.x. This is not an issue in 0.29.X. I noticed that this may be due to the different definition of numpy.broadcast in https://github.com/scoder/numpy/blob/df769ef591f8a21f8e39ee30c80e09d1b7e0a705/numpy/__init__.cython-30.pxd#L230-L232 compared to https://github.com/scoder/numpy/blob/df769ef591f8a21f8e39ee30c80e09d1b7e0a705/numpy/__init__.pxd#L227-L232

Here is an example piece of code where these are meant to be used: https://github.com/zoj613/polyagamma/blob/6696587c8a1ca69090d6fff374a649526f114089/polyagamma/_polyagamma.pyx#L309-L332

Can you open a new issue with the more concrete issues that you need access to the struct because there are no macros for most fields?
(We could consider adding macros, in Cython 3 it is possible to add properties so that attribute access is translated to a macro call.)