ENH: add a pkg-config file and a numpy-config script
#25730
Conversation
| # ------------------------- | ||
|
|
||
| # Note: we can't use Meson's built-in pkgconfig module, because we have to | ||
| # install numpy.pc within site-packages rather than in its normal location. |
There was a problem hiding this comment.
This is technically not true because you can tell pkgconfig.generate() to use a specific install_dir.
Note that in order to handle relocatable wheels you'd then still want to define pkgconfig.relocatable=true inside cibuildwheel and then calculating it as offset from the wheel root instead of /usr would be a bit interesting to negotiate, because no one has actually tried to account for pkg-config files plus headers installed to something that isn't actually the --prefix. Hence it assumes that it needs to relativize everything based on prefix in order to recurse back down to includedir.
So you may prefer configure_file but not for the reasons this comment suggests. Or you may wish to propose changes to the pkgconfig module too. :)
There was a problem hiding this comment.
Hence it assumes that it needs to relativize everything based on prefix in order to recurse back down to includedir.
Indeed. I did try to use pkgconfig.generate(), the problem is that there is no way to tell it where the include dir is.
Or you may wish to propose changes to the pkgconfig module too.
There's two things indeed:
- the docs assume that a library is the first input, it's unclear that header-only support is actually working
includediris now actually a reserved variable; to make this work that restriction should be dropped or a new keyword to set the include dir relative to the install dir should be added.
This is simpler (advantage) but possibly pollutes the include directory and makes updating/removing more compicated (disadvantage). It also get confusing when that system includedir is managed by the operating system package manager (disadvantage). |
I'd be interested in doing this as a follow-up perhaps. The complication though is that it only makes sense I think if we install the headers to |
|
@mattip that's true for wheels installed by the user (and hence it can never be the default), but it doesn't quite apply to distro packaging. Distros that only ship one Python version (and hence one NumPy version) may prefer to have things in the standard lib/include directories. |
Not even. Gentoo allows multiple python versions and your choice of numpy versions, but only one numpy version can be installed at a time (and will be installed for all python versions you ask for). For a set of static headers there is no reason they have to be installed multiple times. The only reason they are installed multiple times if installed to site-packages is as a side effect of making it easier to (It's not a significant savings, I'll grant you that.) |
Good point. However, we do have generated headers as well, and some of their contents may not only be platform-dependent but also depend on |
This will make it much easier to support getting at the location
for headers and to support using numpy as a dependency in downstream
build config files. It should enable removing the complexity of
importing numpy and parsing the output of calling `np.get_include` and
co manually. Instead, things should work transparently with (in the
Meson case): `dependency('numpy')`. CMake is also able to consume
pkg-config files and/or a config tool.
Note that the `numpy.pc` file may or may not be installable in the
standard location expected by pkg-config. By default in a wheel it
isn't; distros are in principle able to target the correct location
(however, no flag to do so is added here, it can be done by hand only
right now).
The advantage of the `numpy-config` tool is that it goes into the
scripts directory (i.e., `site-packages/bin`) and hence is always
available. The advantage of the pkg-config file is that it works without
running Python, and hence is better for cross-compiling.
Support for getting at the numpy/f2py/src directory isn't included here,
because that isn't a regular include directory but rather includes
a source file. And hence a compile arg `-I/path/to/numpy/f2py/src`
is not enough. How to better support that requires some more thought.
In its current form, this should already help most packages using the
NumPy C API.
And to close the loop on the pybind11 comparison, pybind11 doesn't generate headers -- if it does care about the python version it does it in the preprocessor and not in a generator script that creates headers at build time. |
|
LGTM, thanks @rgommers. Let's put this in as-is, we can iterate on it as we see how it works with downstream projects. |
|
Thanks for the quick reviews & merge. On to a Meson PR to now use this:) |
This will make it much easier to support getting at the location for headers and to support using numpy as a dependency in downstream build config files. It should enable removing the complexity of importing numpy and parsing the output of calling
np.get_includeand co. manually. Instead, things should work transparently with (in the Meson case):dependency('numpy'). CMake is also able to consume pkg-config files and/or a config tool.Note that the
numpy.pcfile may or may not be installable in the standard location expected by pkg-config. By default in a wheel it isn't; distros are in principle able to target the correct location (however, no flag to do so is added here, it can be done by hand only right now).The advantage of the
numpy-configtool is that it goes into the scripts directory (i.e.,site-packages/bin) and hence is always available. The advantage of the pkg-config file is that it works without running Python, and hence is better for cross-compiling.Support for getting at the numpy/f2py/src directory isn't included here, because that isn't a regular include directory but rather includes a source file. And hence a compile arg
-I/path/to/numpy/f2py/srcis not enough. How to better support that requires some more thought. In its current form, this should already help most packages using the NumPy C API.The
numpy.pcfile can be used with Meson directly, without needing an upstream change. To do so, try:To add
numpy-configsupport to Meson does require an upstream change, but it's a straightforward ~12 LoC diff: https://github.com/mesonbuild/meson/compare/master...rgommers:meson:add-numpy-dependency?expand=1I wanted to get this in for 2.0.0, since that may be the lowest supported version for quite a while. Having this in will allow most downstream projects to drop their verbose handling of NumPy include paths completely.
@eli-schwartz maybe you want to have a look at this?