-
-
Notifications
You must be signed in to change notification settings - Fork 10.8k
ENH: Meson build system support for F2PY's f2py -c
command
#22225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This reverts commit b334020.
from string import Template | ||
|
||
class MesonTemplate: | ||
"""Template meson build file generation class.""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's tabs here instead of spaces. @NamamiShanker can you please check your editor settings and ensure tabs are always converted to spaces?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure @rgommers. I corrected this file and the others.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like nice progress, thanks @NamamiShanker. I just had a quick look through the build system part of this and made a few comments for things that stood out to me.
py3 = py_mod.find_installation('python3') | ||
py3_dep = py3.dependency() | ||
message(py3.path()) | ||
message(py3.get_install_dir()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should have pure: false
as argument.
@@ -0,0 +1,22 @@ | |||
project('${modulename}', 'c', | |||
version : '0.1', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you want to set the minimum meson
version here, and it should be very recent; lots of Python bugs were fixed recently. I recommend the most recent release, 0.63.2
, and bumping it up to 0.63.3
/0.64.0
when those become available.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Corrected it.
@@ -26,6 +26,9 @@ | |||
def configuration(parent_package='', top_path=None): | |||
config = Configuration('f2py', parent_package, top_path) | |||
config.add_subpackage('tests') | |||
config.add_subpackage('backends') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only needed if there's a backends/setup.py
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not much familiar with distutils
configuration files. Would it be more apt to use the add_data_dir
function instead? With the current configuration, the backend
folder will not be included in the installed numpy/f2py
folder.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add_data_dir
is more appropriate. However, after looking at how it's done elsewhere, add_subpackage
is used more often (even if it gives a warning). So you can leave it as is.
@@ -26,6 +26,9 @@ | |||
def configuration(parent_package='', top_path=None): | |||
config = Configuration('f2py', parent_package, top_path) | |||
config.add_subpackage('tests') | |||
config.add_subpackage('backends') | |||
config.add_data_dir('backends/src') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this does anything, the next line covers it already. You should need one or the other I believe.
@@ -246,18 +246,24 @@ def test_NPY_NO_EXPORT(): | |||
"distutils.unixccompiler", | |||
"dual", | |||
"f2py.auxfuncs", | |||
"f2py.backends", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This does not look like it should be public, or should it? If not, can you please name it f2py._backends
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And the same for all the other things - nothing should be added to this list ideally.
if opt_flags is None: | ||
opt_flags = [] | ||
if arch_flags is None: | ||
arch_flags = [] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These flags don't quite fit anymore (the difference between F77 and F90 is not needed, nor are the exact paths to the compiler; the --opt
, --arch
etc. are specified in a very non-standard way, Meson has a better way to do things like debug
flags). Is there a plan to clean this up? Maybe they should be kept unchanged for now, and then when numpy.distutils
is gone they can be updated?
I am interested in this because I am porting the build system of another package (PySCeS) to try and get rid of the This PR seems to provide just what I need so I tried only compiling the Fortran extension modules on the command line. This is with numpy installed from Using f2py with the default Pertinent versions:
Navigating to the nleq2 source directory, and executing f2py -c nleq2.pyf nleq2.f wnorm.f linalg_nleq2.f zibconst.f zibmon.f zibsec.f works fine and the *.so binary module file is created, and can be imported from Python. However, executing the following fails: f2py -c --backend meson nleq2.pyf nleq2.f wnorm.f linalg_nleq2.f zibconst.f zibmon.f zibsec.f The traceback is too big to include inline, I attach it as a file. We use a custom Trying to debug this, I used a custom |
f2py -c
commandf2py -c
command
@HaoZeke @NamamiShanker we're now close to the 1.26.0 release, and this is still pending. I think we do need this for Python 3.12 support of using |
I overhauled the random example cython compilation to use meson in #24153, maybe worth seeing if anything there is applicable here. |
I'll aim to have this and the previous dependent PR in before the weekend. Should be followed up to ensure most projects start using it (and submit bug reports if needed). |
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
This sounds like a frontend ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a lot of excellent stuff here, but perhaps changes might be best done on #24532..
def macros_substitution(self) -> None: | ||
self.substitutions["macros"] = "" | ||
if self.define_macros: | ||
self.substitutions["macros"] = ",".join(f"\'-D{macro[0]}={macro[1]}\'" if macro[1] else f"-D{macro[0]}" for macro in self.define_macros) | ||
if self.undef_macros: | ||
self.substitutions["macros"] += "," + ",".join(f"\'-U{macro}\'" for macro in self.undef_macros) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These don't pass -D
flags to the underlying preprocessor, that would have to be via Dc_args=-D{macro[0]}={macro[1]}
... (e.g. seen here)
Similar issue with unsetting the defines.
def global_flags_substitution(self) -> None: | ||
fortran_compiler_flags = self.fortran_flags + self.optimization_flags + self.architecture_flags | ||
c_compiler_flags = self.optimization_flags + self.architecture_flags | ||
self.substitutions["fortran_global_args"] = fortran_compiler_flags | ||
self.substitutions["c_global_args"] = c_compiler_flags |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Global flags are generally a bad idea, especially since a lot of these might be used as subprojects, it makes more sense to populate the _project
variants.
self.substitutions['numpy_get_include'] = self.numpy_get_include.absolute() | ||
self.substitutions['f2py_get_include'] = self.f2py_get_include.absolute() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these necessary? The documentation includes the option to get these dynamically from meson
...
def linker_substitution(self) -> None: | ||
self.substitutions["linker_args"] = "" | ||
if self.linker_libpath: | ||
linker_libpath_subs = ",".join(f"-L{libpath}" for libpath in self.linker_libpath) | ||
self.substitutions["linker_args"] += linker_libpath_subs | ||
if self.linker_libname: | ||
linker_libname_subs = ",".join(f"-l{libname}" for libname in self.linker_libname) | ||
self.substitutions["linker_args"] += f",{linker_libname_subs}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same issue as above.
Works with #24532 , thanks! Traceback (most recent call last):
File "/home/jr/.virtualenvs/numpy-test/bin/f2py", line 33, in <module>
sys.exit(load_entry_point('numpy==2.0.0.dev0', 'console_scripts', 'f2py')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 732, in main
run_compile()
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 705, in run_compile
builder.compile()
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 130, in compile
self.write_meson_build(self.build_dir)
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 108, in write_meson_build
src = meson_template.generate_meson_build()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 71, in generate_meson_build
template = Template(self.meson_build_template())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 44, in meson_build_template
raise FileNotFoundError(
FileNotFoundError: [Errno 2] Meson build template /home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/meson.build.template does not exist. Looking in the site-packages directory, I noticed that the file |
Should address numpy#22225 (comment)
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
Should address numpy#22225 (comment)
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from gh-22225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * ENH: Add meson skeleton from gh-2225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address #22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from numpygh-22225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * ENH: Add meson skeleton from numpygh-2225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address numpy#22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from gh-22225 * ENH: Add meson skeleton from gh-2225 * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address #22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Rohit Goswami <rgoswami@quansight.com> Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
This is updated in #24532, so closing this. |
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from numpygh-22225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * ENH: Add meson skeleton from numpygh-2225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address numpy#22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
This PR is a continuation of my older PR for implementing F2PY's fronted with
argparse
.My blog explains the core ideas behind the changes.
This PR introduces a new flag for F2PY -
--backend
which can accept build system for building Python extension modules.An example to produce a Python module out of the classic fibonacci sequence Fortran program will be:-
The output thus produced works correctly:
The default value of
--backend
flag isdistutils
making this feature completely backwards compatible.