Skip to content

ENH: F2PY build output determinism #21187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Apr 29, 2022
Merged

Conversation

HaoZeke
Copy link
Member

@HaoZeke HaoZeke commented Mar 12, 2022

Closes #20385 and adds documentation for usage to fix rgommers/scipy#22 (comment).

Essentially:

  • Changes the default behavior of F2PY to ensure possibly empty files are generated for build determinism
  • Adds an optional --no-empty-gen flag to restore earlier behavior
  • Documents its usage

A few things which are out of scope for this PR:

Might need a release note? (as a new CLI arugment) done

See also #21187 (comment)

@HaoZeke HaoZeke requested a review from melissawm March 12, 2022 23:00
@charris charris changed the title F2PY build output determinism ENH: F2PY build output determinism Mar 13, 2022
@charris
Copy link
Member

charris commented Mar 13, 2022

Needs a release note.

Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to generate both -f2pywrappers.f and -f2pywrappers.f90? An alternative seems to be to let users specify whether the code is F77 or F90; either way one needs a single flag when invoking f2py at the moment. I think that's still fine for determinism - the problem that was causing trouble was more "output file generated depends on details of the code". Or are there cases where f2py switches modes between F77 and F90 based on contents of a file, rather than file extension?

Other comment: why not make this the default straight away? It's fully backwards-compatible.

@HaoZeke
Copy link
Member Author

HaoZeke commented Mar 14, 2022

Do we need to generate both -f2pywrappers.f and -f2pywrappers.f90? An alternative seems to be to let users specify whether the code is F77 or F90; either way one needs a single flag when invoking f2py at the moment. I think that's still fine for determinism - the problem that was causing trouble was more "output file generated depends on details of the code". Or are there cases where f2py switches modes between F77 and F90 based on contents of a file, rather than file extension?

The .f file and .f90 files are created for different use-cases. Since F90 is a super-set of F77, it is possible for a code to trigger the generation of both the .f90 wrapper for F90 features (e.g. module) and also .f for F77 features (e.g. common).

While it isn't a good idea to mix modules and common blocks, since it is allowed by the Fortran standard (but not recommended) we can't really disallow it.

TL:DR; For F77 code the .f90 wrapper is never generated, but for F90 code, there is no way to be sure.

Other comment: why not make this the default straight away? It's fully backwards-compatible.

It is backwards-compatible, but might be confusing/annoying for downstream projects if updating causes the sudden generation of empty files. I would like it to be the default eventually though.

@rgommers
Copy link
Member

While it isn't a good idea to mix modules and common blocks, since it is allowed by the Fortran standard (but not recommended) we can't really disallow it.

Thanks for the explanation. That's too bad for F90 code.

It seems like for F77 though, the .f90 wrapper is never needed? I'm asking again because this deterministic mode doesn't seem to be optimal for SciPy - we now never get the -f2pywrappers2.f90, and we switch to always. They're both deterministic for F77 code (as far as I can tell), it'll just be a bit noisier after this PR. I'm not sure how reliable file extensions are, but if there is to be a CLI flag, could it be something like --always-f77wrappers/--always-f2pywrappers?

It is backwards-compatible, but might be confusing/annoying for downstream projects if updating causes the sudden generation of empty files.

Whenever it does switch the effect is the same. I suggest deciding on the final config, and either switching it now, or never. If it's now, you then don't need the command-line flag.

@HaoZeke
Copy link
Member Author

HaoZeke commented Mar 14, 2022

While it isn't a good idea to mix modules and common blocks, since it is allowed by the Fortran standard (but not recommended) we can't really disallow it.

Thanks for the explanation. That's too bad for F90 code.

It seems like for F77 though, the .f90 wrapper is never needed? I'm asking again because this deterministic mode doesn't seem to be optimal for SciPy - we now never get the -f2pywrappers2.f90, and we switch to always. They're both deterministic for F77 code (as far as I can tell), it'll just be a bit noisier after this PR. I'm not sure how reliable file extensions are, but if there is to be a CLI flag, could it be something like --always-f77wrappers/--always-f2pywrappers?

Thinking about this a little bit more then I think I'll rework this to just generate -f2pywrappers.f if the file is parsed as Fortran 77 (F2PY determines this based on more than just the extension, which is good since the filename extensions are not standardized) and to generate both *wrappers* if not.

That would probably solve this better than having a flag.

It is backwards-compatible, but might be confusing/annoying for downstream projects if updating causes the sudden generation of empty files.

Whenever it does switch the effect is the same. I suggest deciding on the final config, and either switching it now, or never. If it's now, you then don't need the command-line flag.

The only reason to have a flag is to have the option to revert to older behavior (irrespective of the default we decide on).

@HaoZeke
Copy link
Member Author

HaoZeke commented Mar 18, 2022

Discussed briefly at an F2PY meeting. The cleanest fix here is to only generate only one f2pywrappers.{ext} file with the extension changing based on the inputs.

EDIT: This is a larger refactor, and so has been moved to an issue (#21224)

@HaoZeke
Copy link
Member Author

HaoZeke commented Mar 21, 2022

The current behaviour is to generate both wrapper files for F90 sources, and only an F77 wrapper for F77 inputs with --empty-gen.

EDIT: A FutureWarning is raised when --empty-gen is not passed.

@HaoZeke HaoZeke requested a review from rgommers March 21, 2022 03:08
@rgommers
Copy link
Member

Discussed briefly at an F2PY meeting. The cleanest fix here is to only generate only one f2pywrappers.{ext} file with the extension changing based on the inputs.

EDIT: This is a larger refactor, and so has been moved to an issue (#21224)

It's not clear from the issue why this is a larger refactor, maybe expand the description a bit?

Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there, thanks @HaoZeke

Whenever it does switch the effect is the same. I suggest deciding on the final config, and either switching it now, or never. If it's now, you then don't need the command-line flag.

The only reason to have a flag is to have the option to revert to older behavior (irrespective of the default we decide on).

My comment still applies though. I'm not sure the flag is needed at all, but if you (and @pearu, @melissawm) want to have it, then I suggest to just pick a default and stick with it. There's no point switching the default later.

@HaoZeke
Copy link
Member Author

HaoZeke commented Mar 21, 2022

Discussed briefly at an F2PY meeting. The cleanest fix here is to only generate only one f2pywrappers.{ext} file with the extension changing based on the inputs.
EDIT: This is a larger refactor, and so has been moved to an issue (#21224)

It's not clear from the issue why this is a larger refactor, maybe expand the description a bit?

Mostly because of implementation details:

  • No tests for current behavior

Also, it isn't actually clear this needs to be done in the first place, given that mixing F77 formatted code (which is what COMMON blocks generate) and F90 formatted code (generated by modules) might confuse compilers (which may have a different parser for F77 and F90).

To work around this F2PY could generate free-form wrappers to COMMON blocks when the inputs are F90 but IMO the current wrappers are more readable (-f2pywrapper.f --> common block, -f2pywrapper2.f90 --> module / derived type).

Also remove FutureWarning and update documentation

Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
@HaoZeke HaoZeke requested a review from rgommers March 21, 2022 11:49
Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now, thanks @HaoZeke.

I'll leave this open for @pearu or @melissawm to have a look at as well.

@charris charris closed this Apr 6, 2022
@charris charris reopened this Apr 6, 2022
@mattip
Copy link
Member

mattip commented Apr 18, 2022

@pearu or @melissawm does this look OK?

Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Pearu Peterson <pearu.peterson@gmail.com>
Co-authored-by: Melissa Weber Mendonça <melissawm@gmail.com>
@HaoZeke HaoZeke merged commit cec19d7 into numpy:main Apr 29, 2022
@HaoZeke HaoZeke deleted the f2pyDeterminism branch April 29, 2022 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Deterministic F2PY output generation Tracking issue for Meson support of submodules
4 participants