Skip to content

BUG: f2py generated signature file and automatic arrays #21892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
benbovy opened this issue Jun 30, 2022 · 7 comments
Closed

BUG: f2py generated signature file and automatic arrays #21892

benbovy opened this issue Jun 30, 2022 · 7 comments
Assignees
@HaoZeke
Labels
00 - Bug 06 - Regression component: numpy.f2py

Comments

@benbovy
Copy link

benbovy commented Jun 30, 2022

Describe the issue:

Let's consider the following subroutine in a test.f90 file that declares an automatic array from two variables(*):

subroutine test (nx, ny, arr)

  implicit none

  integer, intent(in) :: nx
  integer, intent(in) :: ny
  integer, dimension(nx*ny) :: arr

return
end subroutine test

It used to work with numpy 1.22.0 or earlier, despite the Warning: cross-dependence between variables "ny" and "nx" and the weird shape checks as shown in the generated test.pyf signature file below:

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module test ! in 
    interface  ! in :test
        subroutine test(nx,ny,arr) ! in :test:src/test.f90
            integer, optional,intent(in),check(shape(arr, 0) == nx * ny),depend(ny,arr) :: nx=shape(arr, 0) / nx
            integer, optional,intent(in),check(shape(arr, 0) == nx * ny),depend(nx,arr) :: ny=shape(arr, 0) / nx
            integer dimension(nx * ny) :: arr
        end subroutine test
    end interface 
end python module test

! This file was auto-generated with f2py (version:1.22.0).
! See:
! https://web.archive.org/web/20140822061353/http://cens.ioc.ee/projects/f2py2e

However, with numpy 1.22.1 or later, it generates a test.pyf with "swapped" nx / ny:

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module test ! in 
    interface  ! in :test
        subroutine test(nx,ny,arr) ! in :test:src/test.f90
            integer, optional,intent(in),check(shape(arr, 0) == nx * ny),depend(ny,arr) :: nx=shape(arr, 0) / ny
            integer, optional,intent(in),check(shape(arr, 0) == nx * ny),depend(arr,nx) :: ny=shape(arr, 0) / nx
            integer dimension(nx * ny) :: arr
        end subroutine test
    end interface 
end python module test

! This file was auto-generated with f2py (version:1.22.1).
! See:
! https://web.archive.org/web/20140822061353/http://cens.ioc.ee/projects/f2py2e

Which causes the following recursion error when trying to generate the Python extension from the signature file:

Reading fortran codes...
        Reading file 'test.pyf' (format:free)
Post-processing...
        Block: test
                        Block: test
Traceback (most recent call last):
  File "/Users/me/env/bin/f2py", line 8, in <module>
    sys.exit(main())
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/f2py2e.py", line 693, in main
    run_main(sys.argv[1:])
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/f2py2e.py", line 430, in run_main
    postlist = callcrackfortran(files, options)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/f2py2e.py", line 333, in callcrackfortran
    postlist = crackfortran.crackfortran(files)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 3209, in crackfortran
    postlist = postcrack(grouplist[0])
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 1926, in postcrack
    g = postcrack(g, tab=tab + '\t')
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 1945, in postcrack
    block['body'] = analyzebody(block, args, tab=tab)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2107, in analyzebody
    b = postcrack(b, as_, tab=tab + '\t')
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 1945, in postcrack
    block['body'] = analyzebody(block, args, tab=tab)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2107, in analyzebody
    b = postcrack(b, as_, tab=tab + '\t')
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 1941, in postcrack
    block['vars'] = analyzevars(block)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2458, in analyzevars
    params = get_parameters(vars, get_useparameters(block))
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2329, in get_parameters
    for n in get_sorted_names(vars):
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2257, in get_sorted_names
    depend_dict = _calc_depend_dict(vars)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2250, in _calc_depend_dict
    _get_depend_dict(n, vars, depend_dict)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2236, in _get_depend_dict
    or _get_depend_dict(word, vars, deps):
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2236, in _get_depend_dict
    or _get_depend_dict(word, vars, deps):
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2236, in _get_depend_dict
    or _get_depend_dict(word, vars, deps):
  [Previous line repeated 979 more times]
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/crackfortran.py", line 2228, in _get_depend_dict
    if '=' in vars[name] and not isstring(vars[name]):
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/auxfuncs.py", line 77, in isstring
    return _isstring(var) and not isarray(var)
  File "/Users/me/env/lib/python3.9/site-packages/numpy/f2py/auxfuncs.py", line 72, in _isstring
    return 'typespec' in var and var['typespec'] == 'character' and \
RecursionError: maximum recursion depth exceeded in comparison

The difference between the two generated signature files seems to be a result of #20721.

(*) Do you think the issue here could be considered as a regression? Or is the example here wrong and we should not declare dimensions of automatic arrays from two or more variables? Not sure about this as I'm not a Fortran expert and haven't found much info about this practice, although I'd be surprised that compilers let us do it if it wasn't valid.

Reproduce the code example:

# see f90 / pyf source above

Error message:

No response

NumPy/Python version information:

Numpy 1.22.0 / 1.22.1
Python 3.9
@HaoZeke
Copy link
Member

HaoZeke commented Jun 30, 2022 •
edited
Loading

Thanks for the detailed issue and for tracking down a potential source @benbovy!

The usage is valid, expressions of nx,ny can be used to determine the size of arr. This is definitely a regression which needs to be fixed.

EDIT: Details on automatic shape arrays, since nx*ny is stored to a temporary variable and so has local scope

@HaoZeke HaoZeke self-assigned this Jun 30, 2022
@HaoZeke
Copy link
Member

HaoZeke commented Jun 30, 2022

FWIW with master 1.24.0.dev0+434.ga111b9d07, the generated pyf does compile correctly:

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

subroutine test(nx,ny,arr) ! in test.f90
    integer, required,intent(in) :: nx
    integer, required,intent(in) :: ny
    integer dimension(nx * ny),check(shape(arr, 0) == nx * ny,shape(arr, 0) == nx * ny),depend(ny,nx) :: arr
end subroutine test

! This file was auto-generated with f2py (version:1.24.0.dev0+434.ga111b9d07).
! See:
! https://web.archive.org/web/20140822061353/http://cens.ioc.ee/projects/f2py2e

Which also compiles and works. Closing as fixed in 1.24 but please re-open if not!

@HaoZeke HaoZeke closed this as completed Jun 30, 2022
@HaoZeke HaoZeke mentioned this issue Jun 30, 2022
Closed
@benbovy
Copy link
Author

benbovy commented Jun 30, 2022

👍 Great thanks!

@charris
Copy link
Member

charris commented Jun 30, 2022

@HaoZeke Is this is fixed in 1.23 also?

@HaoZeke
Copy link
Member

HaoZeke commented Jul 1, 2022

@HaoZeke Is this is fixed in 1.23 also?

Unfortunately not, in 1.23 we generate:

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

subroutine test(nx,ny,arr) ! in test.f90
    integer, optional,intent(in),check(shape(arr, 0) == nx * ny),depend(arr,ny) :: nx=shape(arr, 0) / nx
    integer, optional,intent(in),check(shape(arr, 0) == nx * ny),depend(nx,arr) :: ny=shape(arr, 0) / nx
    integer dimension(nx * ny) :: arr
end subroutine test

! This file was auto-generated with f2py (version:1.23.0.dev0).
! See:
! https://web.archive.org/web/20140822061353/http://cens.ioc.ee/projects/f2py2e

Which compiles but will error out on usage.

In [1]: import numpy as np

In [2]: import untitled as ut

In [3]: x = np.ones(12)

In [4]: ut.test(3,4,x)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 ut.test(3,4,x)

error: (shape(arr, 0) == nx * ny) failed for 2nd keyword ny: test:ny=1

@charris
Copy link
Member

charris commented Jul 1, 2022

Unfortunately not, in 1.23 we generate:

Hmm. There have been major changes in f2py since 1.23.x. Is there a simple fix that could be backported? I suppose we could just copy the whole f2py directory back, but I'd be uncomfortable doing that.

@charris
Copy link
Member

charris commented Jul 1, 2022

Here is a list of the 21 changed files:

	modified:   numpy/f2py/auxfuncs.py
	modified:   numpy/f2py/capi_maps.py
	modified:   numpy/f2py/cb_rules.py
	modified:   numpy/f2py/cfuncs.py
	modified:   numpy/f2py/common_rules.py
	modified:   numpy/f2py/crackfortran.py
	modified:   numpy/f2py/f90mod_rules.py
	modified:   numpy/f2py/func2subr.py
	modified:   numpy/f2py/rules.py
	modified:   numpy/f2py/src/fortranobject.c
	modified:   numpy/f2py/src/fortranobject.h
	modified:   numpy/f2py/tests/src/array_from_pyobj/wrapmodule.c
	new file:   numpy/f2py/tests/src/crackfortran/pubprivmod.f90
	modified:   numpy/f2py/tests/test_array_from_pyobj.py
	modified:   numpy/f2py/tests/test_callback.py
	new file:   numpy/f2py/tests/test_character.py
	modified:   numpy/f2py/tests/test_crackfortran.py
	new file:   numpy/f2py/tests/test_docs.py
	modified:   numpy/f2py/tests/test_f2py2e.py
	modified:   numpy/f2py/tests/test_return_character.py
	modified:   numpy/f2py/tests/util.py

@boegel boegel mentioned this issue Sep 8, 2022
Closed
13 tasks
@JulienPeloton JulienPeloton mentioned this issue Jan 2, 2023
Open
benbovy added a commit to regro-cf-autotick-bot/fastscapelib-f2py-feedstock that referenced this issue Sep 28, 2023
@benbovy
numpy version constraints
7c2ff93 
See numpy/numpy#21892
benbovy added a commit to conda-forge/fastscapelib-f2py-feedstock that referenced this issue Sep 29, 2023
@regro-cf-autotick-bot @benbovy
Bump v2.8.4 and rebuild for python312 (#23)
0483adb 
* Rebuild for python312

* MNT: Re-rendered with conda-build 3.26.1, conda-smithy 3.26.2, and conda-forge-pinning 2023.09.26.15.03.47

* numpy version constraints

See numpy/numpy#21892

* try numpy >=1.24 for build only

* try different numpy pinning

* fix numpy pinning?

* bump version v2.8.4

---------

Co-authored-by: Benoit Bovy <benbovy@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@HaoZeke HaoZeke

Labels
00 - Bug 06 - Regression component: numpy.f2py
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@charris @benbovy @HaoZeke