Skip to content

Circular import issue involving getlimits and NPY_DT_DBG_TRACING #12063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ahaldane opened this issue Sep 30, 2018 · 5 comments
Closed

Circular import issue involving getlimits and NPY_DT_DBG_TRACING #12063

ahaldane opened this issue Sep 30, 2018 · 5 comments
Assignees
@eric-wieser

Comments

@ahaldane
Copy link
Member

PR #10602 seems to have triggered a circular import problem when using the dtype-transer debug mode. (set NPY_DT_DBG_TRACING to 1 at the top of dtype_transfer.c).

The problem seems to be that debug tracing tries to print a dtype before the dtype object has been created, I'm still trying to see what happened.

It involves getlimits.py which has been the source of circular import issues in the past.

>>> import numpy as np
Calculating dtype transfer from  to 
Traceback (most recent call last):
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/_dtype.py", line 23, in __repr__
    arg_str = _construction_repr(dtype, include_align=False)
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/_dtype.py", line 77, in _construction_repr
    return _scalar_str(dtype, short=short)
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/_dtype.py", line 81, in _scalar_str
    byteorder = _byte_order_str(dtype)
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/_dtype.py", line 151, in _byte_order_str
    swapped = np.dtype(int).newbyteorder('s')
AttributeError: module 'numpy' has no attribute 'dtype'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 200, in _lock_unlock_module
  File "<frozen importlib._bootstrap>", line 163, in _get_module_lock
SystemError: <built-in function acquire_lock> returned a result with an error set

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/__init__.py", line 72, in <module>
    from . import getlimits
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/getlimits.py", line 125, in <module>
    tiny=_f16(2 ** -14))
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/getlimits.py", line 75, in __init__
    self.epsilon = self.eps = float_to_float(kwargs.pop('eps'))
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/getlimits.py", line 70, in <lambda>
    float_to_float = lambda v : _fr1(float_conv(v))
  File "/home/allan/nmpy_dev/build/testenv/lib/python3.7/site-packages/numpy/core/getlimits.py", line 29, in _fr1
    a = a.copy()
SystemError: <method 'copy' of 'numpy.ndarray' objects> returned a result with an error set
@ahaldane ahaldane mentioned this issue Sep 30, 2018
Merged
ahaldane added a commit to ahaldane/numpy that referenced this issue Oct 10, 2018
@ahaldane
MAINT: delay initialization of getlimits (circular imports)
78e1051 
getlimits previously called numpy code before all of numpy was loaded,
which often causes circular import issues. This delays getlimits init.

Fixes numpy#12063
@eric-wieser
Copy link
Member

The easy fix here is to not have _dtype use np.dtype - it only does so because it wants to obtain a > or < character for the endianness.

@eric-wieser
Copy link
Member

Those SystemErrors can be fixed by checking return values in the code generating the message "Calculating dtype transfer ..."

@eric-wieser
Copy link
Member

eric-wieser commented Oct 26, 2018 •
edited
Loading

Perhaps switching to np.core.dtype is sufficient

Nope. I'm starting to question my understanding of how relative imports work.

@eric-wieser eric-wieser mentioned this issue Oct 26, 2018
Merged
@eric-wieser eric-wieser self-assigned this Oct 26, 2018
eric-wieser added a commit to eric-wieser/numpy that referenced this issue Oct 31, 2018
@eric-wieser
BUG: Avoid SystemErrors by checking the return value of PyPrint
9751469 
Relates to numpygh-12063
@ahaldane
Copy link
Member Author

The easy fix here is to not have _dtype use np.dtype - it only does so because it wants to obtain a > or < character for the endianness.

I think that's only a temporary fix, it doesn't fix the deeper issue:

I was able to fix this particular NPY_DT_DBG_TRACING issue by replacing the numpy import in _dtype.py with from . import numeric as sb as a number of other files in core do. I also needed to move the location of _kind_name to numerictypes.py to avoid circular imports.

But it's not just _dtype.py which imports numpy, multiarray.py, arrayprint.py, memmap.py and others do too. These will have the same problem when getlimits calls numpy code: Exactly the same problem will arise if, for debugging purposes, I try to add calls to PyObject_Print in any code-path in multiarray which is called by getlimits (which is many), which calls one of those modules. In fact I've had this problem when debugging recently.

It does seem slightly incorrect that so many core modules import the parent numpy module. But it seems like a lot of work to undo that.

It also seems incorrect that getlimits.py is calling numpy code before numpy is loaded. The fix for that in #12064, while a bit ugly, is pretty easy.

@eric-wieser
Copy link
Member

It does seem slightly incorrect that so many core modules import the parent numpy module

I think part of the problem is that the internal structure of np.core is largely historic, and fixed for compatibility. The modules function are in often arbitrary, and so it's easier to just import directly from np.core than it is to work out which module to import from (especially in the face of circular imports)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eric-wieser eric-wieser

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eric-wieser @ahaldane