Failure to load .npy file with structured dtype and metadata #14142

colingavin · 2019-07-28T18:43:12Z

The dtype description in an NPY file can be written in a way that prevents the file from being read. The attached array in bad_array.pkl.zip exhibits this problem. In this case the written dtype is [('id', '|V16'), ('has_fit', '|b1'), ('timeax', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('profile', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('area', '<f8')].

(Note that it was obtained from via h5py - if this is an issue with the format of the arrays produced by that library, I'm happy to file a bug there instead.)

Reproducing code example:

import numpy as np
import pickle
import io

with open('bad_array.pkl', 'rb') as f:
    arr = pickle.load(f)

buf = io.BytesIO()
np.save(buf, arr)
buf.seek(0)
np.load(buf)

Error message:

Traceback (most recent call last):
  File "scripts/badnpy.py", line 11, in <module>
    np.load(buf)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/npyio.py", line 453, in load
    pickle_kwargs=pickle_kwargs)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 712, in read_array
    shape, fortran_order, dtype = _read_array_header(fp, version)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 578, in _read_array_header
    d = safe_eval(header)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/utils.py", line 1139, in safe_eval
    return ast.literal_eval(source)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
    map(_convert, node.values)))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 74, in _convert
    return list(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
    return tuple(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
    return tuple(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
    map(_convert, node.values)))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Call object at 0x10b965a58>

Numpy/Python version information:

1.17.0 3.7.2 (default, Jul 14 2019, 10:22:38)
[Clang 9.0.0 (clang-900.0.39.2)]

The text was updated successfully, but these errors were encountered:

eric-wieser · 2019-07-28T19:00:02Z

Loads just fine on 1.15.4 + 64-bit windows, looks like a regression.

Also loads fine on 1.16.0.dev0+d14632c

seberg · 2019-07-28T19:43:10Z

1.17.x and 1.16.2 I am seeing a No error set when coercing the dtype.

eric-wieser · 2019-07-28T19:43:14Z

No issues on 1.18.0.dev0+84e8412 + python 3.5 either. Perhaps only the release has the problem, or perhaps it's a 3.7 issue

seberg · 2019-07-28T19:44:41Z

Oh, hmmm, maybe then it is a python version related thing, interesting. I was testing with 3.7:

np.dtype([('id', '|V16'), ('has_fit', '|b1'), ('timeax', ('|O', {'vlen': np.dtype('float64')})), ('', '|V8'), ('profile', ('|O', {'vlen': np.dtype('float64')})), ('', '|V8'), ('area', '<f8')])

resulting in no error set returns.

eric-wieser · 2019-07-28T19:46:02Z

On 3.5 and 84e8412, that gives ValueError: invalid shape in fixed-type tuple. For some reason the pickle does not cause that error

seberg · 2019-07-28T19:49:13Z

Oh, interesting, on python 2.7, I get a simple invalid datatype error, on 3.7 I get "no error set", I had not tested the pickle, I can reproduce the error on python 3.7 both with 16.2 and master.

eric-wieser · 2019-07-28T21:33:04Z

My hunch is the cause is this line:

numpy/numpy/core/src/multiarray/descriptor.c

Line 501 in edf8a5f

PyObject_Print(PyTuple_GET_ITEM(item, 1), stderr, 0);

which calls print while an exception is in flight.

I have no idea why that print is there

eric-wieser · 2019-07-28T21:33:59Z

Yep, that fixed it

…o an exception being in flight We shouldn't be reporting errors via print anyway Related to numpygh-14142

seberg · 2019-07-29T00:06:12Z

Ah, nice, thanks. Guess I can close this then.

eric-wieser · 2019-07-29T00:08:36Z

Don't think we can close this, the problem I solved is the one you discovered, not the one this issue reports

seberg · 2019-07-29T00:09:25Z

Ohh, sorry. Somewhat thought those were the same, heh...

…o an exception being in flight We shouldn't be reporting errors via print anyway Related to numpygh-14142

charris · 2019-08-20T18:00:42Z

Pushing off to 1.17.2

charris · 2021-11-16T00:59:36Z

Well, heck. Pushed it off again.

seberg · 2022-05-04T16:32:18Z

Will push this one of (as discussed also in the triage meeting), but I am not sure how important this issue is still/now, so please ping to bump it if necessary.

charris · 2024-08-01T15:06:33Z

@tacaswell Is this still a problem?

tacaswell · 2024-08-01T18:01:10Z

Yes, but now with an extra warning:

In [2]: import numpy as np
   ...: import io
   ...:
   ...:
   ...: dt = np.dtype({'names': ['a', 'b'], 'formats':  [float, np.dtype('S3', metadata={'some': 'stuff'})]})
   ...:
   ...: arr = np.array([(1, 'abc'), (2, 'def')], dtype=dt)
   ...: buf = io.BytesIO()
   ...: np.save(buf, arr)
   ...: buf.seek(0)
   ...: np.load(buf)
/home/tcaswell/.virtualenvs/cp313/lib/python3.13/site-packages/numpy/lib/format.py:380: UserWarning: metadata on a dtype is not saved to an npy/npz. Use another format (such as pickle) to store it.
  d['descr'] = dtype_to_descr(array.dtype)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 11
      9 np.save(buf, arr)
     10 buf.seek(0)
---> 11 np.load(buf)

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/numpy/lib/_npyio_impl.py:488, in load(file, mmap_mode, allow_pickle, fix_imports, encoding, max_header_size)
    485         return format.open_memmap(file, mode=mmap_mode,
    486                                   max_header_size=max_header_size)
    487     else:
--> 488         return format.read_array(fid, allow_pickle=allow_pickle,
    489                                  pickle_kwargs=pickle_kwargs,
    490                                  max_header_size=max_header_size)
    491 else:
    492     # Try a pickle
    493     if not allow_pickle:

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/numpy/lib/format.py:809, in read_array(fp, allow_pickle, pickle_kwargs, max_header_size)
    807 version = read_magic(fp)
    808 _check_version(version)
--> 809 shape, fortran_order, dtype = _read_array_header(
    810         fp, version, max_header_size=max_header_size)
    811 if len(shape) == 0:
    812     count = 1

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/numpy/lib/format.py:678, in _read_array_header(fp, version, max_header_size)
    676     raise ValueError(msg.format(d['fortran_order']))
    677 try:
--> 678     dtype = descr_to_dtype(d['descr'])
    679 except TypeError as e:
    680     msg = "descr is not a valid dtype descriptor: {!r}"

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/numpy/lib/format.py:337, in descr_to_dtype(descr)
    335 if len(field) == 2:
    336     name, descr_str = field
--> 337     dt = descr_to_dtype(descr_str)
    338 else:
    339     name, descr_str, shape = field

File ~/.virtualenvs/cp313/lib/python3.13/site-packages/numpy/lib/format.py:327, in descr_to_dtype(descr)
    324 elif isinstance(descr, tuple):
    325     # subtype, will always have a shape descr[1]
    326     dt = descr_to_dtype(descr[0])
--> 327     return numpy.dtype((dt, descr[1]))
    329 titles = []
    330 names = []

ValueError: invalid shape in fixed-type tuple.

In [3]: np.__version__
Out[3]: '2.1.0.dev0+git20240725.0819378'

seberg · 2024-08-01T18:05:04Z

Hmmmm, I somehow thought I once wrote stripping logic, so that it would drop but load. I guess some half-finished PR somewhere.

seberg · 2024-08-01T18:06:42Z

Yeah, #23371 should cause all metdata to be dropped, so it isn't clear to me why it cannot be loaded. It would seem that there must be some whole in the logic.

seberg · 2024-08-08T18:48:29Z

Haha, the change was fine... The problem is that the code just failed to actually use the cleaned up version...

We had logic in place to drop (most) metadata, but the change had a small bug: During saving, we were still using the one with metadata... Maybe doesn't quite close it, but big enough of an improvement for now, I think, so Closes numpygh-14142

eric-wieser changed the title ~~Failure to load .npy file with complex dtype~~ Failure to load .npy file with structured dtype Jul 28, 2019

seberg added 00 - Bug 06 - Regression component: numpy.dtype labels Jul 28, 2019

charris added the 09 - Backport-Candidate PRs tagged should be backported label Jul 28, 2019

charris modified the milestones: 1.17.1 release, 1.16.5 release Jul 28, 2019

eric-wieser added a commit to eric-wieser/numpy that referenced this issue Jul 28, 2019

BUG: Remove stray print that causes a SystemError on python 3.7 due t…

75016bd

…o an exception being in flight We shouldn't be reporting errors via print anyway Related to numpygh-14142

eric-wieser mentioned this issue Jul 28, 2019

BUG: Remove stray print that causes a SystemError on python 3.7 . #14144

Merged

seberg closed this as completed Jul 29, 2019

eric-wieser reopened this Jul 29, 2019

charris pushed a commit to charris/numpy that referenced this issue Jul 29, 2019

BUG: Remove stray print that causes a SystemError on python 3.7 due t…

38b895f

…o an exception being in flight We shouldn't be reporting errors via print anyway Related to numpygh-14142

charris mentioned this issue Jul 29, 2019

BUG: Remove stray print that causes a SystemError on python 3.7. #14158

Merged

charris pushed a commit to charris/numpy that referenced this issue Jul 30, 2019

BUG: Remove stray print that causes a SystemError on python 3.7 due t…

bac5c2c

…o an exception being in flight We shouldn't be reporting errors via print anyway Related to numpygh-14142

charris mentioned this issue Jul 30, 2019

BUG: Remove stray print that causes a SystemError on python 3.7 #14162

Merged

charris modified the milestones: 1.22.0 release, 1.23.0 release Nov 16, 2021

seberg modified the milestones: 1.23.0 release, 1.24.0 release May 4, 2022

charris modified the milestones: 1.24.0 release, 1.25.0 release Nov 19, 2022

seberg mentioned this issue Feb 7, 2023

BUG: unable to read .npy file with non-eval'able metadata #23169

Closed

charris modified the milestones: 1.25.0 release, 2.0.0 release May 24, 2023

seberg modified the milestones: 2.0.0 release, 2.1.0 release Jan 31, 2024

charris modified the milestones: 2.1.0 release, 2.2.0 release Aug 1, 2024

seberg mentioned this issue Aug 8, 2024

BUG: Do not accidentally store dtype metadata in np.save #27143

Merged

charris closed this as completed in #27143 Aug 8, 2024

charris closed this as completed in b9bcca0 Aug 8, 2024

charris mentioned this issue Aug 8, 2024

BUG: Do not accidentally store dtype metadata in np.save #27149

Merged

charris mentioned this issue Aug 9, 2024

BUG: Do not accidentally store dtype metadata in np.save #27151

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failure to load .npy file with structured dtype and metadata #14142

Failure to load .npy file with structured dtype and metadata #14142

colingavin commented Jul 28, 2019 •

edited

Loading

eric-wieser commented Jul 28, 2019 •

edited

Loading

seberg commented Jul 28, 2019

eric-wieser commented Jul 28, 2019

seberg commented Jul 28, 2019

eric-wieser commented Jul 28, 2019 •

edited

Loading

seberg commented Jul 28, 2019

eric-wieser commented Jul 28, 2019 •

edited

Loading

eric-wieser commented Jul 28, 2019

seberg commented Jul 29, 2019

eric-wieser commented Jul 29, 2019

seberg commented Jul 29, 2019

charris commented Aug 20, 2019

charris commented Nov 16, 2021

seberg commented May 4, 2022

charris commented Aug 1, 2024

tacaswell commented Aug 1, 2024

seberg commented Aug 1, 2024

seberg commented Aug 1, 2024

seberg commented Aug 8, 2024

Failure to load .npy file with structured dtype and metadata #14142

Failure to load .npy file with structured dtype and metadata #14142

Comments

colingavin commented Jul 28, 2019 • edited Loading

Reproducing code example:

Error message:

Numpy/Python version information:

eric-wieser commented Jul 28, 2019 • edited Loading

seberg commented Jul 28, 2019

eric-wieser commented Jul 28, 2019

seberg commented Jul 28, 2019

eric-wieser commented Jul 28, 2019 • edited Loading

seberg commented Jul 28, 2019

eric-wieser commented Jul 28, 2019 • edited Loading

eric-wieser commented Jul 28, 2019

seberg commented Jul 29, 2019

eric-wieser commented Jul 29, 2019

seberg commented Jul 29, 2019

charris commented Aug 20, 2019

charris commented Nov 16, 2021

seberg commented May 4, 2022

charris commented Aug 1, 2024

tacaswell commented Aug 1, 2024

seberg commented Aug 1, 2024

seberg commented Aug 1, 2024

seberg commented Aug 8, 2024

colingavin commented Jul 28, 2019 •

edited

Loading

eric-wieser commented Jul 28, 2019 •

edited

Loading

eric-wieser commented Jul 28, 2019 •

edited

Loading

eric-wieser commented Jul 28, 2019 •

edited

Loading