Skip to content

Failure to load .npy file with structured dtype and metadata #14142

Closed
@colingavin

Description

@colingavin

The dtype description in an NPY file can be written in a way that prevents the file from being read. The attached array in bad_array.pkl.zip exhibits this problem. In this case the written dtype is [('id', '|V16'), ('has_fit', '|b1'), ('timeax', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('profile', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('area', '<f8')].

(Note that it was obtained from via h5py - if this is an issue with the format of the arrays produced by that library, I'm happy to file a bug there instead.)

Reproducing code example:

import numpy as np
import pickle
import io

with open('bad_array.pkl', 'rb') as f:
    arr = pickle.load(f)

buf = io.BytesIO()
np.save(buf, arr)
buf.seek(0)
np.load(buf)

Error message:

Traceback (most recent call last):
  File "scripts/badnpy.py", line 11, in <module>
    np.load(buf)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/npyio.py", line 453, in load
    pickle_kwargs=pickle_kwargs)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 712, in read_array
    shape, fortran_order, dtype = _read_array_header(fp, version)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 578, in _read_array_header
    d = safe_eval(header)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/utils.py", line 1139, in safe_eval
    return ast.literal_eval(source)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
    map(_convert, node.values)))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 74, in _convert
    return list(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
    return tuple(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
    return tuple(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
    map(_convert, node.values)))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Call object at 0x10b965a58>

Numpy/Python version information:

1.17.0 3.7.2 (default, Jul 14 2019, 10:22:38)
[Clang 9.0.0 (clang-900.0.39.2)]

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions