Skip to content

int64 is not supported by the standard library's json encoder #27651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chillenb opened this issue Oct 27, 2024 · 3 comments
Open

int64 is not supported by the standard library's json encoder #27651

chillenb opened this issue Oct 27, 2024 · 3 comments
Labels
01 - Enhancement 57 - Close? Issues which may be closable unless discussion continued

Comments

@chillenb
Copy link

Describe the issue:

On x86_64, NumPy ints cannot be serialized with json, unlike Python ints. You should implement a default method to fix this.

Reproduce the code example:

import json
import numpy as np

A = np.zeros(10, dtype=int)
with open("myfile.txt", "w") as f:
    json.dump({'number': A[0]}, f)

Error message:

Traceback (most recent call last):
  File "/home/chillenb/src/fcdmft/serialexample.py", line 6, in <module>
    json.dump({'number': A[0]}, f)
  File "/home/chillenb/miniforge3/envs/fcdmftenv/lib/python3.11/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/home/chillenb/miniforge3/envs/fcdmftenv/lib/python3.11/json/encoder.py", line 432, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/home/chillenb/miniforge3/envs/fcdmftenv/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "/home/chillenb/miniforge3/envs/fcdmftenv/lib/python3.11/json/encoder.py", line 439, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "/home/chillenb/miniforge3/envs/fcdmftenv/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int64 is not JSON serializable

Python and NumPy Versions:

2.0.2
3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0]

Runtime Environment:

[{'numpy_version': '2.0.2',
'python': '3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) '
'[GCC 12.3.0]',
'uname': uname_result(system='Linux', node='pizza', release='6.10.10-arch1-1-custom', version='#1 SMP Tue, 22 Oct 2024 13:11:16 +0000', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2',
'AVX512F',
'AVX512CD',
'AVX512_SKX',
'AVX512_CLX'],
'not_found': ['AVX512_KNL',
'AVX512_KNM',
'AVX512_CNL',
'AVX512_ICL',
'AVX512_SPR']}},
{'filepath': '/home/chillenb/miniforge3/envs/fcdmftenv/lib/libmkl_rt.so.2',
'internal_api': 'mkl',
'num_threads': 48,
'prefix': 'libmkl_rt',
'threading_layer': 'intel',
'user_api': 'blas',
'version': '2023.2-Product'},
{'filepath': '/home/chillenb/miniforge3/envs/fcdmftenv/lib/libomp.so',
'internal_api': 'openmp',
'num_threads': 48,
'prefix': 'libomp',
'user_api': 'openmp',
'version': None}]

Context for the issue:

This is bad because it will require lots of casting to fix, to the detriment of code readability.

It'd be nice if NumPy had a type that behaved exactly like Python ints!

@shoyer shoyer changed the title BUG: NumPy 2.0 breaks lots of code that uses the standard library's json encoder int64 is not supported by the standard library's json encoder Oct 28, 2024
@shoyer
Copy link
Member

shoyer commented Oct 28, 2024

This is not a new NumPy 2.0 issue -- numpy.int64 has never inherited from Python's builtin int, unlike float64 which inherits from float: #17283

It is unlike that NumPy will add this feature, because there is no standard representation of fixed-width integers like int64 in JSON. If you want to represent numpy.int64 as an integer, you should explicitly cast to an integer, e.g., with ndarray.item() or ndarray.tolist().

@chillenb
Copy link
Author

My bad. I should have tried this on an older version of NumPy first. It doesn't work on 1.26 either.

I really wish there were a nice, non-pickle way to serialize everything that can be constructed from dict, set, tuple, list, bool, and the various numeric types. JSON is not fit for that, but at least it's already there.

@charris
Copy link
Member

charris commented Oct 28, 2024

Might take a look at https://github.com/loco-philippe/ntv-numpy. I have no idea if it works in this case.

@ngoldbaum ngoldbaum added the 57 - Close? Issues which may be closable unless discussion continued label Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
01 - Enhancement 57 - Close? Issues which may be closable unless discussion continued
Projects
None yet
Development

No branches or pull requests

5 participants