Skip to content

gh-98306: Support JSON encoding of NaNs and infinities as null #115246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
35 changes: 27 additions & 8 deletions Doc/library/json.rst
Original file line number Diff line number Diff line change
Expand Up @@ -172,9 +172,14 @@ Basic Usage

If *allow_nan* is false (default: ``True``), then it will be a
:exc:`ValueError` to serialize out of range :class:`float` values (``nan``,
``inf``, ``-inf``) in strict compliance of the JSON specification.
If *allow_nan* is true, their JavaScript equivalents (``NaN``,
``Infinity``, ``-Infinity``) will be used.
``inf``, ``-inf``) in strict compliance with the JSON specification. If
*allow_nan* is the string ``'as_null'``, NaNs and infinities will be
converted to a JSON ``null``, matching the behavior of JavaScript's
``JSON.stringify``. If *allow_nan* is true but not equal to ``'as_null'``
then NaNs and infinities are converted to non-quote-delimited strings
``NaN``, ``Infinity`` and ``-Infinity`` in the JSON output. Note that this
represents an extension of the JSON specification, and that the generated
output may not be accepted as valid JSON by third-party JSON parsers.

If *indent* is a non-negative integer or string, then JSON array elements and
object members will be pretty-printed with that indent level. An indent level
Expand Down Expand Up @@ -209,6 +214,11 @@ Basic Usage
.. versionchanged:: 3.6
All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`.

.. versionchanged:: 3.14
Added support for ``allow_nan='as_null'``. Passing any string value
other than ``'as_null'`` for *allow_nan* now triggers a
:exc:`DeprecationWarning`.

.. note::

Unlike :mod:`pickle` and :mod:`marshal`, JSON is not a framed protocol,
Expand Down Expand Up @@ -450,11 +460,16 @@ Encoders and Decoders
prevent an infinite recursion (which would cause a :exc:`RecursionError`).
Otherwise, no such check takes place.

If *allow_nan* is true (the default), then ``NaN``, ``Infinity``, and
``-Infinity`` will be encoded as such. This behavior is not JSON
specification compliant, but is consistent with most JavaScript based
encoders and decoders. Otherwise, it will be a :exc:`ValueError` to encode
such floats.
If *allow_nan* is false (default: ``True``), then it will be a
:exc:`ValueError` to serialize out of range :class:`float` values (``nan``,
``inf``, ``-inf``) in strict compliance with the JSON specification. If
*allow_nan* is the string ``'as_null'``, NaNs and infinities will be
converted to a JSON ``null``, matching the behavior of JavaScript's
``JSON.stringify``. If *allow_nan* is true but not equal to ``'as_null'``
then NaNs and infinities are converted to non-quote-delimited strings
``NaN``, ``Infinity`` and ``-Infinity`` in the JSON output. Note that this
represents an extension of the JSON specification, and that the generated
output may not be accepted as valid JSON by third-party JSON parsers.

If *sort_keys* is true (default: ``False``), then the output of dictionaries
will be sorted by key; this is useful for regression tests to ensure that
Expand Down Expand Up @@ -486,6 +501,10 @@ Encoders and Decoders
.. versionchanged:: 3.6
All parameters are now :ref:`keyword-only <keyword-only_parameter>`.

.. versionchanged:: 3.14
Added support for ``allow_nan='as_null'``. Passing any string value
other than ``'as_null'`` for *allow_nan* now triggers a
:exc:`DeprecationWarning`.

.. method:: default(o)

Expand Down
8 changes: 8 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,14 @@ New Modules
Improved Modules
================

json
----

* Add support for ``allow_nan='as_null'`` when encoding to JSON. This converts
floating-point infinities and NaNs to a JSON ``null``, for alignment
with ECMAScript's ``JSON.stringify``.
(Contributed by Mark Dickinson in :gh:`115246`.)


Optimizations
=============
Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_global_objects_fini_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Include/internal/pycore_global_strings.h
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(arguments)
STRUCT_FOR_ID(argv)
STRUCT_FOR_ID(as_integer_ratio)
STRUCT_FOR_ID(as_null)
STRUCT_FOR_ID(asend)
STRUCT_FOR_ID(ast)
STRUCT_FOR_ID(athrow)
Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_runtime_init_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Include/internal/pycore_unicodeobject_generated.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Lib/json/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ def dumps(obj, *, skipkeys=False, ensure_ascii=True, check_circular=True,
"""
# cached encoder
if (not skipkeys and ensure_ascii and
check_circular and allow_nan and
check_circular and allow_nan is True and
cls is None and indent is None and separators is None and
default is None and not sort_keys and not kw):
return _default_encoder.encode(obj)
Expand Down
12 changes: 11 additions & 1 deletion Lib/json/encoder.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""Implementation of JSONEncoder
"""
import re
import warnings

try:
from _json import encode_basestring_ascii as c_encode_basestring_ascii
Expand Down Expand Up @@ -148,6 +149,13 @@ def __init__(self, *, skipkeys=False, ensure_ascii=True,
self.skipkeys = skipkeys
self.ensure_ascii = ensure_ascii
self.check_circular = check_circular
if isinstance(allow_nan, str) and allow_nan != 'as_null':
warnings.warn(
"in the future, allow_nan will no longer accept strings "
"other than 'as_null'. Use a boolean instead.",
DeprecationWarning,
stacklevel=3,
)
self.allow_nan = allow_nan
self.sort_keys = sort_keys
self.indent = indent
Expand Down Expand Up @@ -236,7 +244,9 @@ def floatstr(o, allow_nan=self.allow_nan,
else:
return _repr(o)

if not allow_nan:
if allow_nan == 'as_null':
return 'null'
elif not allow_nan:
raise ValueError(
"Out of range float values are not JSON compliant: " +
repr(o))
Expand Down
35 changes: 35 additions & 0 deletions Lib/test/test_json/test_float.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
from test.test_json import PyTest, CTest


class NotUsableAsABoolean:
def __bool__(self):
raise TypeError("I refuse to be interpreted as a boolean")


class TestFloat:
def test_floats(self):
for num in [1617161771.7650001, math.pi, math.pi**100, math.pi**-100, 3.1]:
Expand Down Expand Up @@ -29,6 +34,36 @@ def test_allow_nan(self):
msg = f'Out of range float values are not JSON compliant: {val}'
self.assertRaisesRegex(ValueError, msg, self.dumps, [val], allow_nan=False)

def test_allow_nan_null(self):
# when allow_nan is 'as_null', infinities and NaNs convert to 'null'
for val in [float('inf'), float('-inf'), float('nan')]:
with self.subTest(val=val):
out = self.dumps([val], allow_nan='as_null')
res = self.loads(out)
self.assertEqual(res, [None])

# and finite values are treated as normal
for val in [1.25, -23, -0.0, 0.0]:
with self.subTest(val=val):
out = self.dumps([val], allow_nan='as_null')
res = self.loads(out)
self.assertEqual(res, [val])

# testing a mixture
vals = [-1.3, 1e100, -math.inf, 1234, -0.0, math.nan]
out = self.dumps(vals, allow_nan='as_null')
res = self.loads(out)
self.assertEqual(res, [-1.3, 1e100, None, 1234, -0.0, None])

def test_allow_nan_string_deprecation(self):
with self.assertWarns(DeprecationWarning):
self.dumps(2.3, allow_nan='true')

def test_allow_nan_non_boolean(self):
# check that exception gets propagated as expected
with self.assertRaises(TypeError):
self.dumps(math.inf, allow_nan=NotUsableAsABoolean())


class TestPyFloat(TestFloat, PyTest): pass
class TestCFloat(TestFloat, CTest): pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add support for ``allow_nan='as_null'`` when encoding an object to a JSON
string. This converts floating-point infinities and NaNs to a JSON ``null``.
25 changes: 21 additions & 4 deletions Modules/_json.c
Original file line number Diff line number Diff line change
Expand Up @@ -1209,13 +1209,13 @@ encoder_new(PyTypeObject *type, PyObject *args, PyObject *kwds)

PyEncoderObject *s;
PyObject *markers, *defaultfn, *encoder, *indent, *key_separator;
PyObject *item_separator;
PyObject *item_separator, *allow_nan_obj;
int sort_keys, skipkeys, allow_nan;

if (!PyArg_ParseTupleAndKeywords(args, kwds, "OOOOUUppp:make_encoder", kwlist,
if (!PyArg_ParseTupleAndKeywords(args, kwds, "OOOOUUppO:make_encoder", kwlist,
&markers, &defaultfn, &encoder, &indent,
&key_separator, &item_separator,
&sort_keys, &skipkeys, &allow_nan))
&sort_keys, &skipkeys, &allow_nan_obj))
return NULL;

if (markers != Py_None && !PyDict_Check(markers)) {
Expand All @@ -1225,6 +1225,20 @@ encoder_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
return NULL;
}

// allow_nan =
// 0 to disallow nans and infinities
// 1 to convert nans and infinities into corresponding JSON strings
// 2 to convert nans and infinities to a JSON null
if (PyUnicode_Check(allow_nan_obj) &&
_PyUnicode_Equal(allow_nan_obj, &_Py_ID(as_null))) {
allow_nan = 2;
} else {
allow_nan = PyObject_IsTrue(allow_nan_obj);
if (allow_nan < 0) {
return NULL;
}
}

s = (PyEncoderObject *)type->tp_alloc(type, 0);
if (s == NULL)
return NULL;
Expand Down Expand Up @@ -1335,7 +1349,10 @@ encoder_encode_float(PyEncoderObject *s, PyObject *obj)
);
return NULL;
}
if (i > 0) {
else if (s->allow_nan == 2) {
return PyUnicode_FromString("null");
}
else if (i > 0) {
return PyUnicode_FromString("Infinity");
}
else if (i < 0) {
Expand Down
Loading