Skip to content

gh-96512: Modernize int_max_str_digits protection in 3.12 #96568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Doc/c-api/init_config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -828,6 +828,18 @@ PyConfig

Default: ``0``.

.. c:member:: int32_t int_max_str_digits

If greater than 0, enable int conversion digit limitations. ``-1`` means
that :data:`sys.int_info.default_max_str_digits` will be used.

Configured by the :option:`-X int_max_str_digits <-X>` command line
flag or the :envvar:`PYTHONINTMAXSTRDIGITS` environment varable.

Default: ``-1``.

.. versionadded:: 3.12

.. c:member:: int isolated

If greater than ``0``, enable isolated mode:
Expand Down
2 changes: 1 addition & 1 deletion Doc/library/functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -910,7 +910,7 @@ are always available. They are listed here in alphabetical order.
.. versionchanged:: 3.11
The delegation to :meth:`__trunc__` is deprecated.

.. versionchanged:: 3.12
.. versionchanged:: 3.11
:class:`int` string inputs and string representations can be limited to
help avoid denial of service attacks. A :exc:`ValueError` is raised when
the limit is exceeded while converting a string *x* to an :class:`int` or
Expand Down
2 changes: 1 addition & 1 deletion Doc/library/json.rst
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ Basic Usage
be used to use another datatype or parser for JSON integers
(e.g. :class:`float`).

.. versionchanged:: 3.12
.. versionchanged:: 3.11
The default *parse_int* of :func:`int` now limits the maximum length of
the integer string via the interpreter's :ref:`integer string
conversion length limitation <int_max_str_digits>` to help avoid denial
Expand Down
18 changes: 17 additions & 1 deletion Doc/library/stdtypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5523,6 +5523,22 @@ Verification:
... '571186405732').to_bytes(53, 'big')
...

.. versionadded:: 3.11

.. audit-event:: int/digits/to_decimal value

When a conversion from an :class:`int` to decimal is estimated to be larger
than :data:`sys.int_info.str_digits_check_threshold <sys.int_info>` digits
an :ref:`auditing event <auditing>` with the :class:`int` ``value`` as the
argument is raised.

.. audit-event:: int/digits/from_base digits base

When a conversion to an :class:`int` from a base other than 2, 4, 8, 16, or
32 has more than :data:`sys.int_info.str_digits_check_threshold
<sys.int_info>` digits an :ref:`auditing event <auditing>` with the number
of ``digits`` and ``base`` is raised.

.. versionadded:: 3.12

Affected APIs
Expand Down Expand Up @@ -5578,7 +5594,7 @@ Information about the default and minimum can be found in :attr:`sys.int_info`:
* :data:`sys.int_info.str_digits_check_threshold <sys.int_info>` is the lowest
accepted value for the limit (other than 0 which disables it).

.. versionadded:: 3.12
.. versionadded:: 3.11

.. caution::

Expand Down
8 changes: 4 additions & 4 deletions Doc/library/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -544,7 +544,7 @@ always available.
.. versionchanged:: 3.11
Added the ``safe_path`` attribute for :option:`-P` option.

.. versionchanged:: 3.12
.. versionchanged:: 3.11
Added the ``int_max_str_digits`` attribute.


Expand Down Expand Up @@ -732,7 +732,7 @@ always available.
Returns the current value for the :ref:`integer string conversion length
limitation <int_max_str_digits>`. See also :func:`set_int_max_str_digits`.

.. versionadded:: 3.12
.. versionadded:: 3.11

.. function:: getrefcount(object)

Expand Down Expand Up @@ -1029,7 +1029,7 @@ always available.

.. versionadded:: 3.1

.. versionchanged:: 3.12
.. versionchanged:: 3.11
Added ``default_max_str_digits`` and ``str_digits_check_threshold``.


Expand Down Expand Up @@ -1337,7 +1337,7 @@ always available.
<int_max_str_digits>` used by this interpreter. See also
:func:`get_int_max_str_digits`.

.. versionadded:: 3.12
.. versionadded:: 3.11

.. function:: setprofile(profilefunc)

Expand Down
2 changes: 1 addition & 1 deletion Doc/library/test.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1018,7 +1018,7 @@ The :mod:`test.support` module defines the following functions:
context to allow execution of test code that needs a different limit
on the number of digits when converting between an integer and string.

.. versionadded:: 3.12
.. versionadded:: 3.11


The :mod:`test.support` module defines the following classes:
Expand Down
4 changes: 2 additions & 2 deletions Doc/using/cmdline.rst
Original file line number Diff line number Diff line change
Expand Up @@ -585,7 +585,7 @@ Miscellaneous options
.. versionadded:: 3.11
The ``-X frozen_modules`` option.

.. versionadded:: 3.12
.. versionadded:: 3.11
The ``-X int_max_str_digits`` option.

.. versionadded:: 3.12
Expand Down Expand Up @@ -775,7 +775,7 @@ conflict.
interpreter's global :ref:`integer string conversion length limitation
<int_max_str_digits>`.

.. versionadded:: 3.12
.. versionadded:: 3.11

.. envvar:: PYTHONIOENCODING

Expand Down
14 changes: 5 additions & 9 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,15 +84,11 @@ Other Language Changes
(Contributed by Serhiy Storchaka in :gh:`87995`.)

* Converting between :class:`int` and :class:`str` in bases other than 2
(binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal)
now raises a :exc:`ValueError` if the number of digits in string form is
above a limit to avoid potential denial of service attacks due to the
algorithmic complexity. This is a mitigation for `CVE-2020-10735
<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_.
This limit can be configured or disabled by environment variable, command
line flag, or :mod:`sys` APIs. See the :ref:`integer string conversion
length limitation <int_max_str_digits>` documentation. The default limit
is 4300 digits in string form.
(binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now
generates an :ref:`auditing event <auditing>` whenever the estimated number
of digits is above :data:`sys.int_info.str_digits_check_threshold
<sys.int_info>`. See the :ref:`integer string conversion length limitation
<int_max_str_digits>` documentation.

* :class:`memoryview` now supports the half-float type (the "e" format code).
(Contributed by Dong-hee Na and Antoine Pitrou in :gh:`90751`.)
Expand Down
1 change: 1 addition & 0 deletions Include/cpython/initconfig.h
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ typedef struct PyConfig {
wchar_t *check_hash_pycs_mode;
int use_frozen_modules;
int safe_path;
int32_t int_max_str_digits;

/* --- Path configuration inputs ------------ */
int pathconfig_warnings;
Expand Down
5 changes: 3 additions & 2 deletions Include/internal/pycore_initconfig.h
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ PyAPI_FUNC(PyStatus) _PyArgv_AsWstrList(const _PyArgv *args,
PyAPI_FUNC(int) _Py_str_to_int(
const char *str,
int *result);
PyAPI_FUNC(int) _Py_str_to_int32(
const char *str,
int32_t *result);
PyAPI_FUNC(const wchar_t*) _Py_get_xoption(
const PyWideStringList *xoptions,
const wchar_t *name);
Expand Down Expand Up @@ -170,8 +173,6 @@ extern void _Py_DumpPathConfig(PyThreadState *tstate);

PyAPI_FUNC(PyObject*) _Py_Get_Getpath_CodeObject(void);

extern int _Py_global_config_int_max_str_digits; // TODO(gpshead): move this into PyConfig in 3.12 after the backports ship.


/* --- Function used for testing ---------------------------------- */

Expand Down
2 changes: 1 addition & 1 deletion Include/internal/pycore_interp.h
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ struct _is {
struct types_state types;
struct callable_cache callable_cache;

int int_max_str_digits;
int32_t int_max_str_digits;

/* The following fields are here to avoid allocation during init.
The data is exposed through PyInterpreterState pointer fields.
Expand Down
23 changes: 18 additions & 5 deletions Lib/test/audit-tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

import contextlib
import os
from test import support
import sys


Expand Down Expand Up @@ -409,8 +410,6 @@ def hook(event, *args):


def test_sys_getframe():
import sys

def hook(event, args):
if event.startswith("sys."):
print(event, args[0].f_code.co_name)
Expand All @@ -430,10 +429,24 @@ def hook(event, args):
_wmi.exec_query("SELECT * FROM Win32_OperatingSystem")


if __name__ == "__main__":
from test.support import suppress_msvcrt_asserts
def test_int_digits():
def hook(event, args):
if event.startswith("int/digits/"):
print(event, *(hex(arg) for arg in args))

sys.addaudithook(hook)

threshold = sys.int_info.str_digits_check_threshold
with support.adjust_int_max_str_digits(2*threshold):
fives = int('5' * (threshold+20))
int('1' * (threshold*3//2), base=36)
for base in (2, 4, 8, 16, 32): # linear unaudited bases
int('1' * (threshold+1), base)
str(fives)

suppress_msvcrt_asserts()

if __name__ == "__main__":
support.suppress_msvcrt_asserts()

test = sys.argv[1]
globals()[test]()
27 changes: 27 additions & 0 deletions Lib/test/test_audit.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,5 +200,32 @@ def test_wmi_exec_query(self):
self.assertEqual(actual, expected)


def test_int_digits(self):
returncode, events, stderr = self.run_python("test_int_digits")
if returncode:
self.fail(stderr)

if support.verbose:
print(*events, sep='\n')
actual = [(ev[0], ev[2].split(' ')) for ev in events]

threshold = sys.int_info.str_digits_check_threshold
self.assertEqual(
actual.pop(0),
("int/digits/from_base", [hex(threshold+20), hex(10)]),
)
self.assertEqual(
actual.pop(0),
("int/digits/from_base", [hex(threshold*3//2), hex(36)]),
)

event = actual.pop(0)
self.assertEqual(event[0], "int/digits/to_decimal")
self.assertEqual(len(event[1]), 1)
self.assertGreater(len(event[1][0]), threshold//2)

self.assertEqual(actual, [])


if __name__ == "__main__":
unittest.main()
1 change: 1 addition & 0 deletions Lib/test/test_embed.py
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,7 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase):
'install_signal_handlers': 1,
'use_hash_seed': 0,
'hash_seed': 0,
'int_max_str_digits': -1,
'faulthandler': 0,
'tracemalloc': 0,
'perf_profiling': 0,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Added ``int/digits/to_decimal`` and ``int/digits/from_base`` :ref:`auditing
events <auditing>` on :class:`int` non-linear algorithm base conversions
with more than `sys.int_info.str_digits_check_threshold <sys.int_info>`
digits.

This file was deleted.

14 changes: 10 additions & 4 deletions Objects/longobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -1766,8 +1766,11 @@ long_to_decimal_string_internal(PyObject *aa,
*/
if (size_a >= 10 * _PY_LONG_MAX_STR_DIGITS_THRESHOLD
/ (3 * PyLong_SHIFT) + 2) {
if (PySys_Audit("int/digits/to_decimal", "O", aa) < 0) {
return -1;
}
PyInterpreterState *interp = _PyInterpreterState_GET();
int max_str_digits = interp->int_max_str_digits;
int32_t max_str_digits = interp->int_max_str_digits;
if ((max_str_digits > 0) &&
(max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10)) {
PyErr_Format(PyExc_ValueError, _MAX_STR_DIGITS_ERROR_FMT_TO_STR,
Expand Down Expand Up @@ -1837,7 +1840,7 @@ long_to_decimal_string_internal(PyObject *aa,
}
if (strlen > _PY_LONG_MAX_STR_DIGITS_THRESHOLD) {
PyInterpreterState *interp = _PyInterpreterState_GET();
int max_str_digits = interp->int_max_str_digits;
int32_t max_str_digits = interp->int_max_str_digits;
Py_ssize_t strlen_nosign = strlen - negative;
if ((max_str_digits > 0) && (strlen_nosign > max_str_digits)) {
Py_DECREF(scratch);
Expand Down Expand Up @@ -2513,8 +2516,11 @@ digit beyond the first.

/* Limit the size to avoid excessive computation attacks. */
if (digits > _PY_LONG_MAX_STR_DIGITS_THRESHOLD) {
if (PySys_Audit("int/digits/from_base", "ni", digits, base) < 0) {
return NULL;
}
PyInterpreterState *interp = _PyInterpreterState_GET();
int max_str_digits = interp->int_max_str_digits;
int32_t max_str_digits = interp->int_max_str_digits;
if ((max_str_digits > 0) && (digits > max_str_digits)) {
PyErr_Format(PyExc_ValueError, _MAX_STR_DIGITS_ERROR_FMT_TO_INT,
max_str_digits, digits);
Expand Down Expand Up @@ -6196,7 +6202,7 @@ _PyLong_InitTypes(PyInterpreterState *interp)
return _PyStatus_ERR("can't init int info type");
}
}
interp->int_max_str_digits = _Py_global_config_int_max_str_digits;
interp->int_max_str_digits = _PyInterpreterState_GetConfig(interp)->int_max_str_digits;
if (interp->int_max_str_digits == -1) {
interp->int_max_str_digits = _PY_LONG_DEFAULT_MAX_STR_DIGITS;
}
Expand Down
8 changes: 4 additions & 4 deletions Python/clinic/sysmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading