Skip to content

gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes #126938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Dec 10, 2024
+36 −4
Merged

gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes #126938

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
1962567
Fix rejection when non-bitfield fields are larger than 2^16 bytes
Melissa0x1f992 Nov 17, 2024
1900cf5
Merge pull request #1 from Melissa0x1f992/use-bitsize-param
Melissa0x1f992 Nov 17, 2024
2bd972b
📜🤖 Added by blurb_it.
blurb-it[bot] Nov 17, 2024
d2954cd
Update Modules/_ctypes/cfield.c
terryjreedy Nov 17, 2024
0d21229
Remove specific dependency on Python 3.12 for black
Melissa0x1f992 Nov 18, 2024
d8a6f08
Add test case for large fields
Melissa0x1f992 Nov 18, 2024
16bfb44
Merge pull request #2 from Melissa0x1f992/use-bitsize-param
Melissa0x1f992 Nov 18, 2024
7402a8f
Reduce test field size to (2^31 - 1) to support 32 bit systems
Melissa0x1f992 Nov 18, 2024
8994ad0
Merge branch 'python:main' into main
Melissa0x1f992 Nov 18, 2024
db171e0
Reduce test field size to (2^31 - 1) to support 32 bit systems
Melissa0x1f992 Nov 18, 2024
7bc0ceb
Merge pull request #3 from Melissa0x1f992/use-bitsize-param
Melissa0x1f992 Nov 18, 2024
cd8c011
Merge in the main branch
encukou Dec 4, 2024
a3bd171
Remove pre-commit config (moved to #126971)
encukou Dec 4, 2024
5e8aec0
Use PyLong_AsSsize_t to convert to Py_ssize_t
encukou Dec 4, 2024
7d3bb10
Test maximum field size (sys.maxsize / 8, so we can count bits later)
encukou Dec 4, 2024
946f261
Rework the asserts.
encukou Dec 4, 2024
19c1b3b
Test naked c_char as well
encukou Dec 4, 2024
dcf97f2
Better name for the test
encukou Dec 4, 2024
03bc32b
Add comment
encukou Dec 4, 2024
f61faa6
Reword changelog
encukou Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions Lib/test/test_ctypes/test_struct_fields.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import unittest
import sys
from ctypes import Structure, Union, sizeof, c_char, c_int
from ._support import (CField, Py_TPFLAGS_DISALLOW_INSTANTIATION,
Py_TPFLAGS_IMMUTABLETYPE)
Expand Down Expand Up @@ -75,6 +76,28 @@ def __init_subclass__(cls, **kwargs):
'ctypes state is not initialized'):
class Subclass(BrokenStructure): ...

def test_max_field_size_gh126937(self):
# Classes for big structs should be created successfully.
# (But they most likely can't be instantiated.)
# Here we test the exact limit: the number of *bits* must fit
# in Py_ssize_t.

class X(self.cls):
_fields_ = [('char', c_char),]
max_field_size = sys.maxsize // 8

class Y(self.cls):
_fields_ = [('largeField', X * max_field_size)]
class Z(self.cls):
_fields_ = [('largeField', c_char * max_field_size)]

with self.assertRaises(ValueError):
class TooBig(self.cls):
_fields_ = [('largeField', X * (max_field_size + 1))]
with self.assertRaises(ValueError):
class TooBig(self.cls):
_fields_ = [('largeField', c_char * (max_field_size + 1))]

# __set__ and __get__ should raise a TypeError in case their self
# argument is not a ctype instance.
def test___set__(self):
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2024-11-17-21-35-55.gh-issue-126937.qluVM0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Fix :exc:`TypeError` when a :class:`ctypes.Structure` has a field size
that doesn't fit into an unsigned 16-bit integer.
Instead, the maximum number of *bits* is :data:`sys.maxsize`.
10 changes: 8 additions & 2 deletions Modules/_ctypes/cfield.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,16 @@ PyCField_new_impl(PyTypeObject *type, PyObject *name, PyObject *proto,
goto error;
}

Py_ssize_t bit_size = NUM_BITS(size);
if (bit_size) {
if (bit_size_obj != Py_None) {
#ifdef Py_DEBUG
Py_ssize_t bit_size = NUM_BITS(size);
assert(bit_size > 0);
assert(bit_size <= info->size * 8);
// Currently, the bit size is specified redundantly
// in NUM_BITS(size) and bit_size_obj.
// Verify that they match.
assert(PyLong_AsSsize_t(bit_size_obj) == bit_size);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be guaranteed? What if bit_size_obj is not None and is not an int?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can.
The implementation is split between internal Python code in ctypes._layout, and C code here. They are currently tightly coupled. The assert also verifies the Python part.

#endif
switch(info->ffi_type_pointer.type) {
case FFI_TYPE_UINT8:
case FFI_TYPE_UINT16:
Expand Down
4 changes: 2 additions & 2 deletions Modules/_ctypes/stgdict.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ PyCStructUnionType_update_stginfo(PyObject *type, PyObject *fields, int isStruct
if (!tmp) {
goto error;
}
Py_ssize_t total_align = PyLong_AsInt(tmp);
Py_ssize_t total_align = PyLong_AsSsize_t(tmp);
Py_DECREF(tmp);
if (total_align < 0) {
if (!PyErr_Occurred()) {
Expand All @@ -306,7 +306,7 @@ PyCStructUnionType_update_stginfo(PyObject *type, PyObject *fields, int isStruct
if (!tmp) {
goto error;
}
Py_ssize_t total_size = PyLong_AsInt(tmp);
Py_ssize_t total_size = PyLong_AsSsize_t(tmp);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was a reason for using PyLong_AsInt(). This is a new API, so it unlikely was accident. Is there a limitation on total_size to be in the range of int?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was an accident -- my mistake.

Py_DECREF(tmp);
if (total_size < 0) {
if (!PyErr_Occurred()) {
Expand Down
Loading