Skip to content

gh-117031: Add support for arbitrary C integer types for PyMemberDef.type #132550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
+175 −62
Open

gh-117031: Add support for arbitrary C integer types for PyMemberDef.type #132550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b83e7a3
gh-117031: Add support for arbitrary C integer types for PyMemberDef.…
serhiy-storchaka Apr 15, 2025
9bae8bd
Fix errors.
serhiy-storchaka Apr 15, 2025
812814a
Remove some redundant tests.
serhiy-storchaka Apr 15, 2025
75645e1
Merge branch 'main' into structmember-types2
serhiy-storchaka Apr 15, 2025
6c175a8
Apply suggestions from code review
serhiy-storchaka Apr 15, 2025
436144e
Fix reference.
serhiy-storchaka Apr 15, 2025
34eee62
Support combined types.
serhiy-storchaka Apr 17, 2025
cbd947d
Merge remote-tracking branch 'refs/remotes/origin/structmember-types2…
serhiy-storchaka Apr 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Doc/c-api/hash.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ See also the :c:member:`PyTypeObject.tp_hash` member and :ref:`numeric-hash`.
.. c:function:: Py_hash_t Py_HashPointer(const void *ptr)

Hash a pointer value: process the pointer value as an integer (cast it to
``uintptr_t`` internally). The pointer is not dereferenced.
:c:expr:`uintptr_t` internally). The pointer is not dereferenced.

The function cannot fail: it cannot return ``-1``.

Expand Down
18 changes: 13 additions & 5 deletions Doc/c-api/structures.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -615,6 +615,7 @@ using e.g. :keyword:`del` or :py:func:`delattr`.
================================ ============================= ======================
Macro name C type Python type
================================ ============================= ======================
.. c:macro:: Py_T_INTEGER(type) *type* (INT) :py:class:`int`
.. c:macro:: Py_T_BYTE :c:expr:`char` :py:class:`int`
.. c:macro:: Py_T_SHORT :c:expr:`short` :py:class:`int`
.. c:macro:: Py_T_INT :c:expr:`int` :py:class:`int`
Expand All @@ -630,18 +631,21 @@ Macro name C type Python type
.. c:macro:: Py_T_DOUBLE :c:expr:`double` :py:class:`float`
.. c:macro:: Py_T_BOOL :c:expr:`char` :py:class:`bool`
(written as 0 or 1)
.. c:macro:: Py_T_STRING :c:expr:`const char *` (*) :py:class:`str` (RO)
.. c:macro:: Py_T_STRING_INPLACE :c:expr:`const char[]` (*) :py:class:`str` (RO)
.. c:macro:: Py_T_CHAR :c:expr:`char` (0-127) :py:class:`str` (**)
.. c:macro:: Py_T_STRING :c:expr:`const char *` (STR) :py:class:`str` (RO)
.. c:macro:: Py_T_STRING_INPLACE :c:expr:`const char[]` (STR) :py:class:`str` (RO)
.. c:macro:: Py_T_CHAR :c:expr:`char` (0-127) :py:class:`str` (CH)
.. c:macro:: Py_T_OBJECT_EX :c:expr:`PyObject *` :py:class:`object` (D)
================================ ============================= ======================

(*): Zero-terminated, UTF8-encoded C string.
(INT): Macro :c:macro:`Py_T_INTEGER(type) <Py_T_INTEGER>` represents an arbitrary
C integer type *type*.

(STR): Zero-terminated, UTF8-encoded C string.
With :c:macro:`!Py_T_STRING` the C representation is a pointer;
with :c:macro:`!Py_T_STRING_INPLACE` the string is stored directly
in the structure.

(**): String of length 1. Only ASCII is accepted.
(CH): String of length 1. Only ASCII is accepted.

(RO): Implies :c:macro:`Py_READONLY`.

Expand Down Expand Up @@ -687,6 +691,10 @@ Macro name C type Python type

Always ``None``. Must be used with :c:macro:`Py_READONLY`.

.. versionadded:: next
Added :c:macro:`Py_T_INTEGER()`.


Defining Getters and Setters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
2 changes: 2 additions & 0 deletions Doc/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,9 @@
('c:type', 'int32_t'),
('c:type', 'int64_t'),
('c:type', 'intmax_t'),
('c:type', 'intptr_t'),
('c:type', 'off_t'),
('c:type', 'pid_t'),
('c:type', 'ptrdiff_t'),
('c:type', 'siginfo_t'),
('c:type', 'size_t'),
Expand Down
4 changes: 4 additions & 0 deletions Doc/whatsnew/3.14.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1818,6 +1818,10 @@ New features

(Contributed by Victor Stinner in :gh:`120389`.)

* Add support for arbitrary C integer types in :c:member:`PyMemberDef.type`.
The macro :c:macro:`Py_T_INTEGER(type) <Py_T_INTEGER>` represents a C type *type*.
(Contributed by Serhiy Storchaka and Petr Viktorin in :gh:`117031`.)

* Add :c:func:`PyBytes_Join(sep, iterable) <PyBytes_Join>` function,
similar to ``sep.join(iterable)`` in Python.
(Contributed by Victor Stinner in :gh:`121645`.)
Expand Down
2 changes: 2 additions & 0 deletions Include/descrobject.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ struct PyMemberDef {
#define Py_T_PYSSIZET 19 /* Py_ssize_t */
#define _Py_T_NONE 20 // Deprecated. Value is always None.

#define Py_T_INTEGER(type) ((sizeof(type) << 8) | (_Py_IS_TYPE_SIGNED(type) ? 1 : 2))

/* Flags */
#define Py_READONLY 1
#define Py_AUDIT_READ 2 // Added in 3.10, harmless no-op before that
Expand Down
60 changes: 50 additions & 10 deletions Lib/test/test_capi/test_structmembers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
INT_MAX, INT_MIN, UINT_MAX,
LONG_MAX, LONG_MIN, ULONG_MAX,
LLONG_MAX, LLONG_MIN, ULLONG_MAX,
PY_SSIZE_T_MAX, PY_SSIZE_T_MIN,
PY_SSIZE_T_MAX, PY_SSIZE_T_MIN, SIZE_MAX,
SIZEOF_INTMAX_T, SIZEOF_INTPTR_T, SIZEOF_PTRDIFF_T, SIZEOF_OFF_T,
SIZEOF_PID_T, SIZEOF_INT,
)


Expand All @@ -19,6 +21,8 @@ def __init__(self, value):
self.value = value
def __index__(self):
return self.value
def __repr__(self):
return f'Index({self.value!r})'

# There are two classes: one using <structmember.h> and another using
# `Py_`-prefixed API. They should behave the same in Python
Expand Down Expand Up @@ -60,22 +64,22 @@ def _test_warn(self, name, value, expected=None):
if expected is not None:
self.assertEqual(getattr(ts, name), expected)

def _test_overflow(self, name, value):
def _test_overflow(self, name, value, error=OverflowError):
ts = self.ts
self.assertRaises(OverflowError, setattr, ts, name, value)
self.assertRaises(error, setattr, ts, name, value)

def _test_int_range(self, name, minval, maxval, *, hardlimit=None,
indexlimit=None):
indexlimit=None, negvalueerror=OverflowError, wrap=False):
if hardlimit is None:
hardlimit = (minval, maxval)
ts = self.ts
self._test_write(name, minval)
self._test_write(name, maxval)
self._test_write(name, maxval, -1 if wrap else maxval)
hardminval, hardmaxval = hardlimit
self._test_overflow(name, hardminval-1)
self._test_overflow(name, hardminval-1, error=negvalueerror)
self._test_overflow(name, hardmaxval+1)
self._test_overflow(name, 2**1000)
self._test_overflow(name, -2**1000)
self._test_overflow(name, -2**1000, error=negvalueerror)
if hardminval < minval:
self._test_warn(name, hardminval)
self._test_warn(name, minval-1, maxval)
Expand All @@ -88,11 +92,11 @@ def _test_int_range(self, name, minval, maxval, *, hardlimit=None,
self.assertRaises(TypeError, setattr, ts, name, Index(maxval))
else:
self._test_write(name, Index(minval), minval)
self._test_write(name, Index(maxval), maxval)
self._test_overflow(name, Index(hardminval-1))
self._test_write(name, Index(maxval), -1 if wrap else maxval)
self._test_overflow(name, Index(hardminval-1), error=negvalueerror)
self._test_overflow(name, Index(hardmaxval+1))
self._test_overflow(name, Index(2**1000))
self._test_overflow(name, Index(-2**1000))
self._test_overflow(name, Index(-2**1000), error=negvalueerror)
if hardminval < minval:
self._test_warn(name, Index(hardminval))
self._test_warn(name, Index(minval-1), maxval)
Expand Down Expand Up @@ -181,6 +185,42 @@ class ReadWriteTests_OldAPI(ReadWriteTests, unittest.TestCase):
class ReadWriteTests_NewAPI(ReadWriteTests, unittest.TestCase):
cls = _test_structmembersType_NewAPI

def test_size(self):
self._test_int_range('T_SIZE', 0, SIZE_MAX, negvalueerror=ValueError)

def test_int8(self):
self._test_int_range('T_INT8', -2**7, 2**7-1)
self._test_int_range('T_UINT8', 0, 2**8-1, negvalueerror=ValueError)
self._test_int_range('T_XINT8', -2**7, 2**8-1, wrap=True)

def test_int16(self):
self._test_int_range('T_INT16', -2**15, 2**15-1)
self._test_int_range('T_UINT16', 0, 2**16-1, negvalueerror=ValueError)
self._test_int_range('T_XINT16', -2**15, 2**16-1, wrap=True)

def test_int32(self):
self._test_int_range('T_INT32', -2**31, 2**31-1)
self._test_int_range('T_UINT32', 0, 2**32-1, negvalueerror=ValueError)
self._test_int_range('T_XINT32', -2**31, 2**32-1, wrap=True)

def test_int64(self):
self._test_int_range('T_INT64', -2**63, 2**63-1)
self._test_int_range('T_UINT64', 0, 2**64-1, negvalueerror=ValueError)
self._test_int_range('T_XINT64', -2**63, 2**64-1, wrap=True)

def test_intptr(self):
bits = 8*SIZEOF_INTPTR_T
self._test_int_range('T_INTPTR', -2**(bits-1), 2**(bits-1)-1)
self._test_int_range('T_UINTPTR', 0, 2**bits-1, negvalueerror=ValueError)

def test_off(self):
bits = 8*SIZEOF_OFF_T
self._test_int_range('T_OFF', -2**(bits-1), 2**(bits-1)-1)

def test_pid(self):
bits = 8*SIZEOF_PID_T
self._test_int_range('T_PID', -2**(bits-1), 2**(bits-1)-1)


if __name__ == "__main__":
unittest.main()
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/C_API/2024-03-19-21-13-20.gh-issue-117031.0s3Ruq.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add support for arbitrary C integer types in :c:member:`PyMemberDef.type`.
The macro :c:macro:`Py_T_INTEGER(type) <Py_T_INTEGER>` represents a C type *type*.
11 changes: 2 additions & 9 deletions Modules/_multiprocessing/multiprocessing.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,8 @@
* Format codes
*/

#if SIZEOF_VOID_P == SIZEOF_LONG
# define F_POINTER "k"
# define T_POINTER T_ULONG
#elif SIZEOF_VOID_P == SIZEOF_LONG_LONG
# define F_POINTER "K"
# define T_POINTER T_ULONGLONG
#else
# error "can't find format code for unsigned integer of same size as void*"
#endif
#define F_POINTER _Py_PARSE_UINTPTR
#define T_POINTER Py_T_INTEGER(uintptr_t)

#ifdef MS_WINDOWS
# define F_HANDLE F_POINTER
Expand Down
42 changes: 42 additions & 0 deletions Modules/_testcapi/structmember.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,33 @@ typedef struct {
long long_member;
unsigned long ulong_member;
Py_ssize_t pyssizet_member;
size_t size_member;
float float_member;
double double_member;
char inplace_member[6];
long long longlong_member;
unsigned long long ulonglong_member;
char char_member;
int8_t int8_member;
uint8_t uint8_member;
uint8_t xint8_member;
int16_t int16_member;
uint16_t uint16_member;
uint16_t xint16_member;
int32_t int32_member;
uint32_t uint32_member;
uint32_t xint32_member;
int64_t int64_member;
uint64_t uint64_member;
uint64_t xint64_member;
intptr_t intptr_member;
uintptr_t uintptr_member;
#ifdef MS_WINDOWS
long long off_member;
#else
off_t off_member;
#endif
pid_t pid_member;
} all_structmembers;

typedef struct {
Expand All @@ -42,12 +63,33 @@ static struct PyMemberDef test_members_newapi[] = {
{"T_LONG", Py_T_LONG, offsetof(test_structmembers, structmembers.long_member), 0, NULL},
{"T_ULONG", Py_T_ULONG, offsetof(test_structmembers, structmembers.ulong_member), 0, NULL},
{"T_PYSSIZET", Py_T_PYSSIZET, offsetof(test_structmembers, structmembers.pyssizet_member), 0, NULL},
{"T_SIZE", Py_T_INTEGER(size_t), offsetof(test_structmembers, structmembers.size_member), 0, NULL},
{"T_FLOAT", Py_T_FLOAT, offsetof(test_structmembers, structmembers.float_member), 0, NULL},
{"T_DOUBLE", Py_T_DOUBLE, offsetof(test_structmembers, structmembers.double_member), 0, NULL},
{"T_STRING_INPLACE", Py_T_STRING_INPLACE, offsetof(test_structmembers, structmembers.inplace_member), 0, NULL},
{"T_LONGLONG", Py_T_LONGLONG, offsetof(test_structmembers, structmembers.longlong_member), 0, NULL},
{"T_ULONGLONG", Py_T_ULONGLONG, offsetof(test_structmembers, structmembers.ulonglong_member), 0, NULL},
{"T_CHAR", Py_T_CHAR, offsetof(test_structmembers, structmembers.char_member), 0, NULL},
{"T_INT8", Py_T_INTEGER(int8_t), offsetof(test_structmembers, structmembers.int8_member), 0, NULL},
{"T_UINT8", Py_T_INTEGER(uint8_t), offsetof(test_structmembers, structmembers.uint8_member), 0, NULL},
{"T_XINT8", Py_T_INTEGER(int8_t)|Py_T_INTEGER(uint8_t), offsetof(test_structmembers, structmembers.xint8_member), 0, NULL},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get the use case of this weird type. If you consider that it's important to support it, can you at least document it?

{"T_INT16", Py_T_INTEGER(int16_t), offsetof(test_structmembers, structmembers.int16_member), 0, NULL},
{"T_UINT16", Py_T_INTEGER(uint16_t), offsetof(test_structmembers, structmembers.uint16_member), 0, NULL},
{"T_XINT16", Py_T_INTEGER(int16_t)|Py_T_INTEGER(uint16_t), offsetof(test_structmembers, structmembers.xint16_member), 0, NULL},
{"T_INT32", Py_T_INTEGER(int32_t), offsetof(test_structmembers, structmembers.int32_member), 0, NULL},
{"T_UINT32", Py_T_INTEGER(uint32_t), offsetof(test_structmembers, structmembers.uint32_member), 0, NULL},
{"T_XINT32", Py_T_INTEGER(int32_t)|Py_T_INTEGER(uint32_t), offsetof(test_structmembers, structmembers.xint32_member), 0, NULL},
{"T_INT64", Py_T_INTEGER(int64_t), offsetof(test_structmembers, structmembers.int64_member), 0, NULL},
{"T_UINT64", Py_T_INTEGER(uint64_t), offsetof(test_structmembers, structmembers.uint64_member), 0, NULL},
{"T_XINT64", Py_T_INTEGER(int64_t)|Py_T_INTEGER(uint64_t), offsetof(test_structmembers, structmembers.xint64_member), 0, NULL},
{"T_INTPTR", Py_T_INTEGER(intptr_t), offsetof(test_structmembers, structmembers.intptr_member), 0, NULL},
{"T_UINTPTR", Py_T_INTEGER(uintptr_t), offsetof(test_structmembers, structmembers.uintptr_member), 0, NULL},
#ifdef MS_WINDOWS
{"T_OFF", Py_T_INTEGER(long long), offsetof(test_structmembers, structmembers.off_member), 0, NULL},
#else
{"T_OFF", Py_T_INTEGER(off_t), offsetof(test_structmembers, structmembers.off_member), 0, NULL},
#endif
{"T_PID", Py_T_INTEGER(pid_t), offsetof(test_structmembers, structmembers.pid_member), 0, NULL},
{NULL}
};

Expand Down
9 changes: 9 additions & 0 deletions Modules/_testcapimodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3272,6 +3272,15 @@ PyInit__testcapi(void)
PyModule_AddObject(m, "SIZEOF_WCHAR_T", PyLong_FromSsize_t(sizeof(wchar_t)));
PyModule_AddObject(m, "SIZEOF_VOID_P", PyLong_FromSsize_t(sizeof(void*)));
PyModule_AddObject(m, "SIZEOF_TIME_T", PyLong_FromSsize_t(sizeof(time_t)));
PyModule_AddObject(m, "SIZEOF_INTMAX_T", PyLong_FromSsize_t(sizeof(intmax_t)));
PyModule_AddObject(m, "SIZEOF_INTPTR_T", PyLong_FromSsize_t(sizeof(intptr_t)));
PyModule_AddObject(m, "SIZEOF_PTRDIFF_T", PyLong_FromSsize_t(sizeof(ptrdiff_t)));
#ifdef MS_WINDOWS
PyModule_AddObject(m, "SIZEOF_OFF_T", PyLong_FromSsize_t(sizeof(long long)));
#else
PyModule_AddObject(m, "SIZEOF_OFF_T", PyLong_FromSsize_t(sizeof(off_t)));
#endif
PyModule_AddObject(m, "SIZEOF_INT", PyLong_FromSsize_t(sizeof(int)));
PyModule_AddObject(m, "SIZEOF_PID_T", PyLong_FromSsize_t(sizeof(pid_t)));
PyModule_AddObject(m, "Py_Version", PyLong_FromUnsignedLong(Py_Version));
Py_INCREF(&PyInstanceMethod_Type);
Expand Down
Loading
Loading