Skip to content

gh-137210: Add a struct, slot & function for checking an extension's ABI #137212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Doc/c-api/module.rst
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,28 @@ The available slot types are:

.. versionadded:: 3.13

.. c:macro:: Py_mod_abi

A pointer to a :c:struct:`PyABIInfo` structure that describes the ABI that
the extension is using.

When the module is loaded, the :c:struct:`!PyABIInfo` in this slot is checked
using :c:func:`PyABIInfo_Check`.

A suitable :c:struct:`!PyABIInfo` variable can be defined using the
:c:macro:`PyABIInfo_VAR` macro, as in:

.. code-block:: c

PyABIInfo_VAR(abi_info);

static PyModuleDef_Slot mymodule_slots[] = {
{Py_mod_abi, &abi_info},
...
};

.. versionadded:: 3.15


.. _moduledef-dynamic:

Expand Down
151 changes: 148 additions & 3 deletions Doc/c-api/stable.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

.. _stable:

***************
C API Stability
***************
***********************
C API and ABI Stability
***********************

Unless documented otherwise, Python's C API is covered by the Backwards
Compatibility Policy, :pep:`387`.
Expand Down Expand Up @@ -199,6 +199,151 @@
third-party distributors.


ABI Checking
============

.. versionadded:: next

Python includes a rudimentary check for ABI compatibility.

This check is not comprehensive.
It only guards against common cases of incompatible modules being
installed for the wrong interpreter.
It also does not take :ref:`platform incompatibilities <stable-abi-platform>`
into account.
It can only be done after an extension is successfully loaded.

Despite these limitations, it is recommended that extension modules use this
mechanism, so that detectable incompatibilities raise exceptions rather than
crash.

Most modules can use this check via the :c:data:`Py_mod_abi`
slot and the :c:macro:`PyABIInfo_VAR` macro.
The full API is described below for advanced use cases.

.. c:function:: int PyABIInfo_Check(PyABIInfo *info, const char *module_name)

Verify that the given *info* is compatible with the currently running
interpreter.

Return 0 on success. On failure, raise an exception and return -1.

If the ABI is incompatible, the raised exception will be :py:exc:`ImportError`.

The *module_name* argument can be ``NULL``, or point to a NUL-terminated
UTF-8-encoded string used for error messages.

Note that if *info* describes the ABI that the current code uses (as defined
by :c:macro:`PyABIInfo_VAR`, for example), using any other Python C API
may lead to crashes.
In particular, it is not safe to examine the raised exception.

.. versionadded:: next

.. c:macro:: PyABIInfo_VAR(NAME)

Define a static :c:struct:`PyABIInfo` variable with the given *NAME* that
describes the ABI that the current code will use.
This macro expands to:

.. code-block:: c

static PyABIInfo NAME = {
1, 0,
PyABIInfo_DEFAULT_FLAGS,
PY_VERSION_HEX,
PyABIInfo_DEFAULT_ABI_VERSION
}

.. versionadded:: next

.. c:type:: PyABIInfo

.. c:member:: uint8_t abiinfo_major_version

The major version of :c:struct:`PyABIInfo`. Can be set to:

* ``0`` to skip all checking, or
* ``1`` to specify this version of :c:struct:`!PyABIInfo`.

.. c:member:: uint8_t abiinfo_minor_version

The major version of :c:struct:`PyABIInfo`.
Must be set to ``0``; larger values are reserved for backwards-compatible
future versions of :c:struct:`!PyABIInfo`.

.. c:member:: uint16_t flags

.. c:namespace:: NULL

This field is usually set to the following macro:

.. c:macro:: PyABIInfo_DEFAULT_FLAGS

Default flags, based on current values of macros such as
:c:macro:`Py_LIMITED_API` and :c:macro:`Py_GIL_DISABLED`.

Alternately, the field can be set to to the following flags, combined

Check warning on line 286 in Doc/c-api/stable.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

c:data reference target not found: PyFilter_Type [ref.data]
by bitwise OR.
Unused bits must be set to zero.

ABI variant -- one of:

.. c:macro:: PyABIInfo_STABLE

Specifies that the stable ABI is used.

.. c:macro:: PyABIInfo_INTERNAL

Specifies ABI specific to a particular build of CPython.
Internal use only.

Free-threading compatibility -- one of:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're still planning to get rid of these options once FT is merged, right? Maybe we ought to use abiflags here (i.e. a string) instead, for future extensibility? This isn't going to be checked anywhere that it needs to be bitflags.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would you encode being compatible with both GIL and FT?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, we can remove them when there are no GIL-only extensions around any more. That would be a lot of time after FT is the only option.


.. c:macro:: PyABIInfo_FREETHREADED

Specifies ABI compatible with free-threading builds of CPython.
(That is, ones compiled with :option:`--disable-gil`; with ``t``

Check warning on line 306 in Doc/c-api/stable.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

c:type reference target not found: PyGILState_STATE [ref.type]
in :py:data:`sys.abiflags`)

Check warning on line 308 in Doc/c-api/stable.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

c:data reference target not found: PyGetSetDescr_Type [ref.data]
.. c:macro:: PyABIInfo_GIL

Specifies ABI compatible with non-free-threading builds of CPython
(ones compiled *without* :option:`--disable-gil`).

.. c:member:: uint32_t build_version

The version of the Python headers used to build the code, in the format
used by :c:macro:`PY_VERSION_HEX`.

This can be set to ``0`` to skip any checks related to this field.
This option is meant mainly for projects that do not use the CPython
headers directly, and do not emulate a specific version of them.

.. c:member:: uint32_t abi_version

The ABI version.

For the Stable ABI, this field should be the value of
:c:macro:`Py_LIMITED_API`
(except if :c:macro:`Py_LIMITED_API` is ``3``; use
:c:expr:`Py_PACK_VERSION(3, 2)` in that case).
Comment on lines +329 to +330
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems easier for us to just check this on our side than to expect users to get it right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And isn't the legacy value 1 rather than 3?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, users should use PyABIInfo_DEFAULT_ABI_VERSION or PyABIInfo_VAR -- those are our side too.
I prefer the data to be simpler, but if you want, we can keep the exception in.

The legacy value is 3. (Not that it matters that much; it's always compared to a Python version.)


Otherwise, it should be set to :c:macro:`PY_VERSION_HEX`.

It can also be set to ``0`` to skip any checks related to this field.

.. c:namespace:: NULL

.. c:macro:: PyABIInfo_DEFAULT_ABI_VERSION

The value that should be used for this field, based on current
values of macros such as :c:macro:`Py_LIMITED_API`,

Check warning on line 341 in Doc/c-api/stable.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

c:data reference target not found: PyListIter_Type [ref.data]
:c:macro:`PY_VERSION_HEX` and :c:macro:`Py_GIL_DISABLED`.

Check warning on line 342 in Doc/c-api/stable.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs

c:data reference target not found: PyListRevIter_Type [ref.data]

.. versionadded:: next


.. _limited-api-list:

Contents of Limited API
Expand Down
3 changes: 3 additions & 0 deletions Doc/data/stable_abi.dat

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Doc/using/configure.rst
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,9 @@ General Options

.. option:: --disable-gil

.. c:macro:: Py_GIL_DISABLED
:no-typesetting:

Enables support for running Python without the :term:`global interpreter
lock` (GIL): free threading build.

Expand Down
5 changes: 5 additions & 0 deletions Doc/whatsnew/3.15.rst
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,11 @@ New features
a string. See the documentation for caveats.
(Contributed by Petr Viktorin in :gh:`131510`)

* Add API for checking an extension module's ABI compatibility:
:c:data:`Py_mod_abi`, :c:func:`PyABIInfo_Check`, :c:macro:`PyABIInfo_VAR`
and :c:data:`Py_mod_abi`.
(Contributed by Petr Viktorin in :gh:`137210`)


Porting to Python 3.15
----------------------
Expand Down
12 changes: 12 additions & 0 deletions Include/cpython/modsupport.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,15 @@ typedef struct _PyArg_Parser {

PyAPI_FUNC(int) _PyArg_ParseTupleAndKeywordsFast(PyObject *, PyObject *,
struct _PyArg_Parser *, ...);

#ifdef Py_BUILD_CORE
// Internal; defined here to avoid explicitly including pycore_modsupport.h
#define _Py_INTERNAL_ABI_SLOT \
{Py_mod_abi, (void*) &(PyABIInfo) { \
.abiinfo_major_version = 1, \
.abiinfo_minor_version = 0, \
.flags = PyABIInfo_INTERNAL, \
.build_version = PY_VERSION_HEX, \
.abi_version = PY_VERSION_HEX }} \
///////////////////////////////////////////////////////
#endif
6 changes: 6 additions & 0 deletions Include/internal/pycore_unicodeobject.h
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,12 @@ extern int _PyUnicode_FormatAdvancedWriter(
Py_ssize_t start,
Py_ssize_t end);

/* PyUnicodeWriter_Format, with va_list instead of `...` */
extern int _PyUnicodeWriter_FormatV(
PyUnicodeWriter *writer,
const char *format,
va_list vargs);

/* --- UTF-7 Codecs ------------------------------------------------------- */

extern PyObject* _PyUnicode_EncodeUTF7(
Expand Down
119 changes: 72 additions & 47 deletions Include/modsupport.h
Original file line number Diff line number Diff line change
Expand Up @@ -56,55 +56,14 @@ PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def);

#define Py_CLEANUP_SUPPORTED 0x20000

/* The API and ABI versions are left for backwards compatibility.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we just... update them?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To what? Py_VERSION_HEX? Or tell the RM to increment on release?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it particularly matters, it could be inferred from Py_LIMITED_API if that's set.

It's just an arbitrary value with a purpose that isn't really going away, so deprecating them seems unnecessary when we could just keep using them for their intended purposes (and ignore that they didn't get updated for a large number of ABI changes).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, do you want to use an int instead of the PyABIInfo struct?

I don't think reusing these would buy us much.
These version numbers are currently used in PyModule_Create and PyModule_FromDefAndSpec, i.e. single-phase init (legacy) and manual module object creation (where version checking isn't that useful).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or just use it as one of the values that goes into the struct. We can include more information if needed, it just seems like a waste to declare these names as irrelevant when we could just as easily make them relevant.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make them relevant, but what relevance should they have?
We switched to a versioning scheme where Py_VERSION_HEX gives you all the necessary info. Dropping these extra numbers is a win.

If we can find a use case in the future, that's what the versioning in PyABIInfo is for.

They've not been updated since 2006 and 2010, respectively.
API/ABI versioning is now tied to the CPython version.
The *_VERSION and *_STRING symbols should define the same value; as
number and string literal respectively. Make sure the definitions match.
*/
#define PYTHON_API_VERSION 1013
#define PYTHON_API_STRING "1013"
/* The API version is maintained (independently from the Python version)
so we can detect mismatches between the interpreter and dynamically
loaded modules. These are diagnosed by an error message but
the module is still loaded (because the mismatch can only be tested
after loading the module). The error message is intended to
explain the core dump a few seconds later.

The symbol PYTHON_API_STRING defines the same value as a string
literal. *** PLEASE MAKE SURE THE DEFINITIONS MATCH. ***

Please add a line or two to the top of this log for each API
version change:

22-Feb-2006 MvL 1013 PEP 353 - long indices for sequence lengths

19-Aug-2002 GvR 1012 Changes to string object struct for
interning changes, saving 3 bytes.

17-Jul-2001 GvR 1011 Descr-branch, just to be on the safe side

25-Jan-2001 FLD 1010 Parameters added to PyCode_New() and
PyFrame_New(); Python 2.1a2

14-Mar-2000 GvR 1009 Unicode API added

3-Jan-1999 GvR 1007 Decided to change back! (Don't reuse 1008!)

3-Dec-1998 GvR 1008 Python 1.5.2b1

18-Jan-1997 GvR 1007 string interning and other speedups

11-Oct-1996 GvR renamed Py_Ellipses to Py_Ellipsis :-(

30-Jul-1996 GvR Slice and ellipses syntax added

23-Jul-1996 GvR For 1.4 -- better safe than sorry this time :-)

7-Nov-1995 GvR Keyword arguments (should've been done at 1.3 :-( )

10-Jan-1995 GvR Renamed globals to new naming scheme

9-Jan-1995 GvR Initial version (incompatible with older API)
*/

/* The PYTHON_ABI_VERSION is introduced in PEP 384. For the lifetime of
Python 3, it will stay at the value of 3; changes to the limited API
must be performed in a strictly backwards-compatible manner. */
#define PYTHON_ABI_VERSION 3
#define PYTHON_ABI_STRING "3"

Expand Down Expand Up @@ -134,6 +93,72 @@ PyAPI_FUNC(PyObject *) PyModule_FromDefAndSpec2(PyModuleDef *def,

#endif /* New in 3.5 */

/* ABI info & checking (new in 3.15) */
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030f0000
typedef struct PyABIInfo {
uint8_t abiinfo_major_version;
uint8_t abiinfo_minor_version;
uint16_t flags;
uint32_t build_version;
uint32_t abi_version;
} PyABIInfo;
#define PyABIInfo_STABLE 0x0001
#define PyABIInfo_GIL 0x0002
#define PyABIInfo_FREETHREADED 0x0004
#define PyABIInfo_INTERNAL 0x0008

#define PyABIInfo_FREETHREADING_AGNOSTIC (PyABIInfo_GIL|PyABIInfo_FREETHREADED)

PyAPI_FUNC(int) PyABIInfo_Check(PyABIInfo *info, const char *module_name);

// Define the defaults
#ifdef Py_LIMITED_API
#define _PyABIInfo_DEFAULT_FLAG_STABLE PyABIInfo_STABLE
#if Py_LIMITED_API == 3
#define PyABIInfo_DEFAULT_ABI_VERSION _Py_PACK_VERSION(3, 2)
#else
#define PyABIInfo_DEFAULT_ABI_VERSION Py_LIMITED_API
#endif
#else
#define _PyABIInfo_DEFAULT_FLAG_STABLE 0
#define PyABIInfo_DEFAULT_ABI_VERSION PY_VERSION_HEX
#endif
#if defined(Py_LIMITED_API) && defined(_Py_OPAQUE_PYOBJECT)
#define _PyABIInfo_DEFAULT_FLAG_FT PyABIInfo_FREETHREADING_AGNOSTIC
#elif defined(Py_GIL_DISABLED)
#define _PyABIInfo_DEFAULT_FLAG_FT PyABIInfo_FREETHREADED
#else
#define _PyABIInfo_DEFAULT_FLAG_FT PyABIInfo_GIL
#endif
#if defined(Py_BUILD_CORE)
#define _PyABIInfo_DEFAULT_FLAG_INTERNAL PyABIInfo_INTERNAL
#else
#define _PyABIInfo_DEFAULT_FLAG_INTERNAL 0
#endif

#define PyABIInfo_DEFAULT_FLAGS ( \
_PyABIInfo_DEFAULT_FLAG_STABLE \
| _PyABIInfo_DEFAULT_FLAG_FT \
| _PyABIInfo_DEFAULT_FLAG_INTERNAL \
) \
/////////////////////////////////////////////////////////

#define _PyABIInfo_DEFAULT() { \
1, 0, \
PyABIInfo_DEFAULT_FLAGS, \
PY_VERSION_HEX, \
PyABIInfo_DEFAULT_ABI_VERSION } \
/////////////////////////////////////////////////////////

#define PyABIInfo_VAR(NAME) \
static PyABIInfo NAME = _PyABIInfo_DEFAULT;

#undef _PyABIInfo_DEFAULT_STABLE
#undef _PyABIInfo_DEFAULT_FT
#undef _PyABIInfo_DEFAULT_INTERNAL

#endif /* ABI info (new in 3.15) */

#ifndef Py_LIMITED_API
# define Py_CPYTHON_MODSUPPORT_H
# include "cpython/modsupport.h"
Expand Down
Loading
Loading