Skip to content

Expose functions called from the interpreter loop via PyAPI_FUNC #131776

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
martindemello opened this issue Mar 26, 2025 · 7 comments
Open

Expose functions called from the interpreter loop via PyAPI_FUNC #131776

martindemello opened this issue Mar 26, 2025 · 7 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-C-API topic-JIT type-feature A feature request or enhancement

Comments

@martindemello
Copy link
Contributor

Rationale: JIT code often needs access to functions called from the interpreter loop (in ceval.c or bytecodes.c), even if they are technically internal functions. gh-115802 is a good example - it added PyAPI_FUNC to a lot of functions in Include/internal.

Cinder currently relies on a fork of 3.12 with a lot of new functions exposed; some but not all of these have also been exposed in 3.13+. As a start, I would like to send in a PR for the functions we have already needed to expose for cinder (e.g. _PyNumber_InPlacePowerNoMod), but more generally it might be useful to look at all the Py* and _Py* functions used directly in ceval or bytecodes and consider exposing them even if no specific JIT has needed them yet.

@tomasr8 tomasr8 added interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-JIT labels Mar 26, 2025
@brandtbucher
Copy link
Member

more generally it might be useful to look at all the Py* and _Py* functions used directly in ceval or bytecodes and consider exposing them even if no specific JIT has needed them yet.

Eh, I'm inclined to wait until people ask for them. Just exposing a bunch of internal, unstable functions with no users doesn't make much sense to me. At best, nobody will actually use them. At worst, people will start using them. :)

@ZeroIntensity
Copy link
Member

If anything, it would be good to expose things in the unstable API rather than the private API.

@vstinner
Copy link
Member

we have already needed to expose for cinder (e.g. _PyNumber_InPlacePowerNoMod)

If you have a list of functions that would be useful to Cinder, please give the list :-) We may expose these functions in the unstable API, as @ZeroIntensity suggests.

@martindemello
Copy link
Contributor Author

here's the list of functions where (a) cinder exposed them as PyAPI_FUNC in 3.12, (b) cpython has not exposed them in main, and (c) the function still exists:

Include/internal/pycore_pyerrors.h
   _PyErr_GetRaisedException
   _PyErr_SetRaisedException

Include/internal/pycore_typeobject.h
   _PyStaticType_GetState

Include/internal/pycore_instruments.h
   _Py_call_instrumentation
   _Py_call_instrumentation_line
   _Py_call_instrumentation_instruction
   _Py_call_instrumentation_jump
   _Py_call_instrumentation_arg
   _Py_call_instrumentation_2args
   _Py_call_instrumentation_exc2

Include/internal/pycore_genobject.h
   _PyAsyncGenValueWrapperNew

Include/internal/pycore_ceval.h
   _Py_CheckRecursiveCallPy

Include/internal/pycore_dict.h
   _PyObjectDict_SetItem

Include/internal/pycore_abstract.h
   _PyNumber_PowerNoMod
   _PyNumber_InPlacePowerNoMod

Include/internal/pycore_call.h
   _PyStack_UnpackDict
   _PyStack_UnpackDict_FreeNoDecRef

Include/internal/pycore_code.h
   _PyCode_InitAddressRange
   _PyLineTable_NextAddressRange
   _Py_Specialize_LoadSuperAttr
   _Py_Specialize_LoadAttr
   _Py_Specialize_StoreAttr
   _Py_Specialize_LoadGlobal
   _Py_Specialize_StoreSubscr
   _Py_Specialize_Call
   _Py_Specialize_BinaryOp
   _Py_Specialize_CompareOp
   _Py_Specialize_UnpackSequence
   _Py_Specialize_ForIter
   _Py_Specialize_Send
   _Py_Instrument

@picnixz picnixz added the type-feature A feature request or enhancement label Apr 25, 2025
@vstinner
Copy link
Member

Hum, most of these functions are low-level and the internal C API is the right place for them. I'm not sure about adding PyAPI_FUNC() to them.

For now, I would suggest to continue maintaining downstream patches on the Cinder fork to add these PyAPI_FUNC().

I'm not comfortable to add these functions to the public C API.

_PyErr_GetRaisedException
_PyErr_SetRaisedException

Compared to PyErr_GetRaisedException() and PyErr_SetRaisedException(), passing tstate is an optimization. So far, we didn't expose "Tstate" variants to the public C API. There are reserved to the internal C API such as _PyErr_Occurred() (implemented as a static inline function).

_PyStaticType_GetState

I would prefer to not expose managed_static_type_state structure which is low-level.

_PyNumber_PowerNoMod
_PyNumber_InPlacePowerNoMod

You may reimplement these functions in Cinder, there are trivial:

PyObject *
_PyNumber_PowerNoMod(PyObject *lhs, PyObject *rhs)
{   
    return PyNumber_Power(lhs, rhs, Py_None);
}   

PyObject *
_PyNumber_InPlacePowerNoMod(PyObject *lhs, PyObject *rhs)
{   
    return PyNumber_InPlacePower(lhs, rhs, Py_None);
}   

@brandtbucher
Copy link
Member

Just to be clear, these would remain in the internal C API. PyAPI_FUNC would just be added so it's at least possible to link against them for projects that want to include the internal headers.

@martindemello
Copy link
Contributor Author

yes, all we need is to be able to link against it, not any sort of stability guarantees. I like the idea of an alias of PyAPI_FUNC that makes this explicit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-C-API topic-JIT type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

6 participants