Skip to content

sys.setprofile does not dipatch C Extension dunder methods #134243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Vipul-Cariappa opened this issue May 19, 2025 · 3 comments
Closed

sys.setprofile does not dipatch C Extension dunder methods #134243

Vipul-Cariappa opened this issue May 19, 2025 · 3 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@Vipul-Cariappa
Copy link

I am trying to log all the function calls that happen. Within Python, as well as any of the C Extensions. But I see that dunder/magic methods defined within the C extensions are not registered with sys.setprofile.
Is this the expected behaviour? If yes, are there any other means for me to log such events?
Or is this a bug in CPython?

Code to reproduce:

// c_extensions.c
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <stddef.h> /* for offsetof() */

typedef struct {
    PyObject_HEAD PyObject *first; /* first name */
    PyObject *last;                /* last name */
    int number;
} CustomObject;

static void Custom_dealloc(CustomObject *self) {
    Py_XDECREF(self->first);
    Py_XDECREF(self->last);
    Py_TYPE(self)->tp_free((PyObject *)self);
}

static PyObject *Custom_new(PyTypeObject *type, PyObject *args,
                            PyObject *kwds) {
    CustomObject *self;
    self = (CustomObject *)type->tp_alloc(type, 0);
    if (self != NULL) {
        self->first = PyUnicode_FromString("");
        if (self->first == NULL) {
            Py_DECREF(self);
            return NULL;
        }
        self->last = PyUnicode_FromString("");
        if (self->last == NULL) {
            Py_DECREF(self);
            return NULL;
        }
        self->number = 0;
    }
    return (PyObject *)self;
}

static int Custom_init(CustomObject *self, PyObject *args, PyObject *kwds) {
    static char *kwlist[] = {"first", "last", "number", NULL};
    PyObject *first = NULL, *last = NULL;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOi", kwlist, &first, &last,
                                     &self->number))
        return -1;

    if (first) {
        Py_XSETREF(self->first, Py_NewRef(first));
    }
    if (last) {
        Py_XSETREF(self->last, Py_NewRef(last));
    }
    return 0;
}

static PyMemberDef Custom_members[] = {
    {"first", Py_T_OBJECT_EX, offsetof(CustomObject, first), 0, "first name"},
    {"last", Py_T_OBJECT_EX, offsetof(CustomObject, last), 0, "last name"},
    {"number", Py_T_INT, offsetof(CustomObject, number), 0, "custom number"},
    {NULL} /* Sentinel */
};

static PyObject *Custom_name(CustomObject *self, PyObject *Py_UNUSED(ignored)) {
    if (self->first == NULL) {
        PyErr_SetString(PyExc_AttributeError, "first");
        return NULL;
    }
    if (self->last == NULL) {
        PyErr_SetString(PyExc_AttributeError, "last");
        return NULL;
    }
    return PyUnicode_FromFormat("%S %S", self->first, self->last);
}

static PyMethodDef Custom_methods[] = {
    {"name", (PyCFunction)Custom_name, METH_NOARGS,
     "Return the name, combining the first and last name"},
    {NULL} /* Sentinel */
};

static PyTypeObject CustomType = {
    .ob_base = PyVarObject_HEAD_INIT(NULL, 0).tp_name = "custom2.Custom",
    .tp_doc = PyDoc_STR("Custom objects"),
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
    .tp_new = Custom_new,
    .tp_init = (initproc)Custom_init,
    .tp_dealloc = (destructor)Custom_dealloc,
    .tp_members = Custom_members,
    .tp_methods = Custom_methods,
};

static PyObject *print_stderr(PyObject *self, PyObject *args) {
    const char *command;
    int sts;

    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
    sts = fprintf(stderr, "%s\n", command);
    return PyLong_FromLong(sts);
}

static PyMethodDef customMethods[] = {
    {"print_stderr", print_stderr, METH_VARARGS, "Prints."},
    {NULL, NULL, 0, NULL},
};

static PyModuleDef custommodule = {
    .m_base = PyModuleDef_HEAD_INIT,
    .m_name = "custom2",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
    .m_methods = customMethods,
};

PyMODINIT_FUNC PyInit_custom2(void) {
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0)
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    if (PyModule_AddObjectRef(m, "Custom", (PyObject *)&CustomType) < 0) {
        Py_DECREF(m);
        return NULL;
    }

    return m;
}
# main.py
import sys
import custom2  # C Extension


class MyClass:
    def __init__(self, *args, **kwargs):
        pass


def dispatch_handler(frame, event, arg):
    if event.startswith("c_"):
        print(f"{event = } name = {arg.__name__}")
    else:
        print(f"{event = } {frame = }")

    return dispatch_handler


def test_dummy():
    custom2.print_stderr("Hello World")

    c = custom2.Custom(19)  # XXX: Does not dispatch an event for custom2.Custom.__init__
    custom2.print_stderr(c.name())

    c = MyClass()  # XXX: Dispatches an event for MyClass.__init__


if __name__ == "__main__":
    sys.setprofile(dispatch_handler)
    test_dummy()
# setup.py to compile the C Extension
# ???: use pyproject.toml instead

from setuptools import setup, Extension

module = Extension("custom2", sources=["c_extension.c"])

setup(
    name="custom2",
    version="0.0.1",
    description="Python C Extension",
    ext_modules=[module],
)
❯ python ./main.py > output.txt
Hello World
19
event = 'call' frame = <frame at 0x712491d45b40, file '/home/vipul-cariappa/Documents/Workspace/python/debugger/pxc-dbg/tmp/example0/./main.py', line 19, code test_dummy>
event = 'c_call' name = print_stderr
event = 'c_return' name = print_stderr
event = 'c_call' name = name
event = 'c_return' name = name
event = 'c_call' name = print_stderr
event = 'c_return' name = print_stderr
event = 'call' frame = <frame at 0x712491d46740, file '/home/vipul-cariappa/Documents/Workspace/python/debugger/pxc-dbg/tmp/example0/./main.py', line 6, code __init__>
event = 'return' frame = <frame at 0x712491d46740, file '/home/vipul-cariappa/Documents/Workspace/python/debugger/pxc-dbg/tmp/example0/./main.py', line 7, code __init__>
event = 'return' frame = <frame at 0x712491d45b40, file '/home/vipul-cariappa/Documents/Workspace/python/debugger/pxc-dbg/tmp/example0/./main.py', line 25, code test_dummy>
event = 'return' frame = <frame at 0x712491d45b40, file '/home/vipul-cariappa/Documents/Workspace/python/debugger/pxc-dbg/tmp/example0/./main.py', line 30, code <module>>
@picnixz picnixz added type-bug An unexpected behavior, bug, or error interpreter-core (Objects, Python, Grammar, and Parser dirs) labels May 19, 2025
@gaogaotiantian
Copy link
Member

Did you mean your Custom_init function? That won't be recorded in any way CPython provides. It's just not instrumented. Also that's not really a "magic method". First of all, there's no dunder in it (LOL). That's the deeper layer of mechanism where C calls into it without any instrumentation, so the monitoring mechanism is not aware of it.

CPython's monitoring mechanism mostly monitors on Python-level events (even for c calls/returns, it's the call from Python into C, not all the C function calls), if you want information on C-level events, you should need a C profiler/tracer.

@Vipul-Cariappa
Copy link
Author

Did you mean your Custom_init function?

Yes.

That won't be recorded in any way CPython provides. That's the deeper layer of mechanism where C calls into it without any instrumentation, so the monitoring mechanism is not aware of it.

Are there any plans to implement such instrumentation? If yes, I can work on it in my free time.

Reason why I need this:
I am trying to build a cross-language debugger to debug Python C Extensions. And I have got almost everything I initially planned, to work, except for the magic methods.
Link to the debugger: https://github.com/Vipul-Cariappa/pxc-dbg
Demo:

Screencast.From.2025-05-19.20-15-25.mp4

I am open to discussion. And am in need of help and suggestions.

@gaogaotiantian
Copy link
Member

I don't believe there is such a plan. I believe all of our existing monitoring mechanism is based on bytecode transformation - basically we hook a callback on some specific bytecode. However, this method does not trigger any bytecode - again, this is not a magic method. A magic method (special method) is clearly explained here. You are overriding a C-API level function of a PyObject and it won't triger anything from Python monitoring.

@gaogaotiantian gaogaotiantian closed this as not planned Won't fix, can't repro, duplicate, stale Jun 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants