Skip to content

gh-127604: Add C stack dumps to faulthandler #128159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 57 commits into from
Apr 21, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
0ebdc69
Initial implementation for glibc
ZeroIntensity Dec 21, 2024
0ccf9fb
Make it cross-platform buildable.
ZeroIntensity Dec 21, 2024
52945e5
Call it on Windows.
ZeroIntensity Dec 21, 2024
9bdd802
Add a whatsnew entry
ZeroIntensity Dec 21, 2024
66f2641
Add dump_c_stack
ZeroIntensity Dec 21, 2024
0591013
Reverse the loop.
ZeroIntensity Dec 21, 2024
64707a2
Add documentation entries.
ZeroIntensity Dec 21, 2024
8a5b784
Make the whatsnew entry more specific.
ZeroIntensity Dec 21, 2024
4f09358
Add tests.
ZeroIntensity Dec 21, 2024
e1aa619
Add blurb from whatsnew.
ZeroIntensity Dec 21, 2024
6b515d3
Fix failing Sphinx build.
ZeroIntensity Dec 21, 2024
c69ee9d
Fix PyArg_ParseTupleAndKeywords format string.
ZeroIntensity Dec 21, 2024
f08b1dd
Fix Sphinx warnings.
ZeroIntensity Dec 21, 2024
dbb6d25
Add ignore to test_inspect
ZeroIntensity Dec 21, 2024
faf1a3e
Fix name.
ZeroIntensity Dec 21, 2024
ec832aa
Ignore the reentrant variable in the C analyzer.
ZeroIntensity Dec 21, 2024
524f167
Apply suggestions from code review
ZeroIntensity Dec 21, 2024
dd08bcb
Use manpage references.
ZeroIntensity Dec 21, 2024
bda3dcd
Add issue number to whatsnew.
ZeroIntensity Dec 21, 2024
a3564a5
Merge branch 'c-faulthandler' of https://github.com/ZeroIntensity/cpy…
ZeroIntensity Dec 21, 2024
f3fcea1
Rename macros.
ZeroIntensity Dec 21, 2024
db97dd2
Reduce number of printed calls.
ZeroIntensity Dec 21, 2024
c17457f
Apply suggestions from code review
ZeroIntensity Dec 21, 2024
d5f7d4b
Address code review.
ZeroIntensity Dec 21, 2024
e79e661
Merge branch 'c-faulthandler' of https://github.com/ZeroIntensity/cpy…
ZeroIntensity Dec 21, 2024
0c84f8a
Explicitly check for backtrace() and backtrace_symbols()
ZeroIntensity Dec 22, 2024
8198997
Handle no frames returned.
ZeroIntensity Dec 22, 2024
0f670f0
Move the function that calls backtrace() to within the core.
ZeroIntensity Dec 25, 2024
079f186
Merge into main.
ZeroIntensity Jan 26, 2025
892a085
Fix test case.
ZeroIntensity Jan 26, 2025
896abd1
Run make regen-configure
ZeroIntensity Jan 26, 2025
cab079e
Extract behavior in own function.
ZeroIntensity Jan 27, 2025
1784071
PEP-7
ZeroIntensity Jan 27, 2025
3304e2a
Update pycore_traceback.h
ZeroIntensity Feb 4, 2025
aa97f24
Use address_expr
ZeroIntensity Feb 5, 2025
50b4964
Fix wrong argument parsing keycode.
ZeroIntensity Feb 5, 2025
533b1db
Add tests for dump_c_stack()
ZeroIntensity Feb 5, 2025
58b3580
Start homemade implementation of backtrace_symbols()
ZeroIntensity Feb 26, 2025
f62dac8
Merge branch 'main' into c-faulthandler
ZeroIntensity Mar 31, 2025
4feaf09
Fix merge conflicts.
ZeroIntensity Apr 13, 2025
c95369f
Merge branch 'c-faulthandler' of https://github.com/ZeroIntensity/cpy…
ZeroIntensity Apr 13, 2025
95833e7
Untested implementation.
ZeroIntensity Apr 13, 2025
3e2701d
Sorta works.
ZeroIntensity Apr 13, 2025
56127fa
NULL-initialize the arrays.
ZeroIntensity Apr 13, 2025
b70bd43
Use dprintf()
ZeroIntensity Apr 13, 2025
7a070ab
Add proper configure guards.
ZeroIntensity Apr 13, 2025
e9c3d7c
Use faulthandler formatting.
ZeroIntensity Apr 13, 2025
195a539
Protect against compilers without VLAs.
ZeroIntensity Apr 13, 2025
9dd6c3b
Fix incorrect formatting when no symbol name is available.
ZeroIntensity Apr 13, 2025
ce9c39f
Don't use GNU extension.
ZeroIntensity Apr 13, 2025
c344ad7
Handle NULL.
ZeroIntensity Apr 13, 2025
e899792
Fix newlines in the whatsnew.
ZeroIntensity Apr 14, 2025
b136f71
Fix stray newline change.
ZeroIntensity Apr 14, 2025
2c381b9
Add configure guard for dladdr1()
ZeroIntensity Apr 14, 2025
52c0748
Add documentation about compatibility.
ZeroIntensity Apr 14, 2025
bd47026
Fix sphinx role.
ZeroIntensity Apr 14, 2025
07a20d0
Fix styling.
ZeroIntensity Apr 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion Doc/library/faulthandler.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,41 @@ Dumping the traceback
Added support for passing file descriptor to this function.


Dumping the C stack
-------------------

.. versionadded:: next

.. function:: dump_c_stack(file=sys.stderr)

Dump the C stack trace of the current thread into *file*.

If the Python build does not support it or the operating system
does not provide a stack trace, then this prints an error in place
of a dumped C stack.

.. _c-stack-compatibility:

C Stack Compatibility
*********************

If the system does not support the C-level :manpage:`backtrace(3)`,
:manpage:`backtrace_symbols(3)`, or :manpage:`dladdr(3)`, then C stack dumps
will not work. An error will be printed instead of the stack.

Additionally, some compilers do not support :term:`CPython's <CPython>`
implementation of C stack dumps. As a result, a different error may be printed
instead of the stack, even if the the operating system supports dumping stacks.

.. note::

Dumping C stacks can be arbitrarily slow, depending on the DWARF level
of the binaries in the call stack.

Fault handler state
-------------------

.. function:: enable(file=sys.stderr, all_threads=True)
.. function:: enable(file=sys.stderr, all_threads=True, c_stack=True)

Enable the fault handler: install handlers for the :const:`~signal.SIGSEGV`,
:const:`~signal.SIGFPE`, :const:`~signal.SIGABRT`, :const:`~signal.SIGBUS`
Expand All @@ -81,6 +112,10 @@ Fault handler state
The *file* must be kept open until the fault handler is disabled: see
:ref:`issue with file descriptors <faulthandler-fd>`.

If *c_stack* is ``True``, then the C stack trace is printed after the Python
traceback, unless the system does not support it. See :func:`dump_c_stack` for
more information on compatibility.

.. versionchanged:: 3.5
Added support for passing file descriptor to this function.

Expand All @@ -95,6 +130,9 @@ Fault handler state
Only the current thread is dumped if the :term:`GIL` is disabled to
prevent the risk of data races.

.. versionchanged:: next
The dump now displays the C stack trace if *c_stack* is true.

.. function:: disable()

Disable the fault handler: uninstall the signal handlers installed by
Expand Down
9 changes: 9 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -678,6 +678,15 @@ errno
(Contributed by James Roy in :gh:`126585`.)


faulthandler
------------

* Add support for printing the C stack trace on systems that
:ref:`support it <c-stack-compatibility>` via :func:`faulthandler.dump_c_stack`
or via the *c_stack* argument in :func:`faulthandler.enable`.
(Contributed by Peter Bierma in :gh:`127604`.)


fnmatch
-------

Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_faulthandler.h
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ struct _faulthandler_runtime_state {
#ifdef MS_WINDOWS
void *exc_handler;
#endif
int c_stack;
} fatal_error;

struct {
Expand Down
3 changes: 3 additions & 0 deletions Include/internal/pycore_traceback.h
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,9 @@ extern int _PyTraceBack_Print(
extern int _Py_WriteIndentedMargin(int, const char*, PyObject *);
extern int _Py_WriteIndent(int, PyObject *);

// Export for the faulthandler module
PyAPI_FUNC(void) _Py_DumpStack(int fd);

#ifdef __cplusplus
}
#endif
Expand Down
40 changes: 40 additions & 0 deletions Lib/test/test_faulthandler.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,13 @@ def temporary_filename():
finally:
os_helper.unlink(filename)


ADDRESS_EXPR = "0x[0-9a-f]+"
C_STACK_REGEX = [
r"Current thread's C stack trace \(most recent call first\):",
fr'( Binary file ".+"(, at .*(\+|-){ADDRESS_EXPR})? \[{ADDRESS_EXPR}\])|(<.+>)'
]

class FaultHandlerTests(unittest.TestCase):

def get_output(self, code, filename=None, fd=None):
Expand Down Expand Up @@ -103,6 +110,7 @@ def check_error(self, code, lineno, fatal_error, *,
fd=None, know_current_thread=True,
py_fatal_error=False,
garbage_collecting=False,
c_stack=True,
function='<module>'):
"""
Check that the fault handler for fatal errors is enabled and check the
Expand Down Expand Up @@ -134,6 +142,8 @@ def check_error(self, code, lineno, fatal_error, *,
if garbage_collecting and not all_threads_disabled:
regex.append(' Garbage-collecting')
regex.append(fr' File "<string>", line {lineno} in {function}')
if c_stack:
regex.extend(C_STACK_REGEX)
regex = '\n'.join(regex)

if other_regex:
Expand Down Expand Up @@ -950,5 +960,35 @@ def run(self):
_, exitcode = self.get_output(code)
self.assertEqual(exitcode, 0)

def check_c_stack(self, output):
starting_line = output.pop(0)
self.assertRegex(starting_line, C_STACK_REGEX[0])
self.assertGreater(len(output), 0)

for line in output:
with self.subTest(line=line):
Comment on lines +968 to +969
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is totally unrelated, but many times have I been using a construction of the form:

for x1 in x1s:
  ...
  for xn in xns:
      with self.subTest(x1=x1, ..., xn=xn):
          do_test(...)

I'm wondering if it makes sense to have a helper that would be equivalent to the above:

for x1, ..., xn in self.cases(x1=x1s, ..., xn=xns):
	do_test(...)

I'm pretty sure we can use this kind of pattern in the test directory quite a lot, and it can also be useful for other libs, but how about we add this kind of helper to test.support at least? (maybe, let's make it first a mixin class before implementing it on unittest.TestCase)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that could be interesting. What happens when x1s and xns aren't the same length?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, it doesn't matter? we're computing a product not a zip().

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm misunderstanding it. Is:

for a, b in self.cases(a=iter_a, b=iter_b)`:
    print(a, b)

supposed to be equivalent to:

for a in iter_a:
    for b in iter_b:
        print(a, b)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The rationale behind my suggestion is as follows:

for a in A:
	for b in B:
		with self.subTest(a=a, b=b):
			pass

We're already having 3 levels of indentation. Counting the class indentation and method indentation we're just having 20 whitespaces before wer start writing real code...:

class A:
	def f(self):
		for a in A:
			for b in B:
				with self.subTest(a=a, b=b):
					pass

Copy link
Member

@picnixz picnixz Apr 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, if we're pretty short on the keywords passed to subTest() if we want to keep everything under 80 chars (so it's a product() + subTest() combo)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I get it--the a doesn't change with each iteration of cases. I think that could be an interesting addition, but we might also run into PEP-20:

There should be one-- and preferably only one --obvious way to do it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One obvious way, probably but this obvious way makes the code really harder to read IMO. Anyway, this is a bit off-topic and I don't know how much we can use it (but I've personally needed more than once to be able to use products for testing cases).

Note that importlib's tests already has some "parametrize" decorator as in Pytest, so we could also move it up to test.support instead.

if line != '': # Ignore trailing or leading newlines
self.assertRegex(line, C_STACK_REGEX[1])


def test_dump_c_stack(self):
code = dedent("""
import faulthandler
faulthandler.dump_c_stack()
""")
output, exitcode = self.get_output(code)
self.assertEqual(exitcode, 0)
self.check_c_stack(output)


def test_dump_c_stack_file(self):
import tempfile

with tempfile.TemporaryFile("w+") as tmp:
faulthandler.dump_c_stack(file=tmp)
tmp.flush() # Just in case
tmp.seek(0)
self.check_c_stack(tmp.read().split("\n"))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we using two blank lines to separate classes in this file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like there are any other classes here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, most of the file is using 1-blank line. Up to you then.

if __name__ == "__main__":
unittest.main()
2 changes: 1 addition & 1 deletion Lib/test/test_inspect/test_inspect.py
Original file line number Diff line number Diff line change
Expand Up @@ -5760,7 +5760,7 @@ def test_errno_module_has_signatures(self):

def test_faulthandler_module_has_signatures(self):
import faulthandler
unsupported_signature = {'dump_traceback', 'dump_traceback_later', 'enable'}
unsupported_signature = {'dump_traceback', 'dump_traceback_later', 'enable', 'dump_c_stack'}
unsupported_signature |= {name for name in ['register']
if hasattr(faulthandler, name)}
self._test_module_has_signatures(faulthandler, unsupported_signature=unsupported_signature)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Add support for printing the C stack trace on systems that support it via
:func:`faulthandler.dump_c_stack` or via the *c_stack* argument in
:func:`faulthandler.enable`.
60 changes: 57 additions & 3 deletions Modules/faulthandler.c
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@
#include "pycore_sysmodule.h" // _PySys_GetRequiredAttr()
#include "pycore_time.h" // _PyTime_FromSecondsObject()
#include "pycore_traceback.h" // _Py_DumpTracebackThreads

#ifdef HAVE_UNISTD_H
# include <unistd.h> // _exit()
#endif

#include <signal.h> // sigaction()
#include <stdlib.h> // abort()
#if defined(HAVE_PTHREAD_SIGMASK) && !defined(HAVE_BROKEN_PTHREAD_SIGMASK) && defined(HAVE_PTHREAD_H)
Expand Down Expand Up @@ -210,6 +210,25 @@ faulthandler_dump_traceback(int fd, int all_threads,
reentrant = 0;
}

static void
faulthandler_dump_c_stack(int fd)
{
static volatile int reentrant = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it is better to use atomics here? AFAIU volatile have different behavior for msvc for different platforms.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be, but regardless, that should be done in a different PR.


if (reentrant) {
return;
}

reentrant = 1;

if (fatal_error.c_stack) {
PUTS(fd, "\n");
_Py_DumpStack(fd);
}

reentrant = 0;
}

static PyObject*
faulthandler_dump_traceback_py(PyObject *self,
PyObject *args, PyObject *kwargs)
Expand Down Expand Up @@ -260,6 +279,33 @@ faulthandler_dump_traceback_py(PyObject *self,
Py_RETURN_NONE;
}

static PyObject *
faulthandler_dump_c_stack_py(PyObject *self,
PyObject *args, PyObject *kwargs)
{
static char *kwlist[] = {"file", NULL};
PyObject *file = NULL;

if (!PyArg_ParseTupleAndKeywords(args, kwargs,
"|O:dump_c_stack", kwlist,
&file)) {
return NULL;
}

int fd = faulthandler_get_fileno(&file);
if (fd < 0) {
return NULL;
}

_Py_DumpStack(fd);

if (PyErr_CheckSignals()) {
return NULL;
}

Py_RETURN_NONE;
}

static void
faulthandler_disable_fatal_handler(fault_handler_t *handler)
{
Expand Down Expand Up @@ -350,6 +396,7 @@ faulthandler_fatal_error(int signum)

faulthandler_dump_traceback(fd, deduce_all_threads(),
fatal_error.interp);
faulthandler_dump_c_stack(fd);

_Py_DumpExtensionModules(fd, fatal_error.interp);

Expand Down Expand Up @@ -425,6 +472,7 @@ faulthandler_exc_handler(struct _EXCEPTION_POINTERS *exc_info)

faulthandler_dump_traceback(fd, deduce_all_threads(),
fatal_error.interp);
faulthandler_dump_c_stack(fd);

/* call the next exception handler */
return EXCEPTION_CONTINUE_SEARCH;
Expand Down Expand Up @@ -519,14 +567,15 @@ faulthandler_enable(void)
static PyObject*
faulthandler_py_enable(PyObject *self, PyObject *args, PyObject *kwargs)
{
static char *kwlist[] = {"file", "all_threads", NULL};
static char *kwlist[] = {"file", "all_threads", "c_stack", NULL};
PyObject *file = NULL;
int all_threads = 1;
int fd;
int c_stack = 1;
PyThreadState *tstate;

if (!PyArg_ParseTupleAndKeywords(args, kwargs,
"|Op:enable", kwlist, &file, &all_threads))
"|Opp:enable", kwlist, &file, &all_threads, &c_stack))
return NULL;

fd = faulthandler_get_fileno(&file);
Expand All @@ -543,6 +592,7 @@ faulthandler_py_enable(PyObject *self, PyObject *args, PyObject *kwargs)
fatal_error.fd = fd;
fatal_error.all_threads = all_threads;
fatal_error.interp = PyThreadState_GetInterpreter(tstate);
fatal_error.c_stack = c_stack;

if (faulthandler_enable() < 0) {
return NULL;
Expand Down Expand Up @@ -1238,6 +1288,10 @@ static PyMethodDef module_methods[] = {
PyDoc_STR("dump_traceback($module, /, file=sys.stderr, all_threads=True)\n--\n\n"
"Dump the traceback of the current thread, or of all threads "
"if all_threads is True, into file.")},
{"dump_c_stack",
_PyCFunction_CAST(faulthandler_dump_c_stack_py), METH_VARARGS|METH_KEYWORDS,
PyDoc_STR("dump_c_stack($module, /, file=sys.stderr)\n--\n\n"
"Dump the C stack of the current thread.")},
{"dump_traceback_later",
_PyCFunction_CAST(faulthandler_dump_traceback_later), METH_VARARGS|METH_KEYWORDS,
PyDoc_STR("dump_traceback_later($module, /, timeout, repeat=False, file=sys.stderr, exit=False)\n--\n\n"
Expand Down
Loading
Loading