Description
Bug report
Bug description:
Hi all, congrats on the new Python release, very keen on the direction that sub-interpreters are taking!
I noticed that constructing and destructing sub-interpreters now causes memory to be leaked. This was not a problem in 3.11, but is as of 3.12 and occurs on latest main branch too (as of 8c07137). I figure it must relate to the changes made to split up the interpreter states.
I can consistently replicate the issue using simply:
import _xxsubinterpreters
while 1:
interp = _xxsubinterpreters.create()
_xxsubinterpreters.destroy(interp)You'll notice the memory usage climb quite quickly if you look at top/htop.
valgrind/massif think the culprit are interned strings, specifically here:
==219777== at 0x4848A13: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==219777== by 0x2DA59E: PyMem_RawCalloc (obmalloc.c:671)
==219777== by 0x2DA59E: arena_map_get (obmalloc.c:1021)
==219777== by 0x2DA59E: arena_map_mark_used (obmalloc.c:1076)
==219777== by 0x2DAA71: new_arena (obmalloc.c:1209)
==219777== by 0x2DAA71: allocate_from_new_pool (obmalloc.c:1389)
==219777== by 0x2DB00B: pymalloc_alloc (obmalloc.c:1553)
==219777== by 0x2DB00B: _PyObject_Malloc (obmalloc.c:1564)
==219777== by 0x424AF5: gc_alloc (gcmodule.c:2304)
==219777== by 0x424AF5: _PyObject_GC_New (gcmodule.c:2319)
==219777== by 0x2BDF3B: new_dict (dictobject.c:748)
==219777== by 0x2BDF3B: PyDict_New (dictobject.c:851)
==219777== by 0x34CF06: init_interned_dict (unicodeobject.c:248)
==219777== by 0x34CF06: _PyUnicode_InitGlobalObjects (unicodeobject.c:14670)
==219777== by 0x3EDE18: pycore_init_global_objects (pylifecycle.c:678)
==219777== by 0x3EDE18: pycore_interp_init (pylifecycle.c:826)
==219777== by 0x3F07A9: new_interpreter (pylifecycle.c:2079)
==219777== by 0x3F07A9: Py_NewInterpreterFromConfig (pylifecycle.c:2114)
==219777== by 0x4867A52: interp_create (_xxsubinterpretersmodule.c:516)
==219777== by 0x2D0C22: cfunction_call (methodobject.c:537)
==219777== by 0x2716A7: _PyObject_MakeTpCall (call.c:240)
I had a look around that code but I can't spot any obvious bug. I did verify that the interned strings dict created and destroyed in clear_interned_dict() is the same for each sub-interpreter, just in case.
I think every interned string is now also automatically immortal. Does that mean it wouldn't be cleaned up even if the refcount goes down to zero?
CPython versions tested on:
3.11, 3.12, CPython main branch
Operating systems tested on:
Linux
Metadata
Metadata
Assignees
Projects
Status