Skip to content

Py_Initialize() still leaks in libpython3.11 #100773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Gerzer opened this issue Jan 5, 2023 · 4 comments
Open

Py_Initialize() still leaks in libpython3.11 #100773

Gerzer opened this issue Jan 5, 2023 · 4 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@Gerzer
Copy link

Gerzer commented Jan 5, 2023

Bug Report

The following code leaks several hundred kilobytes with the latest libpython from the Ubuntu APT repository:

#include <Python.h>

int main(void) {
        Py_Initialize();
        Py_Finalize();
        return 0;
}

With libpython3.10, it consistently leaks 480 KB, whereas with libpython3.11, it consistently leaks 408 KB. (These are proper IEC power-of-10 kilobytes, not power-of-2 kibibytes.) I’ve verified this by running this code as a Clang-compiled dynamic executable through Valgrind.

Binary Images

libpython3.10 version:

linux-vdso.so.1 (0x0000ffffbf695000)
libpython3.10.so.1.0 => /lib/aarch64-linux-gnu/libpython3.10.so.1.0 (0x0000ffffbf050000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffbeea0000)
libexpat.so.1 => /lib/aarch64-linux-gnu/libexpat.so.1 (0x0000ffffbee60000)
libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000ffffbee30000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffbed90000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffbf65c000)

libpython3.11 version:

linux-vdso.so.1 (0x0000ffffaee59000)
libpython3.11.so.1.0 => /lib/aarch64-linux-gnu/libpython3.11.so.1.0 (0x0000ffffae650000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffae4a0000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffae400000)
libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000ffffae3d0000)
libexpat.so.1 => /lib/aarch64-linux-gnu/libexpat.so.1 (0x0000ffffae390000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffaee20000)

Valgrind Reports

Environment

  • CPython versions tested on:
    • libpython3.10
    • libpython3.11
      (Both from the Ubuntu APT repository)
  • Operating system and architecture:
    • Ubuntu 22.04.1 LTS
    • AArch64
    • Virtual machine on top of Apple’s Virtualization and Hypervisor frameworks
    • macOS Ventura 13.2 (a) Beta (22D7750270d)
    • MacBook Pro with M1 Max
@Gerzer Gerzer added the type-bug An unexpected behavior, bug, or error label Jan 5, 2023
@sobolevn
Copy link
Member

sobolevn commented Jan 6, 2023 •
edited
Loading

cc @vstinner who worked on removing the startup leakages as far as I remember :)

@doctorlai-msrc
Copy link

doctorlai-msrc commented Feb 13, 2023 •
edited
Loading

Leaking detected by Address Santitizer as well:

=================================================================
==22370==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 267212 byte(s) in 100 object(s) allocated from:
    #0 0x7f0a1c916808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f0a1c50d9e7  (/lib/x86_64-linux-gnu/libpython3.8.so.1.0+0x25a9e7)

Direct leak of 536 byte(s) in 1 object(s) allocated from:
    #0 0x7f0a1c916808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f0a1c50d182  (/lib/x86_64-linux-gnu/libpython3.8.so.1.0+0x25a182)

Indirect leak of 25922 byte(s) in 26 object(s) allocated from:
    #0 0x7f0a1c916808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f0a1c50d9e7  (/lib/x86_64-linux-gnu/libpython3.8.so.1.0+0x25a9e7)

SUMMARY: AddressSanitizer: 293670 byte(s) leaked in 127 allocation(s).

@vstinner
Copy link
Member

vstinner commented Apr 5, 2023

It's different if memory is allocated exactly once and never released explicitly by Py_Finalize(), or if each Py_Initialize() call allocates new memory. Most of remaining "leaks" are memory allocated exactly once and then never released by Py_Finalize().

There is still a work-in-progress to convert remaining stdlib extensions to multi-phase initialization. A recent example with _elementtree: fee7a99

If you are curious, you should dig into the pymain_free() function of Modules/main.c to see memory not freed by Py_Finalize(), but freed by Py_RunMain().

@iritkatriel iritkatriel added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Nov 26, 2023
@Mekk
Copy link

Mekk commented Dec 15, 2023 •
edited
Loading

Python 3.12 arrived with plenty of new sanitizer-reported errors :-(
( #113190 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@Mekk @vstinner @iritkatriel @sobolevn @Gerzer @doctorlai-msrc