Skip to content

Extension Module Handles Are Never Closed #114538

Open
@ericsnowcurrently

Description

@ericsnowcurrently

We load extension modules by loading the shared library file (dlopen(), LoadLibraryExW()). However we never unload/release it. In each case we get a handle when load it, but we throw the handle away. We should consider holding on to the handle (maybe in something like PyModuleObject.md_handle).

If we save the handle, we could use it to release the loaded shared library (dlclose(), FreeLibrary()). module_dealloc() would probably be a good place to do so. In every case. releasing it drops an internal refcount, and when it reaches 0 the lib is actually unloaded.

Aside from not leaking, there's another real benefit. Extension modules can (and often do, e.g. static types) store objects in C global variables. Single-phase init modules do not have an opportunity to clean up after themselves (which was part of the motivation for PEP 489). So they don't have a chance to decref any objects they may have stashed away in globals. This is problematic when such stashed objects survive runtime init/fini cycles. 1 Closing the lib means all the extensions C globals are discarded and thus importing the module in a later cycle will result in a clean set of globals. That may mean a few objects get leaked, but that's better than segfaulting.

There are some complexities though. Using dlclose(), etc. must be done carefully. While not critical, it's best to do it in the thread where the "open" was done. Also, once the lib is closed, trying to use any variables or functions defined in it will segfault. That means you can't do it if there's any thread running C code from the module (incl. daemon threads). Likewise if the module registered a callback somewhere or if a reference to one of its static types (or other global objects) is held outside the module. That may even impact GC.

We might consider at least closing the lib if _PyImport_CheckSubinterpIncompatibleExtensionAllowed() fails in _PyImport_LoadDynamicModuleWithSpec(). The risk there is smaller.

FWIW, in practice this isn't a big concern because a loaded extension module typically doesn't get destroyed until not long before the process terminates (though it might be a different story for embedders). I'm bringing it up because it something I noticed and looked into and thought we might at least consider the problem.

Footnotes

  1. The same goes for when an extension is loaded for the first time in an isolated (per-interpreter GIL) subinterpreter rather than in the main interpreter, since the C globals will outlive the interpreter. However, there are some caveats with that too. Also, though single-phase init modules are not supported in isolated interpreters, their init functions still get run.

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions