Skip to content

Python process not exiting after garbage collection #2129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
michaelosthege opened this issue Mar 17, 2023 · 1 comment
Closed

Python process not exiting after garbage collection #2129

michaelosthege opened this issue Mar 17, 2023 · 1 comment

Comments

@michaelosthege
Copy link

michaelosthege commented Mar 17, 2023

Environment

Details

I'm using some .NET libraries from Python, and my problem is at the process shutdown:

The Python script has completed, and Python garbage collection resulted in 0 referring and 0 referred objects:

print(gc.get_referrers())  # → 0
print(gc.get_referents())  # → 0

The Python process is not aware of any child processes:

import multiprocessing
for child in multiprocessing.active_children():
    print(f"Killing {child.pid}")
    child.kill()
# → no print output

There are just 3 threads, but I don't know where they come from and threads can't be killed:

import threading
print(f"There are {threading.active_count()} child threads")
# → There are 3 child threads

If I sys.exit() the process, it still keeps running, so apparently there are child processes that aren't daemons?

As far as I can tell I have cleaned up the .NET objects I instantiated (from Python), but I'm not so sure about it.

I think there might be more .NET threads/processes lurking that are not found by Pythons multiprocessing?
With htop I can see that the main process actually has 11 children, so more than accounted for by the self-inspection above.

If I do py-spy top --pid 1 on the Python process it fails to parse /proc/18/stat which sounds like this might belong to a .NET child process that py-spy just doesn't expect?

So far I have spent about the entire week debugging this, because it's blocking me from upgrading to current .NET and pythonnet versions.
Maybe pythonnet 2.5.2 was more brutal with shutting down child processes?

Do pythonnet folks have some ideas what might cause these symptoms?
Is there anything I can do to let the runtime know to please just shut *** **** down?

UPDATE:
Minutes after posting, I discovered that I had not actually disposed a .NET object with some background threads, even though the Python objects holding a reference to it got garbage collected.

👉 Is this a bug of incomplete garbage collection at the .NET level? Or was it a bug in pythonnet 2.5.2 that .NET threads were killed when everything in the Python parent process was garbage collected?

👉 How am I supposed to dispose .NET objects if they don't get garbage collected even after the Python process garbage collected all references?

👉 Am I running into a new version of #1872 ?

UPDATE:
I found a workaround for my project, using atexit.register to make sure that the destructor/dispose methods are called when the main process is ready to exit. At the same time im 80 % positive that this is the same bug as #1872

@lostmsu
Copy link
Member

lostmsu commented May 5, 2023

1872 was a particular issue, that got fixed. Based on the data you shared you don't even have a bug. Read that thread carefully.

@lostmsu lostmsu closed this as completed May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants