Skip to content

Deleted dotnet object does not get garbage collected #1872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
m-rossi opened this issue Jul 13, 2022 · 7 comments
Closed

Deleted dotnet object does not get garbage collected #1872

m-rossi opened this issue Jul 13, 2022 · 7 comments

Comments

@m-rossi
Copy link
Contributor

m-rossi commented Jul 13, 2022

Environment

  • Pythonnet version: 3.0.0rc3
  • Python version: 3.10.5
  • Operating System: Windows 10 20H2
  • .NET Runtime: 6.0.6

Details

  • Describe what you were trying to get done.

Create dotnet-object and delete it. If I do that a lot I get memory issues. It seems the never get cleared by the Python- or dotnet-gc. This is not a problem in pythonnet 2.5.2

  • What commands did you run to trigger this issue?
import gc
import clr
import psutil
import matplotlib.pyplot as plt
import System

# Create dict store count of objects left
types = [
    str,
    System.String,
]
types = {obj: 0 for obj in types}

N = 1_000
str_size = 1_000_000

memory_py = [0 for _ in range(N + 2)]
memory_py[0] = psutil.virtual_memory().used
for ii in range(N):
    s_py = str('a' * str_size)
    del s_py
    memory_py[ii + 1] = psutil.virtual_memory().used - memory_py[0]

gc.collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_py[ii + 2] = psutil.virtual_memory().used - memory_py[0]
memory_py[0] = 0
{<class 'str'>: 0, <class 'System.String'>: 0}
memory_net = [0 for _ in range(N + 2)]
memory_net[0] = psutil.virtual_memory().used
s_py = str('a' * str_size)
for ii in range(N):
    s_net = System.String(s_py)
    del s_net
    memory_net[ii + 1] = psutil.virtual_memory().used - memory_net[0]

gc.collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_net[ii + 2] = psutil.virtual_memory().used - memory_net[0]
memory_net[0] = 0
{<class 'str'>: 0, <class 'System.String'>: 1000}

You can also visualize the memory consumption:

fig, ax = plt.subplots()
ax.plot(memory_py, label='Python object')
ax.plot(memory_net, label='DotNet object')
ax.legend()
plt.show(fig)

3 0

You can execute the same code with pythonnet 2.5.2 (Python 3.9.13) and get the following figure

2 5

@lostmsu
Copy link
Member

lostmsu commented Jul 13, 2022

You are not calling .NET GC

@lostmsu lostmsu closed this as completed Jul 13, 2022
@m-rossi
Copy link
Contributor Author

m-rossi commented Jul 14, 2022

@lostmsu I get the same results if I add System.GC.Collect()

Full code for reference:

import gc
import clr
import psutil
import matplotlib.pyplot as plt
import System

# Create dict store count of objects left
types = [
    str,
    System.String,
]
types = {obj: 0 for obj in types}

N = 1_000
str_size = 1_000_000

memory_py = [0 for _ in range(N + 2)]
memory_py[0] = psutil.virtual_memory().used
for ii in range(N):
    s_py = str('a' * str_size)
    del s_py
    memory_py[ii + 1] = psutil.virtual_memory().used - memory_py[0]

gc.collect()
System.GC.Collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_py[ii + 2] = psutil.virtual_memory().used - memory_py[0]
memory_py[0] = 0
{<class 'str'>: 0, <class 'System.String'>: 0}
memory_net = [0 for _ in range(N + 2)]
memory_net[0] = psutil.virtual_memory().used
s_py = str('a' * str_size)
for ii in range(N):
    s_net = System.String(s_py)
    del s_net
    memory_net[ii + 1] = psutil.virtual_memory().used - memory_net[0]

gc.collect()
System.GC.Collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_net[ii + 2] = psutil.virtual_memory().used - memory_net[0]
memory_net[0] = 0
{<class 'str'>: 0, <class 'System.String'>: 1000}

@filmor
Copy link
Member

filmor commented Jul 14, 2022

Hmm, this tells us that Python is not releasing the wrapper objects for some reason. Can you check the refcount and referrers for these?

@m-rossi
Copy link
Contributor Author

m-rossi commented Jul 14, 2022

I used objgraph to display the backrefs. The output of gc.get_referrers(obj) is really huge or have you another advice to display the referrers?

import gc
import sys
import clr
import objgraph
import System

s_net = System.String('a')
del s_net

gc.collect()
System.GC.Collect()
for obj in gc.get_objects():
    if type(obj) == System.String:
        print(f'Refcount: {sys.getrefcount(obj)}')
        print(f'Num Referrers: {len(gc.get_referrers(obj))}')
        objgraph.show_backrefs(obj, filename='backref.png')
Refcount: 4
Num Referrers: 2

backref

@lostmsu
Copy link
Member

lostmsu commented Jul 14, 2022

@filmor an update from my debugging the issue:

From this code I am getting a refcount rc of 3:

import ctypes
from System import Uri

u = Uri('http://google.com')
addr = id(u)

rc = ctypes.c_long.from_address(addr).value

del u
# tp_dealloc not called

Replacing Uri with Python's str gives rc 1.

The issue does not seem to happen when objects are created from .NET. E.g. this does not produce extra references:

var uri = new Uri("http://google.com").ToPython();
// uri.Refcount == 1
uri.Dispose();
// `tp_dealloc` called

Looks like we leak references when constructing new .NET objects from Python.

@lostmsu lostmsu reopened this Jul 14, 2022
lostmsu added a commit to losttech/pythonnet that referenced this issue Jul 14, 2022
lostmsu added a commit to losttech/pythonnet that referenced this issue Jul 14, 2022
lostmsu added a commit to losttech/pythonnet that referenced this issue Jul 14, 2022
@filmor
Copy link
Member

filmor commented Jul 14, 2022

@lostmsu Great work, thanks! And thank you @m-rossi for testing and reporting! Could you verify that the newest release branch indeed fixes the issue?

@m-rossi
Copy link
Contributor Author

m-rossi commented Jul 15, 2022

Looks good, the gc sees no System.String-objects left and the memory consumption is better too:
Bild1

Thanks @lostmsu !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants