Skip to content

Memory Leak in python when using a C# DLL #577

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BreitA opened this issue Nov 24, 2017 · 15 comments
Closed

Memory Leak in python when using a C# DLL #577

BreitA opened this issue Nov 24, 2017 · 15 comments

Comments

@BreitA
Copy link

BreitA commented Nov 24, 2017

Environment

  • Pythonnet version: 2.3.0
  • Python version: 3.6.1
  • Operating System: Windows 7 Professional 64bits.
  • .NET Framework 4.6

Details

  • I experience memory leaks when trying to use a C# dll in python to run a simulation and retrieve results using python. When I try to run multiple time the calculation I get leaks
    Here is a template of code I run to experience this memory leak (replaced the name of the dll I use by generic names)
import clr
import os
import sys
import psutil
import gc
dll_path = r''the path to my dll"
sys.path.append(os.path.join(dll_path, 'MyDll.dll'))
clr.AddReference(os.path.join(dll_path, "MyDll"))
while psutil.virtual_memory().percent < 80:
    print('psutil_virtual_memory:', psutil.virtual_memory().percent)
    obj = MyDll.MyObject()
    res = obj.Run() # long simulation is run and I get the results.
    gc.collect()
    clr.System.GC.Collect()

Output :
psutil_virtual_memory: 41.8
psutil_virtual_memory: 44.4
psutil_virtual_memory: 46.6
psutil_virtual_memory: 48.9
.....
until my memory reach 80%

The issue is that running the equivalent code in C# does not give any memory leak but it's when calling it from python we get issues of memory leaks. the python garbage collector doesn't help at all with this. I tried also to call the System GC Collect but without success too.

My question is how can I investigate and try to fix this issue.

@den-run-ai
Copy link
Contributor

Please try out the fix according to this issue:

#541

@BreitA
Copy link
Author

BreitA commented Nov 24, 2017

@denfromufa
If I read well the issue #541 it seems you were saying in this issue

When I test Python calls to .NET, then the memory leak is less obvious, since all resources are released when calling the CPython gc.collect()

My issue is that gc.collect() from python doesn't release my memory.

I saw that you have a branch fixing the memory issue.

Should I try to install the pythonnet version of your branch?

@den-run-ai
Copy link
Contributor

Yes, try this branch developed by @dmitriyse:

#532

@dmitriyse
Copy link
Contributor

I also created branch https://github.com/pythonnet/pythonnet/tree/featured
With all PRs applied. I would recommend to test this branch if something goes wrong.

@BreitA
Copy link
Author

BreitA commented Nov 24, 2017

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets(983,5): warning MSB3644: The reference assembl
ies for framework ".NETFramework,Version=v4.0" were not found. To resolve this, install the SDK or Targeting Pack for t
his framework version or retarget your application to a version of the framework for which you have the SDK or Targetin
g Pack installed. Note that assemblies will be resolved from the Global Assembly Cache (GAC) and will be used in place
of reference assemblies. Therefore your assembly may not be correctly targeted for the framework you intend.

CSC : error CS1617: Invalid option '6' for /langversion; must be ISO-1, ISO-2, 3, 4, 5 or Default

got this warning and this error when doing python setup.py bdist_wheel with your branch. Is there any quick fix I can try?

@den-run-ai
Copy link
Contributor

den-run-ai commented Nov 25, 2017 via email

@den-run-ai
Copy link
Contributor

den-run-ai commented Nov 25, 2017 via email

@den-run-ai
Copy link
Contributor

den-run-ai commented Nov 25, 2017 via email

@BreitA
Copy link
Author

BreitA commented Nov 25, 2017

Sorry I didn't restarted my computer after installing VS I must be tired. So now I fixed this issue I got the following error

Python.Runtime -> E:\python_project\pythonnet-featured\pythonnet-featured\src\runtime\bin\Python.Runtime.dll
  Python.Test -> E:\python_project\pythonnet-featured\pythonnet-featured\src\testing\bin\Python.Test.dll
dynamic.cs(121,28): warning CS0618: 'PythonEngine.RunString(string, IntPtr?, IntPtr?)' is obsolete: 'RunString is depre
cated and will be removed. Use Exec/Eval/RunSimpleString instead.' [E:\python_project\pythonnet-featured\pythonnet-feat
ured\src\embed_tests\Python.EmbeddingTest.csproj]
  Python.EmbeddingTest -> E:\python_project\pythonnet-featured\pythonnet-featured\src\embed_tests\bin\Python.EmbeddingT
  est.exe
  Console -> E:\python_project\pythonnet-featured\pythonnet-featured\src\console\bin\nPython.exe
  clrmodule -> E:\python_project\pythonnet-featured\pythonnet-featured\src\clrmodule\bin\clrmodule.dll
E:\python_project\pythonnet-featured\pythonnet-featured\packages\UnmanagedExports.1.2.7\tools\RGiesecke.DllExport.targe
ts(58,3): error : Microsoft.Build.Utilities.ToolLocationHelper could not find ildasm.exe. [E:\python_project\pythonnet-
featured\pythonnet-featured\src\clrmodule\clrmodule.csproj]

I added to path the winsdk link to this exe but this change nothing. I am on my home machine so the operating system is now :
Operating System: Windows 10 Home 64bits.
Visual studio and .NET :
Microsoft Visual Studio Community 2017
Version 15.4.4
VisualStudio.15.Release/15.4.4+27004.2009
Microsoft .NET Framework
Version 4.7.02046

Installed Version: Community

when using --xplat option ihave the same error about ildasm.exe after it runs through .NET core

I have 3 ildasm.exe :
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.7 Tools\x64\ildasm.exe
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\ildasm.exe

it looks like this

#216

I checked the file in question and it seems it already calls the 1.2.7 of unmanagedexports
is there anything I can do to fix this in my environment or is this in the code of the module ?

@BreitA
Copy link
Author

BreitA commented Nov 25, 2017

I manage to install it from the following binary :

https://ci.appveyor.com/project/pythonnet/pythonnet/branch/featured/job/7daumkda8h36k89t/artifacts

this branch didn't fix my memory leak

@den-run-ai
Copy link
Contributor

You need to provide minimum reproducible example like I did in the linked issue above.

@BreitA
Copy link
Author

BreitA commented Nov 25, 2017

Ok I'll try to do that but it make take a while for me to identify one leaking element of the C# code library I use because I know very few thing about this code yet or about C# in general to begin with and this code is a simulator using multiple nested structure of C# objects.

Anyway your simple example of instancing a double in a for loop doesn't cause memory leak in my case even on the release 2.3.0 version of pythonnet. I will try to instanciate a simple example of a nested structure of C# object to see if it leaks.

I also used a temporary solution to work with until i fix the issue, I encapsulate my call to the C# library in a multiprocessing.Process.
when the process end the memory leak is cleaned with the process and doesn't propagate to the main process.

using multiprocessing.Pool, the memory leak is shared to the pool meaning if you have a lot of elements in your pool the memory leak will skyrocket pretty fast but when the whole pool end the memory leak is also cleaned

Thanks for your time.

@den-run-ai
Copy link
Contributor

You can also use .NET memory profiler such as one of these, but it may not work due to some unmanaged memory:

https://stackoverflow.com/a/100490/2230844

The easiest is commenting out code until the leak is narrowed down to few lines of code.

@den-run-ai
Copy link
Contributor

closing due to inactivity and no reproducible example

@BreitA
Copy link
Author

BreitA commented Apr 22, 2018

@denfromufa Since i didn't have access to the C# code i couldn't do what was suggested. Anyway i circumvent the issue migrating my code to linux with mono and dont have memory leak issues there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants