memory leak - simple function call in a loop #541

den-run-ai · 2017-09-11T19:32:06Z

I'm getting a memory leak with simple calls from .NET to Python function. The issue is reproducible on all pythonnet versions that I tested. When I test Python calls to .NET, then the memory leak is less obvious, since all resources are released when calling the CPython gc.collect(). Calling the same function from .NET does not release any resources, but calling the CLR GC.WaitForPendingFinalizers() blocks the execution.

memoryleakpykw.zip

>>> import clr
>>> import System
>>> for i in range(20000000):
...   a=System.Double(i)
...
>>> import gc
>>> gc.collect()

pyinclr.py

def testmethod():
	return "abc"*3

using System;
using Python.Runtime;

namespace memoryleakpykw
{
    class Program
    {
        static void Main(string[] args)
        {
            PyObject pymodule;
            PyObject gc;
            PyObject testmethod;

            PythonEngine.Initialize();

            IntPtr pylock = PythonEngine.AcquireLock();
            {
                PyObject sys = PythonEngine.ImportModule("sys");// Py.Import("sys");
                gc = PythonEngine.ImportModule("gc");// Py.Import("sys");
                sys.GetAttr("path").InvokeMethod("append",
                    new PyString(@"C:\Users\denis.akhiyarov\source\repos\memoryleakpykw\pyinclr"));
                pymodule = PythonEngine.ImportModule("pyinclr");
                testmethod = pymodule.GetAttr("testmethod");
            }

            PythonEngine.ReleaseLock(pylock);

            Console.WriteLine("starting test for memory leak, press any key");
            Console.ReadKey();

            pylock = PythonEngine.AcquireLock();

            {
                for (int i=1; i<=1000000; i++)
                {
                    testmethod.Invoke();
                }
            }

            gc.InvokeMethod("collect"); 

            PythonEngine.ReleaseLock(pylock);
            
            GC.Collect();
            //GC.WaitForPendingFinalizers(); // this blocks execution
            GC.Collect();

            Console.WriteLine("finished test for memory leak, press any key");
            Console.ReadKey();
        }

    }
}

The text was updated successfully, but these errors were encountered:

den-run-ai · 2017-09-11T19:43:46Z

Actually GC.WaitForPendingFinalizers(); is blocking, not gc.collect!

dmitriyse · 2017-09-11T21:03:42Z

Please try this version of your code:

using System;
using Python.Runtime;

namespace memoryleakpykw
{
    class Program
    {
        static void Main(string[] args)
        {
            PyObject pymodule;
            PyObject gc;
            PyObject testmethod;

            PythonEngine.Initialize();
            var ts = PythonEngine.BeginAllowThread();

            IntPtr pylock = PythonEngine.AcquireLock();
            {
                PyObject sys = PythonEngine.ImportModule("sys");// Py.Import("sys");
                gc = PythonEngine.ImportModule("gc");// Py.Import("sys");
                sys.GetAttr("path").InvokeMethod("append",
                    new PyString(@"C:\Users\denis.akhiyarov\source\repos\memoryleakpykw\pyinclr"));
                pymodule = PythonEngine.ImportModule("pyinclr");
                testmethod = pymodule.GetAttr("testmethod");
            }

            PythonEngine.ReleaseLock(pylock);

            Console.WriteLine("starting test for memory leak, press any key");
            Console.ReadKey();

            pylock = PythonEngine.AcquireLock();

            {
                for (int i=1; i<=1000000; i++)
                {
                    testmethod.Invoke();
                }
            }

            gc.InvokeMethod("collect"); 

            PythonEngine.ReleaseLock(pylock);
            

            GC.Collect();
            GC.WaitForPendingFinalizers(); // should no block
            GC.Collect();

            GC.WaitForPendingFinalizers(); // should no block

            PythonEngine.EndAllowThread(ts);

            Console.WriteLine("finished test for memory leak, press any key");
            Console.ReadKey();
        }

    }
}

den-run-ai · 2017-09-11T22:00:20Z

@dmitriyse I'm not sure how is that supposed to help with the memory leak, but the leak is still there and not released from both GC. WaitForPendingFinalizers still blocks.

den-run-ai · 2017-09-11T22:40:34Z

I got rid of this memory leak by using the following inconvenient disposal code:

PyObject res = testmethod.Invoke();
res.Dispose();

Not sure why res cannot be disposed by either GC?!

den-run-ai · 2017-09-12T15:33:18Z

I still don't understand why managed GC from CLR is not calling the finalizer of PyObject for the results of calls into pythonnet:

        ~PyObject()
        {
            Dispose();
        }

What is holding the reference to these objects or why GC is not activated?

Thanks for the edit @dmitriyse, now when EndAllowThreads is after WaitForPendingFinalizers, then some of resources are released, but not all:

            PythonEngine.ReleaseLock(pylock);

            GC.Collect();
            GC.WaitForPendingFinalizers();
            GC.Collect();

            PythonEngine.EndAllowThreads(ts);

I read these Q&A and nothing looks wrong in PyObject:

https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/dispose-pattern

https://stackoverflow.com/questions/3038571/whats-the-purpose-of-gc-suppressfinalizethis-in-dispose-method

https://stackoverflow.com/questions/45036/will-the-garbage-collector-call-idisposable-dispose-for-me

https://stackoverflow.com/questions/3038571/whats-the-purpose-of-gc-suppressfinalizethis-in-dispose-method

https://stackoverflow.com/questions/538060/proper-use-of-the-idisposable-interface

https://stackoverflow.com/questions/898828/finalize-dispose-pattern-in-c-sharp

dmitriyse · 2017-09-12T15:56:56Z

I tested this code in the branch #532.
And no any memory leaks occurred.

This image was captured after some minutes after start. So memory consumption is totally leak-free.

When I debugged this branch I faced with unpredictable C# GC behavior, after some amount of valid "Dispose" calls, GC does not generate enough finalizer calls. Looks like some unmanaged actions are not allowed inside GC thread.

den-run-ai · 2017-09-13T22:09:10Z

@dmitriyse I tested your branch for finalizer with .NET Framework 4.6.1 and it looks great!

den-run-ai · 2017-09-13T22:11:58Z

Interestingly .NET 4.0 garbage collector behaves much nicer:

den-run-ai · 2017-09-14T16:59:48Z

Here is the testing for .NET Core 2.0, also looks very solid!

den-run-ai · 2017-09-15T16:11:05Z

@dmitriyse here is another leak, even with your valid finalizer. This is when raising exception in Python and catching it in C#:

def testmethod(number=3, astring='abc'):
    raise ValueError("testing for memory leak")
    return astring * int(number)

using System;
using System.ComponentModel;
using Python.Runtime;

namespace memoryleakpykw
{
    class Program
    {
        static void Main(string[] args)
        {
            dynamic pymodule;
            PyObject gc;
            dynamic testmethod;
            PythonEngine.Initialize();
            var ts = PythonEngine.BeginAllowThreads();
            IntPtr pylock = PythonEngine.AcquireLock();
            {
                PyObject sys = PythonEngine.ImportModule("sys");// Py.Import("sys");
                gc = PythonEngine.ImportModule("gc");// Py.Import("sys");
                sys.GetAttr("path").InvokeMethod("append",
                    new PyString(@"C:\Users\denis.akhiyarov\source\repos\memoryleakpykw\pyinclr"));
                pymodule = PythonEngine.ImportModule("pyinclr");
                testmethod = pymodule.testmethod; //.GetAttr("testmethod");
            }
            PythonEngine.ReleaseLock(pylock);
            Console.WriteLine("starting test for memory leak, press any key");
            Console.ReadKey();
            double floatarg1 = 5.1f;
            dynamic res = null;
            {
                for (int i = 1; i <= 1000000; i++)
                {
                    using (Py.GIL())
                    {
                        try { 
                            res = testmethod(Py.kw("number", floatarg1), Py.kw("astring", "bbc"));
                        }
                        catch (Exception e)
                        {
                            if (i % 1000 == 0)
                            {
                                Console.WriteLine(e.Message);
                            }
                        }
                    }
                }
            }
            Console.WriteLine(res);           
            PythonEngine.EndAllowThreads(ts);
            Console.WriteLine("finished test for memory leak, press any key");
            Console.ReadKey();
        }
    }
}

dmitriyse · 2017-12-12T19:36:20Z

Memory leak example with an exception is not so good. I tried the same example but without exception raising and it still produces memory leak (iterations count should be increased).

dmitriyse · 2017-12-12T21:48:29Z

I fixed exception related memory leak. See latest update to PR #532

dmitriyse · 2017-12-12T22:48:42Z

We needs to close this issue after PR #532 merge to master

Zamgunner · 2018-11-14T12:26:36Z

guys, any update on merging this fix to mainline?

den-run-ai mentioned this issue Oct 11, 2017

Memory leak on windows r0x0r/pywebview#137

Closed

den-run-ai mentioned this issue Nov 24, 2017

Memory Leak in python when using a C# DLL #577

Closed

den-run-ai added bug memory leak labels Apr 18, 2018

den-run-ai mentioned this issue Apr 18, 2018

memory de-allocation #663

Closed

dmitriyse mentioned this issue Jun 25, 2018

PyObject finalizer #692

Merged

4 tasks

filmor closed this as completed Jun 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

memory leak - simple function call in a loop #541

memory leak - simple function call in a loop #541

den-run-ai commented Sep 11, 2017 •

edited

Loading

den-run-ai commented Sep 11, 2017

dmitriyse commented Sep 11, 2017 •

edited

Loading

den-run-ai commented Sep 11, 2017

den-run-ai commented Sep 11, 2017

den-run-ai commented Sep 12, 2017

dmitriyse commented Sep 12, 2017 •

edited

Loading

den-run-ai commented Sep 13, 2017

den-run-ai commented Sep 13, 2017

den-run-ai commented Sep 14, 2017

den-run-ai commented Sep 15, 2017 •

edited

Loading

dmitriyse commented Dec 12, 2017 •

edited

Loading

dmitriyse commented Dec 12, 2017 •

edited

Loading

dmitriyse commented Dec 12, 2017

Zamgunner commented Nov 14, 2018

memory leak - simple function call in a loop #541

memory leak - simple function call in a loop #541

Comments

den-run-ai commented Sep 11, 2017 • edited Loading

den-run-ai commented Sep 11, 2017

dmitriyse commented Sep 11, 2017 • edited Loading

den-run-ai commented Sep 11, 2017

den-run-ai commented Sep 11, 2017

den-run-ai commented Sep 12, 2017

dmitriyse commented Sep 12, 2017 • edited Loading

den-run-ai commented Sep 13, 2017

den-run-ai commented Sep 13, 2017

den-run-ai commented Sep 14, 2017

den-run-ai commented Sep 15, 2017 • edited Loading

dmitriyse commented Dec 12, 2017 • edited Loading

dmitriyse commented Dec 12, 2017 • edited Loading

dmitriyse commented Dec 12, 2017

Zamgunner commented Nov 14, 2018

den-run-ai commented Sep 11, 2017 •

edited

Loading

dmitriyse commented Sep 11, 2017 •

edited

Loading

dmitriyse commented Sep 12, 2017 •

edited

Loading

den-run-ai commented Sep 15, 2017 •

edited

Loading

dmitriyse commented Dec 12, 2017 •

edited

Loading

dmitriyse commented Dec 12, 2017 •

edited

Loading