Skip to content

Memory leak in CLRObject #881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
NickSavin opened this issue Jun 14, 2019 · 1 comment
Open

Memory leak in CLRObject #881

NickSavin opened this issue Jun 14, 2019 · 1 comment

Comments

@NickSavin
Copy link
Contributor

Environment

  • Pythonnet version: 2.4 master
  • Python version: 3.7
  • Operating System: Win10 x64

Details

  • Describe what you were trying to get done.

    Hello, we are using PythonNet to execute custom python scripts in our C# application. We pass a set of large .net objects (such as DataTable with a big bunch of data) as a script locals. When we execute such sctripts multiple times we got an OutOfMemoryException

  • What commands did you run to trigger this issue? If you can provide a
    Minimal, Complete, and Verifiable example
    this will help us understand the issue.

    public void Execute(string script, IDictionary<string, object> locals, List<string> searchPaths)
        {
            var modifiedScript = $"{script}{Environment.NewLine}{string.Format(PythonSnippets.SystemEventHandler, pySystemHandlerGuid)}";

            _scriptLocals = new PyDict();

            using (Py.GIL())
            {
                if (locals != null)
                    foreach (var local in locals)
                        _scriptLocals[local.Key] = local.Value.ToPython();

                _pythonScope = Py.CreateScope();
                _pythonScope.Exec(modifiedScript, _scriptLocals);
            }
        }

Here we cache PyScope and PyDict. Pydict contains our large .net objects, translated to Python

    public void ClearCache()
        {
            using (Py.GIL())
            {
                _pythonScope.Exec($"vars().clear()");

                foreach (PyObject pObject in _scriptLocals.Items())
                {
                    pObject.Dispose();
                }

                dynamic gc = Py.Import("gc");
                gc.collect();

                _scriptLocals.Dispose();

                _pythonScope.Dispose();

                gc.collect();
            }
        }

This method we execute each time, after we finished our work with script. As you can see, I tried different way to clean up resources, but memory leak still occures, after multiple executions.
I did a snapshot of memory with .net profiler, and found that there a references to my DataTables from the CLRObject and I did not find a way how to release it.

MemoryConsumption

references

StackTrace


Exception of type 'System.OutOfMemoryException' was thrown.

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Data.RBTree`1.TreePage..ctor(Int32 size)
   at System.Data.RBTree`1.AllocPage(Int32 size)
   at System.Data.RBTree`1.GetNewNode(K key)
   at System.Data.DataTable.SetNewRecordWorker(DataRow row, Int32 proposedRecord, DataRowAction action, Boolean isInMerge, Boolean suppressEnsurePropertyChanged, Int32 position, Boolean fireEvent, Exception& deferredException)
   at System.Data.DataTable.InsertRow(DataRow row, Int64 proposedID, Int32 pos, Boolean fireEvent)
   at System.Data.DataTable.LoadDataRow(Object[] values, Boolean fAcceptChanges)
   at System.Data.ProviderBase.SchemaMapping.LoadDataRow()
   at System.Data.Common.DataAdapter.FillLoadDataRow(SchemaMapping mapping)
   at System.Data.Common.DataAdapter.FillFromReader(DataSet dataset, DataTable datatable, String srcTable, DataReaderContainer dataReader, Int32 startRecord, Int32 maxRecords, DataColumn parentChapterColumn, Object parentChapterValue)
   at System.Data.Common.DataAdapter.Fill(DataTable[] dataTables, IDataReader dataReader, Int32 startRecord, Int32 maxRecords)
   at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)
   at System.Data.Common.DbDataAdapter.Fill(DataTable[] dataTables, Int32 startRecord, Int32 maxRecords, IDbCommand command, CommandBehavior behavior)
   at System.Data.Common.DbDataAdapter.Fill(DataTable dataTable)
   at Geobank.Shared.Data.Server.GetTableFillSchema(IConnection connection, DbConnection cn, DbTransaction tr, String sql, SqlParameter[] parameters)
   at Geobank.Shared.Data.Server.GetTableFillSchema(IConnection connection, String sql, SqlParameter[] parameters)
   at Geobank.Module.Forms.Services.FormDataViewCacheRepository.GenerateDataView(Guid dataSetGuid)
   at Geobank.Module.Forms.Services.FormDataViewCacheRepository.AddDataSetComponent(Guid guid)
   at Geobank.Module.Forms.Services.FormDataViewCacheRepository.Add(Guid DataSetGuid)
   at Geobank.Module.Forms.Services.FormDesignerExecuteManager.InitializeDataViewsRepository(IDataSource dataSource, IEnumerable`1 dataBindings)
   at Geobank.Module.Forms.Services.FormDesignerExecuteManager.CreateExecutableForm(IFormDesignerComponent formDesignerComponent, IDataSource dataSource, IDictionary`2 locals)
   at Geobank.Module.Forms.ViewModels.FormDesignerEditViewModel.ExecuteForm(Object sender)
   at Geobank.Shared.Helpers.RelayCommand.Execute(Object parameter)
   at DevExpress.Xpf.Bars.Native.CommandSourceHelper.Execute(ICommand command, Object commandParameter, IInputElement commandTarget)
   at DevExpress.Xpf.Bars.BarItem.ExecuteCommand(ICommand commandToExecute, Object commandParameterToExecute, IInputElement actualCommandTarget)
   at DevExpress.Xpf.Bars.BarItem.<>c__DisplayClass308_0.<OnItemClick>b__0()
   at DevExpress.Xpf.Bars.BarItem.OnItemClick(BarItemLink link, IBarItemLinkControl linkControl)
   at DevExpress.Xpf.Bars.BarItemLinkBase.OnClick(IBarItemLinkControl linkControl)
   at DevExpress.Xpf.Bars.BarItemLinkControlStrategy.OnClick()
   at DevExpress.Xpf.Bars.BarButtonItemLinkControlStrategy.OnMouseLeftButtonUp(MouseButtonEventArgs args)
   at DevExpress.Xpf.Bars.BarItemLinkControlBase.OnMouseLeftButtonUp(MouseButtonEventArgs e)
   at System.Windows.UIElement.OnMouseLeftButtonUpThunk(Object sender, MouseButtonEventArgs e)
   at System.Windows.Input.MouseButtonEventArgs.InvokeEventHandler(Delegate genericHandler, Object genericTarget)
   at System.Windows.RoutedEventArgs.InvokeHandler(Delegate handler, Object target)
   at System.Windows.RoutedEventHandlerInfo.InvokeHandler(Object target, RoutedEventArgs routedEventArgs)
   at System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)
   at System.Windows.UIElement.ReRaiseEventAs(DependencyObject sender, RoutedEventArgs args, RoutedEvent newEvent)
   at System.Windows.UIElement.OnMouseUpThunk(Object sender, MouseButtonEventArgs e)
   at System.Windows.Input.MouseButtonEventArgs.InvokeEventHandler(Delegate genericHandler, Object genericTarget)
   at System.Windows.RoutedEventArgs.InvokeHandler(Delegate handler, Object target)
   at System.Windows.RoutedEventHandlerInfo.InvokeHandler(Object target, RoutedEventArgs routedEventArgs)
   at System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)
   at System.Windows.UIElement.RaiseEventImpl(DependencyObject sender, RoutedEventArgs args)
   at System.Windows.UIElement.RaiseTrustedEvent(RoutedEventArgs args)
   at System.Windows.UIElement.RaiseEvent(RoutedEventArgs args, Boolean trusted)
   at System.Windows.Input.InputManager.ProcessStagingArea()
   at System.Windows.Input.InputManager.ProcessInput(InputEventArgs input)
   at System.Windows.Input.InputProviderSite.ReportInput(InputReport inputReport)
   at System.Windows.Interop.HwndMouseInputProvider.ReportInput(IntPtr hwnd, InputMode mode, Int32 timestamp, RawMouseActions actions, Int32 x, Int32 y, Int32 wheel)
   at System.Windows.Interop.HwndMouseInputProvider.FilterMessage(IntPtr hwnd, WindowMessage msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at System.Windows.Interop.HwndSource.InputFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
@NickSavin NickSavin changed the title Memory leack in CLRObject Memory leak in CLRObject Jun 14, 2019
filmor pushed a commit that referenced this issue Jun 17, 2019
@lostmsu @filmor
Enable C# parameters of type object accept any argument, passed fro…
43c972d 
…m Python (#853)

* added a regression test for #881

#811

* when converting to object, wrap values of unknown type into PyObject instead of failing

This enables overload resolution with object parameters to behave the same way PyObject parameters behave - e.g. allow any Python object to be passed as PyObject as fallback.

Resolves #811
lostmsu added a commit to losttech/pythonnet that referenced this issue Oct 21, 2019
@lostmsu
added a regression test for pythonnet#881
63ccb6b 
pythonnet#811
lostmsu added a commit to losttech/pythonnet that referenced this issue Feb 27, 2020
@lostmsu
added a regression test for pythonnet#881
7daa47f 
pythonnet#811
filmor pushed a commit that referenced this issue Mar 3, 2020
@lostmsu
Enable C# parameters of type object accept any argument, passed fro…
8ad1062 
…m Python (#889)

* added a regression test for #881
* when converting to object, wrap values of unknown type into PyObject instead of failing

This enables overload resolution with object parameters to behave the same way PyObject parameters behave - e.g. allow any Python object to be passed as PyObject as fallback.

Resolves #811

* fixed ObjectField conversion test
* fixed test_object_indexer to pass on custom class key
* use object() instance in OverloadResolution_UnknownToObject test
@filmor
Copy link
Member

filmor commented Jun 9, 2020

@NickSavin Can you check whether things have improved with the current master/2.5.0rc2?

AlexCatarino pushed a commit to QuantConnect/pythonnet that referenced this issue Jun 18, 2020
@lostmsu @AlexCatarino
Enable C# parameters of type object accept any argument, passed fro…
46df1e4 
…m Python (pythonnet#853)

* added a regression test for pythonnet#881

pythonnet#811

* when converting to object, wrap values of unknown type into PyObject instead of failing

This enables overload resolution with object parameters to behave the same way PyObject parameters behave - e.g. allow any Python object to be passed as PyObject as fallback.

Resolves pythonnet#811
AlexCatarino pushed a commit to QuantConnect/pythonnet that referenced this issue Jun 29, 2020
@lostmsu @AlexCatarino
Enable C# parameters of type object accept any argument, passed fro…
16c4701 
…m Python (pythonnet#889)

* added a regression test for pythonnet#881
* when converting to object, wrap values of unknown type into PyObject instead of failing

This enables overload resolution with object parameters to behave the same way PyObject parameters behave - e.g. allow any Python object to be passed as PyObject as fallback.

Resolves pythonnet#811

* fixed ObjectField conversion test
* fixed test_object_indexer to pass on custom class key
* use object() instance in OverloadResolution_UnknownToObject test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@filmor @NickSavin