Memory leak when plotting multiple figures with the macOS backend #19769

tgrohens · 2021-03-25T16:08:15Z

Bug report

Bug summary

Hi,

I believe I have found a memory issue when plotting multiple figures with the macOS backend.

Here is a minimal working example:

import sys
import matplotlib.pyplot as plt
import numpy as np

import matplotlib
#matplotlib.use('agg')

data = np.random.rand(60, 60)

def plot_expr(plot_name):

    plt.figure(figsize=(9, 8))

    for i in range(60):
        plt.plot(data[i, :])

    plt.savefig(plot_name)

    plt.close('all')


if __name__ == "__main__":

    print(f'python: {sys.version}')
    print(f'matplotlib: {matplotlib.__version__}')
    print(f'backend: {matplotlib.get_backend()}')

    for i in range(50):
        print(f'{i}...')
        plot_expr(f'leak_{i}')

When running the code with memory-profiler, I obtain something like:

Even though I am explicitly calling the plt.close() function.

When using the 'agg' backend instead of the 'MacOSX' backend by uncommenting line 6, I get:

Which I would expect to be the normal behavior.

As I have been unable to reproduce this on Linux, I assume that the issue is with the macOS backend.

Matplotlib version

Operating system: macOS 11.2.1
Matplotlib version 3.3.4
Matplotlib backend MacOSX
Python version: 3.7.9

I installed Python via venv and matplotlib via pip.

Cheers,

Théotime

The text was updated successfully, but these errors were encountered:

jklymak · 2021-03-25T16:56:52Z

does this persist when you use Qt5Agg?
does it persist when you manually invoke garbage collection?

tgrohens · 2021-03-25T17:05:35Z

No, the leak does not seem to persist with Qt5Agg (with or without manual GC).
Yes, I have the issue even after adding gc.collect() after the plt.close() call.

tacaswell · 2021-03-25T22:21:48Z

If you throw a plt.pause(.1) into the loop does the memory get cleared? We have seen issues where if you are creating figures that in turn create GUI objects at the c/c++/objectivec level. Depending exactly how the event loops work under the hood we can have the situation:

create figure (allocate memory)
destroy the figure (do not actually destroy the GUI objects, but just tell the event loop you would like to destroy those objects)
the event loop notices your request, makes sure it is fully safe, tears down the GUI object and releases the memory

but, if you are running this in a tight loop like the example you are never letting the event loop run so you have N GUI windows hanging out waiting to be cleared.

tgrohens · 2021-03-26T10:20:19Z

Hey,

Nope, even with a plt.pause(1.0) thrown in I still get the same behavior.

anntzer · 2021-03-26T11:57:46Z

Perhaps try using objgraph.show_growth() (https://mg.pov.lt/objgraph/#memory-leak-example) to see whether it's Python objects leaking or (Obj)C ones?

tgrohens · 2021-03-26T12:12:14Z

I certainly can do that.

Here's the result for the last two iterations of the loops (deltas are the same from one iteration to the next), with limit=25:

48...
dict                          87426     +1680
weakref                       47218      +922
tuple                         43318      +801
function                      50804      +799
builtin_function_or_method    37394      +735
list                          27544      +472
cell                          14544      +257
method                        12511      +254
_XYPair                        9359      +191
Path                           8938      +182
CompositeGenericTransform      8183      +167
Bbox                           7694      +156
WeakMethod                     6223      +127
Affine2D                       5733      +117
Line2D                         5733      +117
MarkerStyle                    5733      +117
IdentityTransform              4362       +89
BboxTransformTo                3822       +78
TransformedBbox                3773       +77
TransformedPath                3577       +73
FontProperties                 2205       +45
Text                           2205       +45
ScaledTranslation               637       +13
XTick                           490       +10
YTick                           441        +9

49...
dict                          89106     +1680
weakref                       48140      +922
tuple                         44119      +801
function                      51603      +799
builtin_function_or_method    38129      +735
list                          28016      +472
cell                          14801      +257
method                        12765      +254
_XYPair                        9550      +191
Path                           9120      +182
CompositeGenericTransform      8350      +167
Bbox                           7850      +156
WeakMethod                     6350      +127
Affine2D                       5850      +117
Line2D                         5850      +117
MarkerStyle                    5850      +117
IdentityTransform              4451       +89
BboxTransformTo                3900       +78
TransformedBbox                3850       +77
TransformedPath                3650       +73
FontProperties                 2250       +45
Text                           2250       +45
ScaledTranslation               650       +13
XTick                           500       +10
YTick                           450        +9

It does look like matplotlib objects to me.

dstansby · 2021-12-21T16:23:49Z

I did a bit of digging, and the back reference graphs I get tracing from Figure have only one difference between Agg and macosx. FigureManagerMac has a ref count of 3, whereas FigureManagerBase a ref count of 2.

So I suspect that a reference to FigureManagerMac is hanging around somewhere in the C layer, because it's not showing up on the python reference graphs. I guess this is probably somewhere in _macosx.FigureManager.

Here's the code I'm using if anyone's interested:

import gc
import random

import matplotlib.pyplot as plt
import objgraph

import matplotlib
matplotlib.use('macosx')


def plot_expr(plot_name):

    fig = plt.figure(figsize=(9, 8))
    plt.close('all')
    objgraph.show_growth(limit=50)

    if plot_name == 'leak_2':
        objgraph.show_backrefs(
            random.choice(objgraph.by_type('Figure')),
            filename='chain_mac.png', max_depth=50, refcounts=True)


if __name__ == "__main__":
    for i in range(3):
        print(f'{i}...')
        plot_expr(f'leak_{i}')

dstansby · 2021-12-21T16:30:30Z

I don't think I have the knowledge to track this down in the C layer (if that's indeed where it is), but FWIW this issue is present at least back to Matplotlib v3.2.2.

QuLogic · 2021-12-22T01:51:41Z

Is the reference count the same with Qt backends?

There is a reference held in the ObjC side on the Window class. It is taken in initWithContentRect and removed in dealloc. Maybe there is something related to the lifecycle issue in #21788.

dstansby · 2023-01-27T23:07:43Z

Coming back to this, the growth I'm getting with the code in #19769 (comment) is

dict                           6635       +50
list                           4643       +19
ReferenceType                  3711       +13
tuple                          7059       +12
cell                           4414       +12
function                      12858       +11
_StrongRef                       27        +9
method                           81        +6
count                            22        +6
set                             701        +5
CallbackRegistry                 15        +5
WeakMethod                       15        +5
builtin_function_or_method     2827        +3
Grouper                          10        +2
_XYPair                           6        +2
FigureManagerMac                  3        +1
TimerMac                          3        +1
NavigationToolbar2Mac             3        +1
Stack                             3        +1
FigureCanvasMac                   3        +1
LockDraw                          3        +1
Figure                            3        +1
Bbox                              3        +1
Affine2D                          3        +1
TransformedBbox                   3        +1
BboxTransformTo                   3        +1
Rectangle                         3        +1
SubplotParams                     3        +1
_AxesStack                        3        +1

And the chain for the canvas is below. It looks like there might be an issue with TimerMac being in a circular reference with a __del__ defined (in red below)? Again I'm far from an expert, but might be worth someone who understands the mac code checking that TimerMac (fully implemented in the C layer) doesn't have any memory issues?

greglucas · 2023-04-27T14:25:20Z

This was fixed by: #23059

QuLogic added GUI: MacOSX Performance labels Mar 25, 2021

dstansby added the status: confirmed bug label Dec 21, 2021

neutrinoceros mentioned this issue Feb 16, 2022

BUG: memory leak while plotting in a loop yt-project/yt#3513

Closed

k1m190r mentioned this issue Apr 21, 2022

[BUG] sporadic test failure of test_evaluator test_plots on Mac Python 3.7 sktime/sktime#2368

Open

This was referenced May 18, 2022

FIX: Decrease figure refcount on close of a macosx figure #23059

Merged

[Bug]: macosx timers don't fire if plt.show() hasn't been called #23061

Closed

greglucas closed this as completed Apr 27, 2023

QuLogic added this to the v3.8.0 milestone Apr 27, 2023

billetd mentioned this issue Jan 25, 2024

BUG: Memory leak with matplotlib MacOSX backend, version 3.8.0 and below SuperDARN/pydarn#370

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory leak when plotting multiple figures with the macOS backend #19769

Memory leak when plotting multiple figures with the macOS backend #19769

tgrohens commented Mar 25, 2021 •

edited

Loading

jklymak commented Mar 25, 2021

tgrohens commented Mar 25, 2021

tacaswell commented Mar 25, 2021

tgrohens commented Mar 26, 2021

anntzer commented Mar 26, 2021

tgrohens commented Mar 26, 2021

dstansby commented Dec 21, 2021 •

edited

Loading

dstansby commented Dec 21, 2021

QuLogic commented Dec 22, 2021

dstansby commented Jan 27, 2023

greglucas commented Apr 27, 2023

Memory leak when plotting multiple figures with the macOS backend #19769

Memory leak when plotting multiple figures with the macOS backend #19769

Comments

tgrohens commented Mar 25, 2021 • edited Loading

Bug report

jklymak commented Mar 25, 2021

tgrohens commented Mar 25, 2021

tacaswell commented Mar 25, 2021

tgrohens commented Mar 26, 2021

anntzer commented Mar 26, 2021

tgrohens commented Mar 26, 2021

dstansby commented Dec 21, 2021 • edited Loading

dstansby commented Dec 21, 2021

QuLogic commented Dec 22, 2021

dstansby commented Jan 27, 2023

greglucas commented Apr 27, 2023

tgrohens commented Mar 25, 2021 •

edited

Loading

dstansby commented Dec 21, 2021 •

edited

Loading