Skip to content

add cache_frame_data kwarg into FuncAnimation. fixes #8528. #11894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

pclove1
Copy link
Contributor

@pclove1 pclove1 commented Aug 20, 2018

PR Summary

PR Checklist

  • Has Pytest style unit tests
  • Code is PEP 8 compliant
  • New features are documented, with examples if plot related
  • Documentation is sphinx and numpydoc compliant
  • Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
  • Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

@pclove1 pclove1 force-pushed the fix-memory-leak-in-FuncAnimation branch 2 times, most recently from 05bd0c3 to 6254645 Compare August 20, 2018 05:12
@dopplershift
Copy link
Contributor

But _save_seq isn't unused. It internally caches "data" for the frames for the case where you're generating the animation live off some generative source--or at least that's the idea.

Is this solving a problem you've run into? The discussion in #8528 indicates that removing this did not solve the issue for the user there.

@pclove1
Copy link
Contributor Author

pclove1 commented Aug 21, 2018

Thanks for your feedback, @dopplershift

My understanding is that _save_seq is unnecessary, because artists returned by func in FuncAnimation are responsible for holding references to "data" for them to be re-drawn.

Below is a code snippet which exhibits a memory leak at about 1MB/sec before this fix.

#!/usr/bin/env python3
import weakref

from matplotlib import animation
from matplotlib import pyplot as plt
import numpy as np

def main():
    fig, ax = plt.subplots()

    ax.set_xlim([0, 10])
    ax.set_ylim([0, 1])

    np.random.seed(19680801)

    line, = ax.plot([], [])

    def init():
       line.set_data([], [])
       return line,

    def animate(frame):
       line.set_data(frame['x'], frame['y'])
       return line,

    def frames_generator():
       for _ in range(1000):
           x = np.linspace(0, 10, 10000)
           y = np.random.rand(10000)
           yield { 'x': x, 'y': y }

    anim = animation.FuncAnimation(fig, animate, init_func=init,
                                   frames=frames_generator,
                                   save_count=1000)

    plt.show()

if __name__ == '__main__':
    main()

Another way to interpret this bug is that FuncAnimation was holding all frames in memory until it gets destroyed.

Why does FuncAnimation need to hold all frames even if some of them were already drawn?
Could you help to share code snippet showing why _save_seq is necessary if I'm missing something?

@tacaswell tacaswell added this to the v3.1 milestone Aug 23, 2018
@tacaswell
Copy link
Member

If you call save on an animation that was consuming from a generator, add anim.save(...) to your example.

Copy link
Member

@tacaswell tacaswell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As implemented this is removing the ability to save an animation feed from a generator with the same data you just saw.

Currently the last save_count frames will be saved. This implementation will save the frist save_count of re-calling the generator function. If you passed in a generator object, and let it be exhausted I think this implementation will save no frames.

I do like the test, but it should check that all but save_count have been gc'd.

@tacaswell
Copy link
Member

If you change your generator to run through 10k iterations and change save_count to 10 does the memory use plateau?

We probably should change the logic on save_count to treat non-None user input as the gold standard (not the length of the iterable).

There probably should be a flag to disable caching. It is the right thing in many but not all cases.

@pclove1
Copy link
Contributor Author

pclove1 commented Aug 25, 2018

Thanks for your feedkback, @tacaswell

So, if I understood your comment correctly, Animation is required to support saving its content even after it has already been drawn/shown interactively.
Is this an actual intended/expected behavior?
From user perspective, I think that one would be interested in either drawing it interactively or saving it, not both against the same animation object. I would expect one to create another animation if s/he wants both.

Assuming it's required to support save() even after drawing is complete, I'm afraid to say that this contract forces the implementation of Animation expensive because it needs to keep all frame data.

Having said that, I like your last comment on having a flag to disable caching in FuncAnimation.
If you could help to suggest a good name for the flag, I'm happy to update this PR to respect new flag which wouldn't cache frame data.

@tacaswell
Copy link
Member

So, if I understood your comment correctly, Animation is required to support saving its content even after it has already been drawn/shown interactively.
Is this an actual intended/expected behavior?
From user perspective, I think that one would be interested in either drawing it interactively or saving it, not both against the same animation object. I would expect one to create another animation if s/he wants both.

This is the current behavior (and has been there from when the code came in in 2010 70e401f) so we can not change it with out a deprecation cycle. This is one of the challenges of working on these old, big projects, it is easy to pick any part of the API and say "It would be better if...." but someone is the past thought the current API was a good one and it is likely that there are users depending on that behavior that we can not break without warning (and a good reason). https://xkcd.com/1172/ is real!

Assuming it's required to support save() even after drawing is complete, I'm afraid to say that this contract forces the implementation of Animation expensive because it needs to keep all frame data.

This only gets expensive under some conditions (consuming from a generator that generates and yields objects with a memory footpirnt). itertools.tee has the same problem.

I have personally used this feature when working interactively (make Animation object, watch it on the screen, be happy and do ani.save(...)). Given that you must hold on to the animation object to have it run interactively (or the callbacks get garbage collected), it seems very user hostile to not allow it to also be saved from that object.

Maybe cache_frame_data=True as the kwarg? This should both prevent the caching and cause save to go back to then input.

Could this problem also be solved by passing save_count=0 to FuncAnimation?

@pclove1 pclove1 force-pushed the fix-memory-leak-in-FuncAnimation branch from 6254645 to 017d88b Compare September 3, 2018 22:05
@pclove1 pclove1 changed the title remove unused '_save_seq' data attribute in 'FuncAnimation' class. fixes #8528. add cache_frame_data kwarg into FuncAnimation. fixes #8528. Sep 3, 2018
@pclove1
Copy link
Contributor Author

pclove1 commented Sep 3, 2018

Thanks for your detailed/helpful feedback, @tacaswell
I've updated PR to add cache_frame_data=True as kwarg into FuncAnimation.

@pclove1 pclove1 force-pushed the fix-memory-leak-in-FuncAnimation branch 2 times, most recently from 48c3d9a to ad7103a Compare October 9, 2018 03:33
@pclove1 pclove1 force-pushed the fix-memory-leak-in-FuncAnimation branch from ad7103a to f8b14c8 Compare December 30, 2018 02:12
@tacaswell tacaswell dismissed their stale review December 30, 2018 17:31

Implementation changed.

Copy link
Member

@tacaswell tacaswell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modulo a whats_new entry in https://github.com/matplotlib/matplotlib/tree/master/doc/users/next_whats_new and maybe that extra assertion in the test.

@tacaswell
Copy link
Member

Thanks for following up on this @pclove1 !

@pclove1
Copy link
Contributor Author

pclove1 commented Dec 31, 2018

@tacaswell Thanks for your helpful reviews.
I've added whats_new entry and adopted additional assertion in the test.

.

- add ```2018-12-30-add-cache_frame_data-kwarg-to-FuncAnimation.rst``` into ```doc/users/next_whats_new```
@pclove1 pclove1 force-pushed the fix-memory-leak-in-FuncAnimation branch from 3d4f958 to 1f908e6 Compare January 9, 2019 16:11
@pclove1
Copy link
Contributor Author

pclove1 commented Jan 9, 2019

Squashed commits and rebased against the latest master head

@timhoffm
Copy link
Member

Travis failure is #13137 and unrelated to the PR.

@timhoffm timhoffm merged commit b794828 into matplotlib:master Jan 10, 2019
@timhoffm
Copy link
Member

Thanks, and congratulations on your first contribution to matplotlib. Hope to see you back some time.

@pclove1
Copy link
Contributor Author

pclove1 commented Jan 18, 2019

Like wise.
It was quite meaningful experience for me to contribute to matplotlib.
Thanks for helpful reviews, all of you 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Funcanimation memory leak?
6 participants