Skip to content

MEP27 Decouple pyplot from backends (refactoring Gcf out of backend code) #4143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 55 commits into from

Conversation

OceanWolf
Copy link
Member

@OceanWolf OceanWolf commented Feb 22, 2015

This PR exists as an alternative to #2624

See https://github.com/OceanWolf/matplotlib/blob/backend-refactor/doc/devel/MEP/MEP27.rst for details.

Tl;dr:
FigureManagerBase ➡️ FigureManager + WindowBase
ShowBase ➡️ Gcf.show_all + MainLoopBase

The main refactor (+gtk3 as an example) goes here, and will have no effect on the individual backends, work for other backends will occur in separate branches as indicated.

To accomplish this, we shall do this in two passes:

  1. The first pass we refactor out all generic backend code out of the gtk3 backend, replacing these with common methods, i.e. we refactor out Gcf to backend_bases.py.
  2. The second pass we then refactor Gcf out from backend_bases.py.
  3. Other backends in separate branches, see list above for progress and make changes to this PR for global changes (none since the 2nd backend).

@OceanWolf
Copy link
Member Author

@tacaswell Not finished yet, still some refactoring to go, just putting the PR for early feedback to catch things that I have perhaps missed. I have finished the first pass (out of 2, I hope), so it should all work.

I have worked solely on the GTK3Cairo backend so far. All other backends should work as normal, though I haven't tested (and I still see no Travis here).

@OceanWolf
Copy link
Member Author

Atm, I use the existing structure, i.e. using new_figure_manager to pass the classes we want to use. However I do like the way we get the new_figure_manager in matplotlib/backends/__init__.py and thought about getting the specific classes we want to use directly from there, something like:

from matplotlib.backends import Window, Toolbar2, Canvas

and in matplotlib.backends.__init__.py put:

_temp = __import__('backend_'+backend, globals(), locals(), ['Window', 'Toobar2', 'Canvas'], -1)
Window = _temp.Window
Toolbar2 = _temp.Toolbar2
Canvas = _temp.Canvas

@tacaswell
Copy link
Member

I provisionally think this is a good idea. This deserves more attention than I can give it right now, sorry 😞

Would this method also provide a (sane) path to change which interactive backend you are using on the fly?

I don't think it will, but make sure that this does not interfere with how we do saving (which is (more-or-less) by replacing the canvas on the Figure object and re-calling draw and then restoring the old canvas).

@OceanWolf
Copy link
Member Author

Thank you, I shall push ahead then with the additions to matplotlib.backends.__init__.py (which I think looks a lot nicer, tucking all the backend specific code properly away behind the scenes); and the next Refactor pass.

Perhaps not a sane path, (at least not right now), but I guess it will provide a saner path for later (as with this we get some clear structure to how the backends work). I have never tried to change backends and so don't know the current path, so I don't really know.

I imagine that with the additions to __init__.py we can do away with the backend new_figure_manager function, just leaving it there for a deprecation cycle.

@OceanWolf
Copy link
Member Author

@tacaswell Almost finished, just finishing the refactoring Gcf out of matplotlib.backend_bases.ShowBase, tough cookie, but I have managed it quite well I think, especially with respect to switching backends (I now have it with very few lines of code, to switch backends without having to close the previous backend -- if wanted --, the refactor enables it to magically take care of all of that :). I haven't touched the backend switching code, but have built in flexibility to add it later).

One question arises though, should the ShowBase callable instance (aka backends.backend_xxx.show, get called from anywhere, or just pyplot/Gcf? I need to get an idea of the use-cases of this (Note I have refactored ShowBase as FigureManagerBase, but I use the old terms so that we both speak the same language). I just need to come up with a suitable place(s) to put the code. Having Gcf everywhere makes it difficult to figure out to what extent it should have a role ;).

If show should only get called via Gcf, then I shall probably create a method Gcf.show_all() that calls the refactored show.

@OceanWolf
Copy link
Member Author

I mean I feel 95% sure, that the backend show() callable-class should only get called by Gcf, just because afaik, we have no other way of tracking FigureManagers, so to show all figures, seems impossible.

Okay, I think have talked myself round to believing my assumption as correct and will give it a try. If I have overlooked something here please let me know!

@OceanWolf OceanWolf force-pushed the backend-refactor branch 2 times, most recently from d822f38 to 7dba22f Compare February 23, 2015 00:53
@tacaswell tacaswell added this to the next point release milestone Feb 23, 2015
@OceanWolf
Copy link
Member Author

Okay, so I have finished the 2nd refactor pass so open for comments, I have tested examples/slider_demo on both Gtk3Cairo which I have refactored, and on TkAgg which I haven't, and both work fine, so I feel happy!

I did have problems with absolute imports due to __init__.py, but wrapping it in a function fixed that.

Will feel good to get this landed as it will eradicate the backend hacks from #3652. Of course I imagine changes to WindowBase will most likely occur, but personally I like to learn as I go, when problems present themselves. Too many different backend mechanisms to study in one go.

@OceanWolf OceanWolf changed the title Refactoring Gcf out of backend code (W-I-P) Refactoring Gcf out of backend code Feb 23, 2015
@OceanWolf OceanWolf force-pushed the backend-refactor branch 2 times, most recently from 94b7bce to 3cfaec1 Compare February 23, 2015 12:37
@OceanWolf
Copy link
Member Author

Yay Travis working again, but boo failing, did I introduce an error here, or does this represent a problem with the testing framework?

Looking at the log for 2.6 I get the following, and the IOError: Conversion command failed: makes it look like a testing framework bug to me (the pickling bug comes from figure.Figure I think, I shall investigate).

======================================================================
ERROR: test suite for <class 'matplotlib.tests.test_pickle.test_complete'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/plugins/multiprocess.py", line 788, in run
    self.setUp()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/plugins/multiprocess.py", line 770, in setupContext
    super(NoSharedFixtureContextSuite, self).setupContext(context)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/util.py", line 470, in try_run
    return func()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 134, in setup_class
    cls._func()
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/tests/test_pickle.py", line 193, in test_complete
    assert_not_equal(plt._pylab_helpers.Gcf.figs, {})
AssertionError: {} == {}

======================================================================
ERROR: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 51, in failer
    result = f(*args, **kwargs)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 186, in do_test
    self._tol, in_decorator=True)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/compare.py", line 310, in compare_images
    expected = convert(expected, True)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/compare.py", line 192, in convert
    converter[extension](filename, newname)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/compare.py", line 118, in convert
    raise IOError(msg)
IOError: Conversion command failed:

======================================================================
FAIL: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 51, in failer
    result = f(*args, **kwargs)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 196, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: /home/travis/build/matplotlib/tmp_test_dir/result_images/test_patheffects/collection.png vs. /home/travis/build/matplotlib/tmp_test_dir/result_images/test_patheffects/collection-expected.png (RMS 31.876)

======================================================================
FAIL: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 51, in failer
    result = f(*args, **kwargs)
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 196, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: /home/travis/build/matplotlib/tmp_test_dir/result_images/test_patheffects/collection_pdf.png vs. /home/travis/build/matplotlib/tmp_test_dir/result_images/test_patheffects/collection-expected_pdf.png (RMS 38.061)

@OceanWolf OceanWolf force-pushed the backend-refactor branch 3 times, most recently from ceec49f to 61a7090 Compare February 26, 2015 16:30
@OceanWolf
Copy link
Member Author

Okay, now that Travis gives the thumbs up to this PR, I now see it time to start pushing out to other backends.

The strategy for this:

  1. I want to keep this PR open for the moment as a meta/base PR to track the rest of the backends and create branches from this to do add in the necessary code.
  2. Changes to the base code will get made to this PR to round off any nasty GTK-isms I have used.
  3. In the first comment I have put a list of backends to convert and their status, as a backend gets refactored, I will tick it off and add a link to the diff between that branch and this.
  4. I have ordered the list of backends in the order I plan to refactor (starting from the bottom), in a way which I think will give the fastest polishing of the base code contained in this PR. The sooner it gets done, the more time for feedback on the final version of this PR (suggestions welcome for PRs backends I should look at sooner, or if anyone wants to create their own branches to get this done sooner then let me know, I will mark a backend as w-i-p when I begin working on it).

@OceanWolf
Copy link
Member Author

@tacaswell Things look good, I should have all the backends (apart from the two mac ones) converted by the end of the weekend. I don't use macs, only linux and sparingly win. Does it matter if they don't get converted right now? Or I could try and code it blind.

Also anything I should do to ease the review? @fariza suggested opening an MEP for this.

@OceanWolf
Copy link
Member Author

Gah, help, why does this not build? I cannot see what I have changed since the last okay from travis that could have caused this build doc failiure...

The build gives the following as one line.

matplotlib/lib/matplotlib/backend_bases.py:docstring of
matplotlib.backend_bases.WindowBase.destroy:2:
WARNING: Inline emphasis start-string without end-string.

@jenshnielsen
Copy link
Member

I think you need to escape the * in that doc string TEXT is default for emphasising text in RST so it complains that you start an emphasised section but not end it. See http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#emphasis

A backslash should do

@OceanWolf
Copy link
Member Author

Thanks @jenshnielsen , that has fixed it :D. On a side note, when I run make.py html it gives me an intermittent error (every other build) Expected 2-dimensional array, got 1 on mplot3d/surface3d_demo.py Anything to do with #3675?

@WeatherGod
Copy link
Member

Ok, now that is weird. surface3d_demo.py explicitly computes its own data:

X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X**2 + Y**2)
Z = np.sin(R)

That is always 2d, no matter what. It would be helpful to know exactly
where the warning is emitted from, if possible. Even stranger that it
happens every other time.

On Tue, Mar 3, 2015 at 12:03 PM, OceanWolf notifications@github.com wrote:

Thanks @jenshnielsen https://github.com/jenshnielsen , that has fixed
it :D. On a side note, when I run make.py html it gives me an
intermittent error (every other build) Expected 2-dimensional array, got 1
on mplot3d/surface3d_demo.py Anything to do with #3675
#3675?


Reply to this email directly or view it on GitHub
#4143 (comment)
.

@jenshnielsen
Copy link
Member

I think I have seen that error when building the docs. It happened because I had a leftover old version of mpl_toolkits in my sitepackages folder from before everything was moved to the matplotlib folder.

@OceanWolf
Copy link
Member Author

Define nights... the international nature of this project means we should discuss times in UTC (Universal Coordinated Time).

@fariza
Copy link
Member

fariza commented Aug 9, 2016

In that case I'm UTC-4

So the best for me is after UTC 1:00am

On Aug 9, 2016 8:15 AM, "OceanWolf" notifications@github.com wrote:

Define nights... the international nature of this project means we should
discuss times in UTC (Universal Coordinated Time).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#4143 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABa86ch9oftmGW2vu8SdGztccoPbPxmAks5qeG9dgaJpZM4Dj1l2
.

self._is_gui = hasattr(self._backend, 'Window')
if not self._is_gui:
self.window = None
return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think return here is going to work (init returns the object not a value/None).

What about moving the gui setup (window, toolbar, ....) to another method, that doesn't get called if is not gui.?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

give me copy paste code to show that it doesn't work... __init__ works as a normal method, i.e. it has an implicit return, i.e. return None at the end...

I think you get confused with __new__ as new does that... not sure what object.new looks like but you get the gist from this:

def __new__(class, *args, **kwargs):
  instance = super().__new__(class, *args, **kwargs)  # calls __init__(self, *args, **kwargs)
  return instance

basically:

  1. From the code MyClass(*args, **kwargs)
    2.__new__(MyClass, *args, **kwargs) gets called.
  2. That magically creates the instance and calls __init__ on it.
  3. Returns the instance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got confused because init must return None.
I just remembered that we shouldn't return things from init ;)

Copy link
Member Author

@OceanWolf OceanWolf Aug 9, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, because it happens with __new__, you can do fun things with __new__...

class NaughtyBoy(object):
  def __init__(self):
    print('Just a naughty boy')

class Messiah(object):
  def __init__(self):
    print('Hallelujah!')

  def __new__(cls, *args, **kwargs):
    print('Not the Messiah')
    return NaughtyBoy()

person = Messiah()
print('I wanted the Messiah, but I got a', type(person))

@OceanWolf
Copy link
Member Author

At the moment I work on UTC+2, so a bit difficult for us arrange a time... I guess I could work a night...

@fariza
Copy link
Member

fariza commented Aug 15, 2016

Weekend before UTC 2pm?

On Aug 15, 2016 6:37 PM, "OceanWolf" notifications@github.com wrote:

At the moment I work on UTC+2, so a bit difficult for us arrange a time...
I guess I could work a night...


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#4143 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABa86YIb996EWg63mCWKMhzwxi3K8itxks5qgOougaJpZM4Dj1l2
.

@fariza
Copy link
Member

fariza commented Sep 28, 2016

@tacaswell @mdboom @OceanWolf can we look for a time and date to take a look at this PR?

I understand that there might be problems/corrections with the specific implementation, that is something we can work on.
But... I was wondering if there is any concern/resistance/dislike with the overall approach

@fariza
Copy link
Member

fariza commented Oct 21, 2016

@tacaswell Can we include this in one of the weekly phone calls?

@efiring
Copy link
Member

efiring commented Oct 22, 2016

A couple weeks ago we discussed this on the Monday conference and agreed that this work is the primary target for 2.1, so it has high priority as soon as 2.0 is out.

@fariza
Copy link
Member

fariza commented Oct 25, 2016

Great!

On Oct 22, 2016 4:17 PM, "Eric Firing" notifications@github.com wrote:

A couple weeks ago we discussed this on the Monday conference and agreed
that this work is the primary target for 2.1, so it has high priority as
soon as 2.0 is out.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#4143 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABa86XdXdoW1s8J8DcAcfPOKZMxLJH8Wks5q2m9qgaJpZM4Dj1l2
.

@tacaswell
Copy link
Member

I have rebased this with the same name in my repo.

@fariza
Copy link
Member

fariza commented Apr 26, 2017 via email

@WeatherGod
Copy link
Member

WeatherGod commented Apr 27, 2017 via email

@fariza
Copy link
Member

fariza commented Apr 29, 2017 via email

@OceanWolf
Copy link
Member Author

Still out of it for a bit, hope to get back in to the groove soon.

@anntzer
Copy link
Contributor

anntzer commented Jun 19, 2017

Instead of MainLoop.begin and MainLoop.end, I think it may be more pythonic to make MainLoop a contextmanager (__enter__ and __end__).

@tacaswell tacaswell modified the milestones: 2.1 (next point release), 2.2 (next next feature release) Sep 24, 2017
@jklymak
Copy link
Member

jklymak commented May 30, 2018

Closing in lieu of #8777 Not sure if #8777 will ever go in, but...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
MEP: MEP22 tool manager MEP: MEP27 decouple pyplot from backends
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants