[ENH]: Does pyplot need to call switch_backend #22739

hmaarrfk · 2022-03-31T03:29:19Z

Problem

It seems that simply importing pyplot can have serious effects on what backends are chosen, and how they interact with one and other.

In

matplotlib/lib/matplotlib/__init__.py

Line 1137 in da9533d

# if pyplot is not already imported, do not import it. Doing

They say that simply

from matplotlib import pyplot

can trigger setting the default backend.

This is somewhat frustrating for people who use interactive sessions.

Running the code

from matplotlib import pyplot
import matplotlib
matplotlib.use("WxAgg")

all at once will work in ipython (from the terminal). But running it one line at a time, won't.

This is somewhat strange.

Now that things like __getattr__ exist, would be possible to further delay setting the default backend?

The issue arises when you want to build an application, and other people's code might be imported before yours. You are then at their mercy to not set the backend (implicitly!) before you set yours.

Proposed solution

No solution. I'm sure there is some strange historic thing I'm missing.

Honestly, you created a great API with pyplot. People use it to get colormaps, and for all sorts of things.

XREF: DeepLabCut/DeepLabCut#1730

The text was updated successfully, but these errors were encountered:

WeatherGod · 2022-03-31T03:59:42Z

The fact that pyplot triggers the backend creation is documented, and so is the fact that this causes side-effects. EDIT: At least, we used to extensively document this... I was about to post a bunch of links from the docs talking about this, but now I can't find the relevant information. The best I can get is https://matplotlib.org/stable/users/explain/interactive.html That being said, work is ongoing to defer the backend initialization to later points in the figure lifetime. Perhaps you can help with that effort?

…

On Wed, Mar 30, 2022 at 11:29 PM Mark Harfouche ***@***.***> wrote: Problem It seems that simply importing pyplot can have serious effects on what backends are chosen, and how they interact with one and other. In https://github.com/matplotlib/matplotlib/blob/da9533d507c1f06ceee5078c9cbbd99897809d31/lib/matplotlib/__init__.py#L1137 They say that simply from matplotlib import pyplot can trigger setting the default backend. This is somewhat frustrating for people who use interactive sessions. Running the code from matplotlib import pyplot import matplotlib matplotlib.use("WxAgg") all at once will work in ipython (from the terminal). But running it one line at a time, won't. This is somewhat strange. Now that things like __getattr__ exist, would be possible to further delay setting the default backend? The issue arises when you want to build an application, and other people's code might be imported before yours. You are then at their mercy to not set the backend (implicitly!) before you set yours. Proposed solution No solution. I'm sure there is some strange historic thing I'm missing. Honestly, you created a great API with pyplot. People use it to get colormaps, and for all sorts of things. XREF: DeepLabCut/DeepLabCut#1730 <DeepLabCut/DeepLabCut#1730> — Reply to this email directly, view it on GitHub <#22739>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACHF6CHT52IKVM6YRCD2RTVCULZXANCNFSM5SD7ZBWA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

hmaarrfk · 2022-03-31T04:29:04Z

ok, understood. I didn't know if patches would be welcome that would defer that.

Is there an API that application developers can use that is independent of the chosen backend? That is, they would specify their own handles/backend name and matplotlib will forcibly use that?

timhoffm · 2022-03-31T06:43:36Z

Deferring backed selection has been implemented in #22005, but is not yet released.

Is there an API that application developers can use that is independent of the chosen backend?

I'm not the backend expert here, but I'm not aware of a reasonably simple way to do this.

General question: Is there a reason that we need global exclusive backend state and cannot have a per-figure backend setting?

jklymak · 2022-03-31T07:14:43Z

General question: Is there a reason that we need global exclusive backend state and cannot have a per-figure backend setting?

I don't know - is it really worth the trouble to allow multiple GUI backends to run at the same time? I could see us allowing one GUI to be closed and a different one opened after the GUI is closed, but I'm not sure its a good idea for one python process to have multiple GUI frameworks running at the same time?

We probably should revisit the roadmap for https://github.com/matplotlib/mpl-gui

hmaarrfk · 2022-03-31T12:01:38Z

Well... I feel like it might be address in the next release.
#22005

hmaarrfk · 2022-03-31T12:01:57Z

Can't wait!

timhoffm · 2022-03-31T20:31:19Z

@hmaarrfk does #22005 address your needs? If so we can close this issue.

hmaarrfk · 2022-03-31T21:37:27Z

I'm not entirely sure. It is kinda hard to compile matplotlib and backporting the patch wasn't super easy.

There still seems to be eager code in:
https://github.com/matplotlib/matplotlib/blob/main/lib/matplotlib/pyplot.py#L2246

so i really wnat to have the time to test before saying it is fixed.

hmaarrfk · 2022-03-31T21:38:46Z

The comment I referred to in the init file seems suspicious now too.

anntzer · 2022-03-31T22:04:51Z

Is there a reason that we need global exclusive backend state and cannot have a per-figure backend setting?

You cannot have e.g. both qt and tk running as they'll fight over https://docs.python.org/3/c-api/veryhigh.html#c.PyOS_InputHook. You could allow e.g. qtcairo + qtagg or any interactive backend + any noninteractive backend, but I'm not sure it's worth the engineering...

timhoffm · 2022-03-31T22:52:04Z

You cannot have e.g. both qt and tk running as they'll fight over https://docs.python.org/3/c-api/veryhigh.html#c.PyOS_InputHook.

I can if I have different processes 😝

tacaswell · 2022-03-31T22:55:19Z

matplotlib/lib/matplotlib/pyplot.py

Lines 2246 to 2250 in d8ede1a

    
           if (rcParams["backend_fallback"] 
        
                   and dict.__getitem__(rcParams, "backend") in ( 
        
                       set(_interactive_bk) - {'WebAgg', 'nbAgg'}) 
        
                   and cbook._get_running_interactive_framework()): 
        
               dict.__setitem__(rcParams, "backend", rcsetup._auto_backend_sentinel)

says:

if the user asked for fallback (meaning get a working gui not fail on the gui they asked for)
and the backend is one of ours that we know uses a GUI toolkit (but excluding the web backends)
and would that there is already a running event loop (but not starting an event loop)

then set the backend to "automatic" and let the later machinery sort out detecting which one is running and doing the correct selection then.

tacaswell · 2022-03-31T23:03:45Z

matplotlib/lib/matplotlib/__init__.py

Lines 1137 to 1155 in da9533d

    
           # if pyplot is not already imported, do not import it.  Doing 
        
           # so may trigger a `plt.switch_backend` to the _default_ backend 
        
           # before we get a chance to change to the one the user just requested 
        
           plt = sys.modules.get('matplotlib.pyplot') 
        
           # if pyplot is imported, then try to change backends 
        
           if plt is not None: 
        
               try: 
        
                   # we need this import check here to re-raise if the 
        
                   # user does not have the libraries to support their 
        
                   # chosen backend installed. 
        
                   plt.switch_backend(name) 
        
               except ImportError: 
        
                   if force: 
        
                       raise 
        
           # if we have not imported pyplot, then we can set the rcParam 
        
           # value which will be respected when the user finally imports 
        
           # pyplot 
        
           else: 
        
               rcParams['backend'] = backend

This code is saying "if the user explicitly called mpl.use(...) then make sure right now that is something we can actually do. That comment needs to be adjusted now that #22005 is merged.

I am morbidly curious why it was going wrong before but I do think it is fixed.

I'm not entirely sure. It is kinda hard to compile matplotlib and backporting the patch wasn't super easy.

Where are you trying to backport it to?

If you can compile skimage, I would also expect Matplotlib to compile "out of the box" as well...

tacaswell · 2022-04-01T00:02:09Z

And thank to #22733 which just got merged

python -m pip install --upgrade --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple matplotlib

will now get you nightlies!

ianhi · 2022-04-01T00:49:32Z

Slight correction to that install command. To get everything to work properly with dependencies you need to run:

python -m pip install --upgrade --pre --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple --extra-index-url https://pypi.org/simple matplotlib

hmaarrfk · 2022-04-01T00:56:40Z

If you can compile skimage, I would also expect Matplotlib to compile "out of the box" as well...

Touchée .

I guess I'm using conda for most environment management.
It becomes a pain to remove matplotlib-core and replace it with matplotlib

I'll give it a try in a few hours.

WeatherGod · 2022-04-01T01:16:42Z

Is there an API that application developers can use that is independent of the chosen backend? That is, they would specify their own handles/backend name and matplotlib will forcibly use that?

You might be interested in my book, "Interactive Applications using Matplotlib". It is a little out of date now (published almost 7 years ago!), but many of the principles are still true. In it, it describes how to embed a figure as a widget in an existing GUI application. It also shows how to embed GUI widgets into a regular matplotlib figure window.

Furthermore, it also talks about some of the GUI-agnostic widgets that are available so that you can have an application that'll work in just about any interactive environment.

hmaarrfk · 2022-04-01T02:37:26Z

I'm not going to say I can't get it to compile from source, but I am going to say that for my conda environment, I tried:

conda uninstall --force --offline --yes matplotlib base
pip install -e . -vv --no-deps

and it resulted in:

$ ipython
Python 3.9.9 (main, Dec 29 2021, 07:47:36)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from matplotlib import pyplot
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from matplotlib import pyplot

File ~/git/matplotlib/lib/matplotlib/__init__.py:208, in <module>
    203         if parse_version(module.__version__) < parse_version(minver):
    204             raise ImportError(f"Matplotlib requires {modname}>={minver}; "
    205                               f"you have {module.__version__}")
--> 208 _check_versions()
    211 # The decorator ensures this always returns the same handler (and it is only
    212 # attached once).
    213 @functools.lru_cache()
    214 def _ensure_handler():

File ~/git/matplotlib/lib/matplotlib/__init__.py:193, in _check_versions()
    189 def _check_versions():
    190
    191     # Quickfix to ensure Microsoft Visual C++ redistributable
    192     # DLLs are loaded before importing kiwisolver
--> 193     from . import ft2font
    195     for modname, minver in [
    196             ("cycler", "0.10"),
    197             ("dateutil", "2.7"),
   (...)
    200             ("pyparsing", "2.2.1"),
    201     ]:
    202         module = importlib.import_module(modname)

ImportError: /home/mark/git/matplotlib/lib/matplotlib/ft2font.cpython-39-x86_64-linux-gnu.so: undefined symbol: FT_Load_Glyph

Kinda just as hard as I expected it to be (i.e. not to work on the first try).

The link provided on anaconda installed successfully.

Finally, it does resolve the problem of allowing you to select the WxAgg backend.

I then uninstalled it, and reinstalled the version from conda-forge, and it seems to work!

Thanks all!

tacaswell · 2022-04-01T02:41:52Z

Oh, that makes sense 😞 . We build our own freetype pinned to a very old version (so the tests will pass because every version of freetype moves the text around a tiny bit) but conda provides a newer one. I suspect at runtime you ended up with crossed libraries....

I will not claim to understand what the wheel is doing under the hood to make it work though!

hmaarrfk · 2022-04-01T03:02:17Z

@tacaswell thank you for the hint about freetype. using a slightly modified build script from conda-forge
https://github.com/conda-forge/matplotlib-feedstock/blob/main/recipe/build_base.sh
it does indeed import.

Time for me to sleep. Goodnight

hmaarrfk added the New feature label Mar 31, 2022

hmaarrfk closed this as completed Apr 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH]: Does pyplot need to call switch_backend #22739

[ENH]: Does pyplot need to call switch_backend #22739

hmaarrfk commented Mar 31, 2022

WeatherGod commented Mar 31, 2022 via email

hmaarrfk commented Mar 31, 2022

timhoffm commented Mar 31, 2022

jklymak commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

timhoffm commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

anntzer commented Mar 31, 2022

timhoffm commented Mar 31, 2022

tacaswell commented Mar 31, 2022

tacaswell commented Mar 31, 2022

tacaswell commented Apr 1, 2022

ianhi commented Apr 1, 2022

hmaarrfk commented Apr 1, 2022

WeatherGod commented Apr 1, 2022

hmaarrfk commented Apr 1, 2022

tacaswell commented Apr 1, 2022

hmaarrfk commented Apr 1, 2022

[ENH]: Does pyplot need to call switch_backend #22739

[ENH]: Does pyplot need to call switch_backend #22739

Comments

hmaarrfk commented Mar 31, 2022

Problem

Proposed solution

WeatherGod commented Mar 31, 2022 via email

hmaarrfk commented Mar 31, 2022

timhoffm commented Mar 31, 2022

jklymak commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

timhoffm commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

hmaarrfk commented Mar 31, 2022

anntzer commented Mar 31, 2022

timhoffm commented Mar 31, 2022

tacaswell commented Mar 31, 2022

tacaswell commented Mar 31, 2022

tacaswell commented Apr 1, 2022

ianhi commented Apr 1, 2022

hmaarrfk commented Apr 1, 2022

WeatherGod commented Apr 1, 2022

hmaarrfk commented Apr 1, 2022

tacaswell commented Apr 1, 2022

hmaarrfk commented Apr 1, 2022