Update virtualenv + OSX + conda guide #6418

egpbos · 2016-05-13T15:27:55Z

I was working on a pull request (in version 1.5.1+1707.g506df7b.dirty) on OSX (10.9.5) with Python 2.7 in a Conda environment. Using the current virtualenv FAQ (http://matplotlib.org/devdocs/faq/virtualenv_faq.html, specifically the frameworkpython script solution), I was not able to load matplotlib.

In the discussion of the pull request, @efiring suggested a procedure (#6358 (comment)) that did work after adding one final step. Inside the Conda enviroment you want to develop matplotlib in, the procedure is as follows (mostly copied from @efiring):

Use conda to make a test environment with the desired python version, and use conda to install matplotlib in it.
Use conda to uninstall matplotlib.
In your github clone of matplotlib, make a feature branch, and make the desired changes to it.
In the root of that clone directory run pip install ..
Install the Framework version of python with conda install python.app.
Open a new terminal and activate the conda environment (or run hash -r (bash) / rehash (zsh) in the old terminal to make sure your shell finds the conda version of pythonw instead of the OSX version).
Use pythonw instead of python from now on.

The text was updated successfully, but these errors were encountered:

tacaswell · 2016-05-13T15:40:45Z

Can you put that text into a PR into the documentation?

egpbos · 2016-05-15T15:06:26Z

Actually, two more steps are necessary:

Edit $CONDA_ENV_PATH/bin/activate; at the bottom of the file add:

export OLD_PYTHONPATH=$PYTHONPATH
export OLD_DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH
export PYTHONPATH=$CONDA_ENV_PATH/lib/python2.7/site-packages
export DYLD_FALLBACK_LIBRARY_PATH=$CONDA_ENV_PATH/lib

Edit $CONDA_ENV_PATH/bin/deactivate; at the top of the file add:

export PYTHONPATH=$OLD_PYTHONPATH
export DYLD_FALLBACK_LIBRARY_PATH=$OLD_DYLD_FALLBACK_LIBRARY_PATH
unset OLD_PYTHONPATH
unset OLD_DYLD_FALLBACK_LIBRARY_PATH

I'll write it in the docs some time soon.

jenshnielsen · 2016-05-15T15:19:04Z

Conda supports putting activation scrips in a activate.d and deactivate.d directory http://conda.pydata.org/docs/using/envs.html#saved-environment-variables I suggest doing that instead of modifying the activate and deactivate directly

egpbos · 2016-05-15T15:19:55Z

Sounds good, will do.

efiring · 2016-05-15T16:38:54Z

You should not be setting PYTHONPATH that way; there is no need to put site-packages on it because that location is compiled in. If you need to disable your PYTHONPATH that is your own special case, not a general condition for using conda. The same applies to DYLD_FALLBACK_LIBRARY_PATH. I'm sure it doesn't need to be set as you have it for the conda environment, and normally it would not even exist. If it is in your environment then at most you need to unset it when entering the conda environment.

egpbos · 2016-05-15T17:25:38Z

It's not in my regular enviroment. I didn't come up with this myself, this solution is advertized in README.osx:

Note that your environment is somewhat important. Some conda users have
found that, to run the tests, their PYTHONPATH must include
/path/to/anaconda/.../site-packages and their DYLD_FALLBACK_LIBRARY_PATH
must include /path/to/anaconda/lib.

When I first tried your suggestion (steps 1-4 above), I had these variables set and it worked. I then thought that perhaps I could now turn them off again, but when I did, first the freetype lib could no longer be found (which was fixed by setting local_freetype to True in setup.cfg) and when that was fixed libpng started complaining:

/Users/patrick/anaconda/envs/mpl_dev/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py in <module>()
     17 
     18 import matplotlib
---> 19 from matplotlib.backends import _macosx
     20 
     21 from .backend_agg import RendererAgg, FigureCanvasAgg

ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev/lib/python2.7/site-packages/matplotlib/backends/_macosx.so, 2): Symbol not found: __cg_png_create_info_struct
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /Users/patrick/anaconda/envs/mpl_dev/lib/libpng16.16.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO

efiring · 2016-05-16T00:00:49Z

I hope we can get this straight and understand what is going on here. I am still pretty sure that having a PYTHONPATH set should not be needed, and I doubt that the DYLD variable is really needed in general either. It would be good to get this sorted out in README.osx as well.
Have you tried my original instructions (that is, with a fresh conda environment) with neither PYTHONPATH nor DYLD_FALLBACK_LIBRARY_PATH set? My guess is that the problem you encountered was a result of having started with those variables set. Otherwise I don't understand why you ran into the ImportError, and I don't. I do have local_freetype set to True in setup.cfg; this is necessary to make the image comparisons with text pass.

egpbos · 2016-05-16T09:30:27Z

Ok, I did the following (in a terminal without DYLD vars, but with a PYTHONPATH with only paths to some custom modules, no critical stuff):

git clone git@github.com:matplotlib/matplotlib.git matplotlib_2
conda create --name mpl_dev2 python
source activate mpl_dev2
conda install matplotlib
conda uninstall matplotlib
conda install python.app
cd matplotlib_2
rehash
pip install .

Then in python:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/textpath.py", line 17, in <module>
    import matplotlib.font_manager as font_manager
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/font_manager.py", line 58, in <module>
    from matplotlib import ft2font
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so, 2): Library not loaded: libfreetype.6.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so
  Reason: image not found

In pythonw:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/textpath.py", line 17, in <module>
    import matplotlib.font_manager as font_manager
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/font_manager.py", line 58, in <module>
    from matplotlib import ft2font
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so, 2): Library not loaded: libfreetype.6.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/ft2font.so
  Reason: image not found

egpbos · 2016-05-16T09:37:40Z

Second try, in the same env (in the matplotlib_2 dir):

make clean
pip uninstall matplotlib
sed -e 's/#local_freetype = False/local_freetype = True/' setup.cfg.template > setup.cfg
pip install .

Then in python:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/textpath.py", line 20, in <module>
    from matplotlib.mathtext import MathTextParser
  File "/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/mathtext.py", line 62, in <module>
    import matplotlib._png as _png
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/_png.so, 2): Library not loaded: libpng16.16.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev2/lib/python2.7/site-packages/matplotlib/_png.so
  Reason: image not found

And in pythonw the same.

egpbos · 2016-05-16T09:44:35Z

Third try: without my custom PYTHONPATH, new conda env, inside the matplotlib_2 dir

make clean
unset PYTHONPATH
conda create --name mpl_dev3 python
source activate mpl_dev3
conda install matplotlib
conda uninstall matplotlib
conda install python.app
rehash
pip install .  # still with local_freetype = True set in setup.cfg

Then python(w) says, again:

>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>
    import matplotlib.colorbar
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>
    import matplotlib.collections as collections
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>
    import matplotlib.backend_bases as backend_bases
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 63, in <module>
    import matplotlib.textpath as textpath
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/textpath.py", line 20, in <module>
    from matplotlib.mathtext import MathTextParser
  File "/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/mathtext.py", line 62, in <module>
    import matplotlib._png as _png
ImportError: dlopen(/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so, 2): Library not loaded: libpng16.16.dylib
  Referenced from: /Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so
  Reason: image not found

egpbos · 2016-05-16T09:46:33Z

Finally, inside mpl_dev3:

export DYLD_FALLBACK_LIBRARY_PATH=/Users/patrick/anaconda/envs/mpl_dev3/lib
pythonw

>>> import matplotlib.pyplot as plt
>>>

So, indeed, PYTHONPATH does not seem necessary, but DYLD_FALLBACK_LIBRARY_PATH, I can't seem to go without...

egpbos · 2016-05-16T10:06:46Z

@efiring Is it possible that you have some custom built libs that your install finds, e.g. from Macports / Homebrew? I have Macports installed and I see now that I also have libpng there. I tried upgrading libpng, but to no avail.

egpbos · 2016-05-16T10:21:28Z

Related: #3416.

$ otool -L /Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so 
/Users/patrick/anaconda/envs/mpl_dev3/lib/python2.7/site-packages/matplotlib/_png.so:
    libpng16.16.dylib (compatibility version 34.0.0, current version 34.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 60.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 2577.0.0)
$ pkg-config libpng --libs
-L/opt/local/lib -lpng16

Seems like it is indeed building against the Macports install, i.e. if it uses pkg-config to find libpng...

egpbos · 2016-05-16T10:31:18Z

$ conda install pkg-config
$ rehash
$ pkg-config libpng --libs
-L/Users/patrick/anaconda/envs/mpl_dev3/lib -lpng16

That's better...

Then uninstall and reinstall matplotlib from source. Again: no fish. Is pkg-config used at all during build?

egpbos · 2016-05-16T11:13:51Z

Turns out functools32 was installed by pip at some point. Removed that, reinstalled with conda. Then uninstalled matplotlib, make clean inside the git repo and reinstalled with pip. Still libpng complains.

egpbos · 2016-05-16T11:32:23Z

Also tried building with LDFLAGS and CFLAGS set to include the conda env lib and include paths. Same error. Ideas?

jenshnielsen · 2016-05-16T11:43:22Z

All of these are build time discovery steps. I don't think they have any influence at runtime. You can try setting rpath via LDFLAGS to point to the environment lib dir. Something like LDFLAGS="-rpath=/path/to/condaenv/lib" (Completely untested)

efiring · 2016-05-16T18:48:13Z

@egpbos, Thank you for that extensive testing. Yes, I am using homebrew, which has libpng16.dylib, but I also have a libpng16.dylib in my anaconda/lib.
I still don't understand what is going on here. It seems like our configurations are very similar...

efiring · 2016-05-16T22:34:08Z

@egpbos, Thank you for that extensive testing. Yes, I am using homebrew, which has libpng16.dylib, but I also have a libpng16.dylib in my anaconda/lib.
I still don't understand what is going on here. It seems like our configurations are very similar...

story645 · 2016-06-17T21:35:11Z

Just adding a note that:

from matplotlib.testing.decorators import knownfailureif

will trigger this. @jenshnielsen 's suggested fix of setting the backend:

import matplotlib
matplotlib.use('Agg') #backend framework error                                       
from matplotlib.testing.decorators import knownfailureif

will work if your tests aren't backend specific.

egpbos · 2018-04-09T06:15:02Z

As mentioned in #11006 (comment), things seem to have gotten a bit simpler on my most recent try (other machine by now, so clean environment compared to the one used for the above discussion of this issue).

For reference, here's what I need now to get it correctly installed from scratch (inside a clean matplotlib clone):

conda create -n mpl_dev matplotlib clang clangxx libpng freetype pkg-config python.app
conda activate mpl_dev
conda uninstall matplotlib
export CC=clang
export CXX=clang++
python -mpip install -e .

And then run with pythonw to use the macosx backend (or switch to Agg or TkAgg in matplotlibrc).

rcomer · 2021-01-01T12:03:19Z

Having this week made my first matplotlib branch, I must admit I found the developers' installation instructions slightly terrifying! I would have been a lot more comfortable with something conda based, as that's what I'm used to.

I've no idea if this would be realistic for matplotlib, but in iris we have a yml file that defines its dependencies and everything you could possibly need during testing (sample data, sphinx, flake8, black, etc.) This makes creating a conda development environment ridiculously easy.

timhoffm · 2021-01-01T16:34:08Z

There's a draft for a conda yml at #17096.

Still needs clarification on the handling of some optional dependencies (maybe even only check if they are used in the tests). I'm happy to get feedback if the yml works.

rcomer · 2022-10-22T16:00:17Z

The Conda yml file is now on main and is nicely documented here:
https://matplotlib.org/stable/devel/development_setup.html#create-a-dedicated-environment

So I wonder if this issue could be closed.

timhoffm · 2022-10-22T17:25:57Z

Yes, I think this can be closed. Thanks for the notification!

tacaswell added this to the 2.0.1 (next bug fix release) milestone May 13, 2016

tacaswell added the Documentation label May 13, 2016

jenshnielsen mentioned this issue Jul 15, 2016

Matplotlib.pyplot 2.0.0b2 fails to import with Conda Python 3.5 on OS X #6746

Closed

QuLogic modified the milestones: 2.0.1 (next bug fix release), 2.0.2 (next bug fix release) May 3, 2017

brentp mentioned this issue May 16, 2017

X server dependency error brentp/peddy#29

Closed

tacaswell modified the milestones: 2.1.1 (next bug fix release), 2.2 (next feature release) Oct 9, 2017

dstansby added the OS: Apple label Jul 21, 2019

story645 mentioned this issue May 4, 2020

Consolidate Install Docs #17327

Closed

6 tasks

rcomer mentioned this issue Jan 4, 2021

Add conda environment.yml for developers #17096

Merged

timhoffm closed this as completed Oct 22, 2022

QuLogic modified the milestones: future releases, v3.6.0 Jul 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update virtualenv + OSX + conda guide #6418

Update virtualenv + OSX + conda guide #6418

egpbos commented May 13, 2016 •

edited

Loading

tacaswell commented May 13, 2016

egpbos commented May 15, 2016

jenshnielsen commented May 15, 2016

egpbos commented May 15, 2016

efiring commented May 15, 2016

egpbos commented May 15, 2016

efiring commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

jenshnielsen commented May 16, 2016

efiring commented May 16, 2016

efiring commented May 16, 2016

story645 commented Jun 17, 2016

egpbos commented Apr 9, 2018

rcomer commented Jan 1, 2021

timhoffm commented Jan 1, 2021

rcomer commented Oct 22, 2022

timhoffm commented Oct 22, 2022

Update virtualenv + OSX + conda guide #6418

Update virtualenv + OSX + conda guide #6418

Comments

egpbos commented May 13, 2016 • edited Loading

tacaswell commented May 13, 2016

egpbos commented May 15, 2016

jenshnielsen commented May 15, 2016

egpbos commented May 15, 2016

efiring commented May 15, 2016

egpbos commented May 15, 2016

efiring commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

egpbos commented May 16, 2016

jenshnielsen commented May 16, 2016

efiring commented May 16, 2016

efiring commented May 16, 2016

story645 commented Jun 17, 2016

egpbos commented Apr 9, 2018

rcomer commented Jan 1, 2021

timhoffm commented Jan 1, 2021

rcomer commented Oct 22, 2022

timhoffm commented Oct 22, 2022

egpbos commented May 13, 2016 •

edited

Loading