I feel we still haven't got to the bottom of the root cause. If the cache is corrupted (or from an earlier version or matplotlib or Python), it is rebuilt and then saved to disk, with the next import using the rebuilt "fixed" cache. (At least that's how it's supposed to work, and works in my experimentation when upgrading from 1.3.1 to 1.4.2 as in #3794). So somehow the rebuilt cache is not getting written back out to disk properly, perhaps? I don't think the cache needs to be versioned to accomplish this -- it's already effectively versioned by being a pickle -- we just need to make sure that rebuilding doesn't happen repeatedly.

The font caches is versioned, https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/font_manager.py#L1017 The question is more about why the _rebuild is not writing back out to disk on some systems.

Possibly also related is that one report about slow imports showing the
cache getting loaded, but then still going ahead and querying the system
for all of its fonts. I can't find the issue link at the moment, but it was
fairly recent (in the past couple of months) and also raised the question
about the font cache being a pickle and it potentially being a security
issue.

On Wed, Aug 26, 2015 at 5:31 PM, Thomas A Caswell notifications@github.com
wrote:

The font caches is versioned,
https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/font_manager.py#L1017
The question is more about why the _rebuild is not writing back out to
disk on some systems.

—
Reply to this email directly or view it on GitHub
#4993 (comment)
.

@WeatherGod maybe you're thinking of #4756?

yes, that is the one.

On Wed, Aug 26, 2015 at 6:11 PM, Martin Spacek notifications@github.com
wrote:

@WeatherGod https://github.com/WeatherGod maybe you're thinking of #4756
#4756?

—
Reply to this email directly or view it on GitHub
#4993 (comment)
.

Some thoughts:

• if you have a TRAVIS environment variable, the cache is always rebuilt
• if your $MPLCONFIGDIR is not writable the font cache will be rebuilt and written to a temporary directory, which will not be reused later; similarly for $HOME/.matplotlib or $HOME/.cache/matplotlib if $MPLCONFIGDIR is unset
• if a font file is found for which os.path.isfile is false (symlinked fonts?) the cache is rebuilt

With regards to possibly symlinked fonts, maybe it makes sense to apply
os.path.abspath() to font filenames that are discovered as the cache is
built?

On Thu, Sep 3, 2015 at 6:22 AM, Jouni K. Seppänen notifications@github.com
wrote:

Some thoughts:

• if you have a TRAVIS environment variable
  

  matplotlib/lib/matplotlib/font_manager.py

  Line 1388 in 673feb0

  if not 'TRAVIS' in os.environ:

  ,
  the cache is always rebuilt
• if your $MPLCONFIGDIR is not writable
  

  matplotlib/lib/matplotlib/__init__.py

  Line 596 in 673feb0

  if not _is_writable_dir(configdir):

  
  the font cache will be rebuilt and written to a temporary directory, which
  will not be reused later; similarly for $HOME/.matplotlib or
  $HOME/.cache/matplotlib
  

  matplotlib/lib/matplotlib/__init__.py

  Line 609 in 673feb0

  if _is_writable_dir(p):

  
  if $MPLCONFIGDIR is unset
• if a font file is found for which os.path.isfile is false
  

  matplotlib/lib/matplotlib/font_manager.py

  Line 1305 in 673feb0

  if not os.path.isfile(result):

  
  (symlinked fonts?) the cache is rebuilt

—
Reply to this email directly or view it on GitHub
#4993 (comment)
.

The symlinked font thing may be the issue (and would explain why I couldn't reproduce). I'll create a symlinked font on my system and see if that triggers this bug. If that's what it is, it should be easy to resolve.

Currently, switching between importing mpl 1.5 and mpl 2.0 repeatedly triggers a rebuilding of the font cache (as they keep overwriting each other). Of course that was probably already the case "in theory" before, but this is probably going to happen more often in practice now that conda is widespread. Plus, some git archeology suggests that all the changes to the font cache version so far occured between the release of 0.99 and 1.0 :-)

So perhaps matplotlib should keep its cache in ~/.cache/matplotlib/$cacheversion/... (or similar) instead?

EDIT: I'm going to make this a 2.0 milestone, but I'm open to arguments that this is not such a big deal. Another (simpler?) option would be to store the cache in a subfolder of the venv/conda env if one is active, so that at least the issue only appears when updating matplotlib in a given environment, not when switching from a mpl=1.5 env to a mpl=2.0 env.

This is not release critical, so I am bumping it to 2.1 (thought I like @anntzer's idea to have a folder per version).

If the symlinked font thing is really an issue, I'd bump fixing it. I often set my $MPLCONFIG variable to point to a symlinked path and have been dealing with repeated warning about fc-list (#5836) for months now. I never guessed it could be related to symlinking until I saw this thread.

The versioning of the cache should be straight forward:

• find where the font cache path is created (around L1423 in font_manager.py)
• add the version as part of the path

Still not sure I understand the relation to the symlinks here. @kadrlica Do you get the warning at import or the first time you draw some text? On my system, symlinks report as files:

dd35) ✔ ~ 
19:06 $ touch /tmp/foo
(dd35) ✔ ~ 
19:06 $ cd /tmp/
(dd35) ✔ /tmp 
19:06 $ ln -s foo foo2
(dd35) ✔ /tmp 
19:06 $ ipython
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:53:06) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: res = '/tmp/foo2'

In [2]: import os.path

In [3]: os.path.isfile(res)
Out[3]: True

In [4]: os.path.isfile('/tmp/foo')
Out[4]: True

Ok, I think that the symlink was a red herring (though I was surprised that MacOS Sierra has \tmp point to \private\tmp).

I think I was actually getting caught by the same font cache versioning issue. I was switching between matplotlib v1.3.1 and matplotlib v1.5.3, which have different behaviors when overwriting the font cache. I think v1.3.1 overwrites the font cache generated by v1.5.3, while v1.5.3 creates a temporary font cache each time it finds font cache created by v1.3.1. Thus, when I used v1.3.1 it was "corrupting" the font cache and when I switched to v1.5.3 it was stuck making temporary font caches. The comments from @mdboom and @anntzer make me think that this is not the expected behavior for v1.5.3 (they suggest that the font cache should be rebuilt and written to disk).

Here's an example of this behavior

> export MPLCONFIGDIR=/tmp
> export MATPLOTLIBRC=/tmp
> rm -rf $MPLCONFIGDIR/fontList.cache $MPLCONFIGDIR/tex.cache

> # Use matplotlib v1.5.3
> source activate mpl_1.5.3
> # This import builds the fontList for v1.5.3
> python -c "import matplotlib; print matplotlib.__version__; import pylab"
1.5.3
/usr/local/anaconda2/envs/mpl_1.5.3/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
> # And this one works as expected
> python -c "import matplotlib; print matplotlib.__version__; import pylab"
1.5.3

> # Switch to matplotlib v1.3.1
> source activate mpl_1.3.1
> # This one builds a fontList for v1.3.1
> python -c "import matplotlib; print matplotlib.__version__; import pylab"
1.3.1
> # And this one works as expected
> python -c "import matplotlib; print matplotlib.__version__; import pylab"
1.3.1

> # Use matplotlib v1.5.3
> source activate mpl_1.5.3
> # This finds the fontList for v1.3.1 and creates a tmp version
> python -c "import matplotlib; print matplotlib.__version__; import pylab"
1.5.3
/usr/local/anaconda2/envs/mpl_1.5.3/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
> # ... and again
> python -c "import matplotlib; print matplotlib.__version__; import pylab"
1.5.3
/usr/local/anaconda2/envs/mpl_1.5.3/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

Playing around with this, here is what I've found (if this has not already been discovered):

When going from an older version of MPL to 1.5.3, pickle_load itself causes FontManager.__init__ to be called. At least according to the debugger, it goes,

-> data = pickle.load(fh)
-> FontManager.__init__ called

Calling FontManager.__init__ eventually calls the method that rebuilds the cache (since init is intended to do that). However this never calls _rebuild to write this to file since no exception is generated because the font manager is successfully loaded via pickle.

When using a cache generated originally by 1.5.3, pickle_load does not call init. Therefore everything works as expected.

I think this explains why the cache is not re-written to file (it is because _rebuild is never called), while also explaining why it searches for cache again even though it can find it (sort of.. it does not explain why pickle calls init here).

Since pickle is no longer used, a part of this problem should not occur in the future. In terms of versioning, there is already a check for font_manager version that rebuilds on change. If it turns out the reason this issue occurred is that the font_manager was modified in some way that caused pickle.load to call init then 2.x does not have this specific issue: unlike MPL 1.3 -> 1.5, which retained font_manager version, 2.0 changes it, thus causing _rebuild to run. In terms of versioning for other reasons (e.g. conda justification/users constantly switching between MPL versions thus causing constant rebuilding), none of this reply is very relevant.

is it possible to include me as a participant?

Closed by #10245 (see #10245 (comment)).