Skip to content

Available fonts are ignored by font_manager #10201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mikkokotila opened this issue Jan 9, 2018 · 39 comments
Closed

Available fonts are ignored by font_manager #10201

mikkokotila opened this issue Jan 9, 2018 · 39 comments
Labels
status: needs clarification Issues that need more information to resolve. topic: text
Milestone

Comments

@mikkokotila
Copy link

mikkokotila commented Jan 9, 2018

There are two issues:

  1. font_manager is not finding fonts that are in the system > this is easily resolved by rebuilding the cache, which perhaps should be a default True as it is very fast to do it seems.

  2. even when the font is found, Matplotlib forcefully choose one of the default fonts > I did not find a resolution for this.

The font_manager utility seems to have some issues in exposing fonts that are installed in the system to matplotlib. For example:

from matplotlib import font_manager
font_manager.findfont('Noto Sans Tibetan')

Is referring to a font that I have in the system, and this yields:

'/Users/mikko/dev/boke/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUni.ttf'

(BTW: it is very strange to get this path, as it refers to totally separate virtualenv than the one I'm executing these commands from!?)

So for some reason, it forcefully ignores the font that I have. To prove the point, I will try the same with made up font name:

font_manager.findfont('sdfdsfds')

/Users/mikko/dev/fontest/lib/python3.6/site-packages/matplotlib/font_manager.py:1320: UserWarning: findfont: Font family ['sdfdsfds'] not found. Falling back to DejaVu Sans

To get this far i.e. being able to identify the font I have in the first place, I needed to do:

font_manager.findfont('Noto Sans Tibetan', rebuild_if_missing=True)

Any suggestion for overcoming this?

@tacaswell
Copy link
Member

In general rebuilding the font-cache is not fast (we have reports of it taking 30+ seconds for some users). It depends on how many fonts you have installed and your file system (which is the reason we have the font cache to begin with!).

The font cache is in the home directory and shared by all python enviroments. However we include paths to fonts that came as part of the Matplotlib data payload. We should add a hash of the python executable path to the cache name.

For looking for the nonsense name, that is the intended behavior, we always need to find some font to render the text on the theory than rendering with the wrong font is better than not rendering at all if a font is not found.

For finding the wrong font for 'Noto Sans Tibetan' it is not clear to me if rebuilding the cache helped or not. If it did not, can you look in ~/.cache/matplotlib/fontlist.json and see if the expect file is found? The possible issues I see are a) the file is someplace we are not looking b) the font is one we can not parse / use c) there is a bug in the font selection logic.

@mikkokotila
Copy link
Author

mikkokotila commented Jan 9, 2018

Thanks! No the issue is not resolved i.e. Sans Tibetan does did not become available for me at any point. I tried an alternate way just to see if there is a way to at least get a font property created:

import matplotlib.font_manager as font_manager
import matplotlib.pyplot as plt
font_path = '/Library/Fonts/NotoSansTibetanRegular.ttf'
prop = font_manager.FontProperties(fname=font_path)

This works and I can also then use the plt.text() and it does display the text correctly. But this just allows the creation of the font property, and I can't use that in Seaborn when I'm drawing the plots. This seems to indicate that the option 'b' is over-ruled.

What comes to ~/.cache/matplotlib/fontlist.json I don't seem to have such a file.

@mikkokotila
Copy link
Author

To add, when I try yet another alternate way:

from matplotlib import rcParams

rcParams['font.family'] = 'sans-serif'
rcParams['font.sans-serif'] = ['Noto Sans Tibetan Regular']

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.plot([1, 2, 3], label='test')

ax.legend()
plt.show()

I get:

/Users/mikko/dev/boke/lib/python3.6/site-packages/matplotlib/font_manager.py:1320: UserWarning: findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))

To re-iterate from the first message, what you see here i.e. /dev/boke/... is a path of a different virtualenv. So I'm running this code (and the other codes) from virtualenv by the name of 'fontest' but some of the executions seem to take place in 'boke' virtualenv. Is this expected? If yes, this could be part of the problem...

@mikkokotila
Copy link
Author

In the meantime, here is the workaround using a barplot as an example:

import matplotlib.font_manager as font_manager
import matplotlib.pyplot as plt

def vbar(x, labels, xlabel, font_path):
    
    font_path = font_path
    prop = font_manager.FontProperties(fname=font_path)
    
    dim = range(len(x))

    plt.bar(dim, x)
    plt.xlabel(s=xlabel, fontproperties=prop)
    plt.xticks(dim, labels, fontproperties=prop)

I looked at the Seaborn codes and I think this will not be trivial to achieve there. Also, it seems based on issues that Unicode support might not be high up in the priorities. If you ask me, I prefer to use just matplotlib for several reasons...not the least of which is the fact that I can choose what I'm exposed to in terms of params ;)

@jklymak
Copy link
Member

jklymak commented Jan 9, 2018

The fact that font manager is being called from the wrong environment means you don’t have matpltolib properly installed. I’d fix that first before you do any debugging of the underlying problem.

@mikkokotila
Copy link
Author

@jklymak, when you say, is not installed properly, that's interesting as I just use the standard approach of not installing anything on my actual system, but just installing what I need on each env as I go. This seems to be the only meaningful way to work with python. Or am I missing something. To add, I just use pip install matplotlib to do the installation.

@WeatherGod
Copy link
Member

WeatherGod commented Jan 10, 2018 via email

@mikkokotila
Copy link
Author

@WeatherGod Sure sure, though this takes us far from the end-user experience I reported. Let me know if you need any additional info regarding the issue at hand, and I'm happy to work together to get to the bottom of this. I think using Unicode should be seamless in this day and age, and based on looking around online I'm not the only one having problems. There might be something off with the way fonts are handled in general (family vs. type etc, folders, etc) or even a bug in the code. Let me know if there is anything I can do / help with. Seaborn users don't seem to have any actual resolution to this and the maintainer refers questions to you guys.

@anntzer
Copy link
Contributor

anntzer commented Jan 10, 2018

Re: #10201 (comment) (second paragraph):

I think the "easy" way to fix this, when generating fontlist.json is to check whether the font path is contained within os.path.dirname(matplotlib.__file__). If it is, then write the path as a relative one. When reading the font cache, do the reverse operation.

@mikkokotila I think it's ~/.matplotlib/fontlist.json for osx.

@tacaswell tacaswell added this to the v3.0 milestone Jan 10, 2018
@anntzer
Copy link
Contributor

anntzer commented Jan 11, 2018

Ah, perhaps a relevant point:

  1. conda-forge installs its own version of fontconfig as a dependency for matplotlib (so in a conda env, fc-list refers to the version of fontconfig installed by conda).
  2. on OSX, conda-forge's fc-list does not find any system fonts (on Linux, conda's fc-list needs to build its own cache but does find system fonts). It is because the fonts.conf it uses still looks into /usr/share/fonts only. Compare with e.g. homebrew's fontconfig, which is configured to also look into OSX's fonts directories (https://github.com/Homebrew/homebrew-core/blob/master/Formula/fontconfig.rb#L35). [anaconda's fonts.conf appears to have the same issue but anaconda's matplotlib does not declare fontconfig as a dependency so we likely don't have the issue unless the user also installed fontconfig (not tested)]

I think this is at least in part a packaging bug on conda(forge)'s part.

@jklymak
Copy link
Member

jklymak commented Jan 11, 2018

Given /usr/share/fonts doesn't exist on macOS, I guess a workaround is to soft link to /Library/Fonts?

@anntzer
Copy link
Contributor

anntzer commented Jan 11, 2018

I'd say the workaround is for conda{,-forge} to fix their builds... you can't assume users have sudo rights.

@mikkokotila
Copy link
Author

@anntzer for the avoidance of doubt, I don't have conda, just plain vanilla python. Generally for workarounds, unfortunately, workarounds that depend on others, particularly someone as "big" as Anaconda are not really a workaround is it? Rather wishful thinking ;)

In any case, really great to see this in the v3.0 milestone 👍

@anntzer
Copy link
Contributor

anntzer commented Jan 11, 2018

@mikkokotila Did you figure out whether the missing font was listed in fontlist.json? Does is appear in the output of fc-list (assuming you have fontconfig installed)?

@mikkokotila
Copy link
Author

@anntzer I don't have that file anywhere in the system it seems. Also, I don't have fontconfig. Using the font_manager way to get the list of fonts, yes those fonts I wanted to use but could not use were listed in that output. One thing I noted was that the permission for the font I could not use and those I could use was different, those that were working belonged to the group 'wheel'. I think that might be relevant, but at that point, I had already found the way to get the plots done and had to move on as most of the day had gone to troubleshooting this. The other thing I think has something to do with this possibility is the distinction between font-family and font-type/face but I'm not entirely clear with this, as it would require learning more about font management in general. As far as I could gather, it would be far preferred to just have one setting which is the actual font you have to use. Dealing with fonts is so far from the skillsets of most users for something like matplotlib, that I think it will be great service to figure this out in a way that works cross-system without the user ever having to think about it. Also maybe it would be good to include a section for the workaround I'm using in the current manual. If you think that's a good idea, I'm happy to write something up with code examples. I could go back and conclude the system level aspect properly if you think it's worthwhile.

@mikkokotila
Copy link
Author

To reiterate from previous message (if I understand the upstream tag right), I don't think this has anything to do with Anaconda as I'm not having Anaconda. It seems that would be a separate issue entirely and bundling these two would not benefit resolving either issue properly. The way I'm using python is the "best practice" way i.e. from clean OS install directly start using environments and install everything at that level so any system level issues would have to relate with that type of use case.

@anntzer
Copy link
Contributor

anntzer commented Jan 12, 2018

Yup, got the fact that this is independent of conda. We just need to figure out what's happening...
Run

from matplotlib.font_manager import _rebuild; _rebuild()

This should create ~/.cache/fontList.json (or, just to be sure, it should go to whatever folder from matplotlib import get_cachedir; print(get_cachedir()) indicates (or does that give None?)).
Also, what does from matplotlib.font_manager import OSXInstalledFonts; print(OSXInstalledFonts()) give?

@tacaswell
Copy link
Member

@mikkokotila Thanks for sticking with us through debugging this. We understand it can be a frustrating process.

I agree it would be good to improve the documentation on this.

@jkseppan
Copy link
Member

I think this is about at least two separate issues:

  • the font cache is global across all Python installations, so whichever virtualenv you happen to be using when initializing the cache gets to insert its view of the available fonts, which will include the fallback fonts distributed with matplotlib
  • the way to select a font is by specifying features in a slightly css-like way that doesn't work exactly the way css does and that can produce counterintuitive results; the font manager attempts to do something reasonable across users and systems that have different fonts installed but makes it a little bit hard to get the exact fonts you want on your particular system

In particular, the way to get Noto Sans Tibetan Regular through rcParams is

rcParams['font.family'] = 'sans-serif'
rcParams['font.weight'] = 'regular'                  # you can omit this, it's the default
rcParams['font.sans-serif'] = ['Noto Sans Tibetan']

and if you want Noto Sans Tibetan Bold, you change font.weight to 'bold' instead.

@mikkokotila
Copy link
Author

@jkseppan to avoid doubt, that is the method that is not working. Also, I think it should be considered if there is a better way to do this i.e. where the font is declared directly without regard for the family of the font.

@jkseppan
Copy link
Member

That method works on my Mac, so there is likely a bug that affects you. You mention that you don't have fontconfig, so perhaps that is the pertinent difference.

@anntzer
Copy link
Contributor

anntzer commented Jan 13, 2018

@mikkokotila Can you please let us know what you get after following #10201 (comment)? This will help us figure out what is going wrong...

Also, what is the actual path where NotoSansTibetan-Regular.ttf is installed on your machine?

"In theory"

rcParams["font.family"] = "Noto Sans Tibetan"

should be enough to select the font...

@anntzer
Copy link
Contributor

anntzer commented Jan 16, 2018

(As a side note, looks like there's a better way to get the list of fonts on OSX, per https://apple.stackexchange.com/a/243746. You can get xml (actually, plist) output and Python's plistlib reads it just fine. Hmm...).

@mikkokotila Can you provide the output of running system_profiler SPFontsDataType at the terminal, so that we can understand better what your font installation looks like?

@mikkokotila
Copy link
Author

@anntzer Very sorry for dropping the ball with this. Was traveling in the Himalayas and when came back, ended up busy with some other things. I ran the command that you request and the dump is almost 1mb in size...and further I realized that I might be on a different system right now from where I was then. I think it's better I reproduce the problem, after having re-read this thread and then report back. What do you think?

In the meantime, I believe it would be great to move away from the requirement to declare the font-family as opposed to just directly (and exclusively) declaring the font itself.

@mikkokotila
Copy link
Author

@jklymak while this would not address the problem related with the way the actual fonts are called through the family, there might be another issue with the soft linking to ~/Library/Fonts. Actually that folder is empty (except for the font I had myself forcefully put there when troubleshooting for this). The main folder for fonts seems to be:

/FontCollections and the insides looks like:

-rw-------  1 mikko  staff   2.3K Jan  1  2016 Fixed Width.collection
-rw-------  1 mikko  staff   872B Jan  1  2016 Fun.collection
-rw-------  1 mikko  staff   815B Jan  1  2016 Modern.collection
-rw-------  1 mikko  staff   856B Jan  1  2016 PDF.collection
-rw-------  1 mikko  staff   1.0K Jan  1  2016 Traditional.collection
-rw-------  1 mikko  staff   1.1K Jan  1  2016 Web.collection

@anntzer
Copy link
Contributor

anntzer commented Mar 31, 2018

AFAICT this is not a standard font location (see e.g. https://support.apple.com/en-us/HT201722) so it's not surprising that we're not picking up the font...
Does Noto Sans Tibetan appear in the output of system_profiler SPFontsDataType?

@mikkokotila
Copy link
Author

@anntzer yes it does appear in the system_profiler SPFontsDataType output. I don't think that this problem can not be reduced to the Mac OS X default font folder problem.

@anntzer
Copy link
Contributor

anntzer commented Mar 31, 2018

What's the corresponding entry in SPFontsDataType? (specifically, is the Location: field correct?)

@mikkokotila
Copy link
Author

To highlight the font-face approach, versus the current approach...

rcParams.items

I get this:

'font.cursive': ['Apple Chancery', 'Textile', 'Zapf Chancery', 'Sand', 'Script MT', 'Felipa', 'cursive'], 'font.family': ['Verdana'], 'font.fantasy': ['Comic Sans MS', 'Chicago', 'Charcoal', 'ImpactWestern', 'Humor Sans', 'xkcd', 'fantasy'], 'font.monospace': ['DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Computer Modern Typewriter', 'Andale Mono', 'Nimbus Mono L', 'Courier New', 'Courier', 'Fixed', 'Terminal', 'monospace'], 'font.sans-serif': ['DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', 'sans-serif'], 'font.serif': ['DejaVu Serif', 'Bitstream Vera Serif', 'Computer Modern Roman', 'New Century Schoolbook', 'Century Schoolbook L', 'Utopia', 'ITC Bookman', 'Bookman', 'Nimbus Roman No9 L', 'Times New Roman', 'Times', 'Palatino', 'Charter', 'serif'],

Instead of being nested into families, there should be one parameter, which is font-face, and that's all it takes to invoke a given font. There should be no reason to not do it like that, or have I misunderstood something?

@mingwandroid
Copy link

For anyone who was encountering this using conda/AD, I've changed fontconfig on the Anaconda Defaults channel on macOS to also look at the system fonts directory. Cheers.

@anntzer
Copy link
Contributor

anntzer commented Jun 19, 2018

The nested families is due to the mimicking of CSS font selection (see also #10249), and it's not relevant to the problem at hand.

Answering #10201 (comment) may help here.

@tacaswell tacaswell modified the milestones: v3.0, v3.1 Aug 11, 2018
@rpspragg
Copy link

rpspragg commented Sep 10, 2018

Does anyone know if this issue is still open? I'm having this issue in MatPlotLib 2.2.2 running in Spyder, Python 3.6.5, Win10. I've tried clearing my font cache, and when it rebuilds the font is in the fontList.json file, but the my kernel is still giving me a font not found error message.

~~Update:
Problem solved for me. I should read the documentation first, but I will keep this here in case it helps anyone else. My issue was I was trying to define the font family in a custom style file, i.e., .mplstyle, and i was putting the font.family property in quotes, which the documentation clearly gives examples of it not being in quotes.

@anntzer
Copy link
Contributor

anntzer commented Sep 11, 2018

I'm going to close this for lack of reproducibility and information. Feel free to request a reopen with more info.

@anntzer anntzer closed this as completed Sep 11, 2018
@socomhacker1
Copy link

I don't know if I should request a reopen for this but I had basically the same issue using Macos. When I used import matplotlib.pyplot as plt I received the follow error:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/ghost/.matplotlib/fontlist-v300.json'
I was able to fix this by setting the constant ``USE_FONTCONFIG` to true in the font_manager.py file.
Fontconfig is the standard way to lookup fonts on X11 platform according the the Remarks in font_manager.py so if a font is installed it is more likely to be found.

@anntzer
Copy link
Contributor

anntzer commented Oct 17, 2018

That's a separate issue, already fixed in master (3.0.1 "should" be released in a few weeks with the fix); see #12291 and linked stuff.

@socomhacker1
Copy link

socomhacker1 commented Oct 17, 2018 via email

@bcrafton
Copy link

I was getting the same error on matplotlib 2.2.2 and then upgraded to matplotlib 2.2.3 and the error went away for me

@gusmd
Copy link

gusmd commented Nov 19, 2018

Posting this in case it helps anyone, since this is one of the first links on Google on this topic.

I had a similar issue when updating to a debug version of 3.0 from 2.x where my font cache only had about 1/3 of the the fonts on my system, hence many fonts wouldn't be found. I solved it by rebuilding the font cache using a Release version of Matplotlib 3.0 (standard install from PyPI).

For comparison, calling len(font_manager.findsystemFonts()) returns 152 on my debug build (built myself from source), and 708 on my release build (installed through pip).

@nora-illanes
Copy link

I still have this issue, here is my case:
findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans.

#13139

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: needs clarification Issues that need more information to resolve. topic: text
Projects
None yet
Development

No branches or pull requests