can you post the code you used to generate these? Since you are mixing the native Matplotlib fontsize declarations with LaTex, it might not lead to correct results.

Thank you for the hint! Sorry, I didn't have time to create a simple MWE. I've tried setting the font sizes explicitly in pt, and that solved the problem. I thought internally, small x-small etc. would translate to LaTex's \small{} and \footnotesize{} font sizes which would be a nice feature. As for now, you would have to set all desired font sizes in pt manually.

If you look at https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/font_manager.py it shows the following scaling values...

font_scalings = {
    'xx-small' : 0.579,
    'x-small'  : 0.694,
    'small'    : 0.833,
    'medium'   : 1.0,
    'large'    : 1.200,
    'x-large'  : 1.440,
    'xx-large' : 1.728,
    'larger'   : 1.2,
    'smaller'  : 0.833,
    None       : 1.0}

My guess is, the LaTex font you are using only allows certain quantized font sizes. This might result in the same quantized fontsize corresponding to both 'small', 'x-small' scalings (I am not 100% sure if this is the correct explanation).

I think you are right. So I guess, the only way for now to accomplish consistent font sizes to use within a LaTex document is to specify all the font sizes in pt directly. This is a little cumbersome but probably not a problem of Matplotlib but rather a task for an extra package.

So I guess, the only way for now to accomplish consistent font sizes to use within a LaTex document is to specify all the font sizes in pt directly.

or you could use relative sizes i.e. \small{}, \large{} etc

But how would I specify that in Matplotlib? I have tried setting the fontsizes in the .rc file to \footnotesize{} etc. but the syntax does not work and I don't see anything in the font manger that points towards such an option.

Here is a small example showing how to control font size when you are using LaTex with Matplotlib.
If this answers your question, please feel free to close this issue.

import numpy as np
import matplotlib.pyplot as plt

t = np.arange(0.0, 1.0 + 0.01, 0.01)
s = np.cos(4 * np.pi * t) + 2

plt.rc('text', usetex=True)
plt.rc('text.latex', preamble=r'\usepackage{fourier}, \usepackage[T1]{fontenc}')
plt.plot(t, s)

plt.xlabel(r'{\Huge{Huge}}, Normal, {\tiny{tiny}}')
plt.ylabel(r'{\Huge{Huge}}, Normal, {\tiny{tiny}}')
plt.tight_layout()
plt.savefig('tex_demo.png')

Ah yes I see what you mean. This, to be honest, seems to be even more cumbersome then putting in the correct font sizes in pt in the rc or style file.

Regarding the issue: I still feel like this could be better documented in the LaTex guide of Matplotlib, as one of the goals of Matplotlib is to create publish ready plots. If you don't feel like, this is a Matplotlib issue, I don't have a problem with closing it.

Fixes to documentation or code most welcome. It does seem that xsmall and xxsmall should give different font sizes.

I will take this up over the weekend and try to document this (in addition to #11586 related to the use of TeX. I did start a PR #11718 but couldn't follow through).

This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear. Thanks for your help!

I think this is actually caused by the "optical sizes" of a font (sometimes found in OpenType variable fonts). When you zoom in on certain small pt sized fonts, they will look distorted, but from the normal intended zoom level it is more appealing to the human eye. This is tricky to track down due to the subtlety, but once you know what to look for, it's easy to find resources explaining it (e.g. link, link, link, link).

Sorry for bumping a 5+ year old issue. I figured this would be helpful for people that stumble across this issue in the future (like myself), to know that it's a feature and not a bug in matplotlib or LaTeX.