Bug summary

Placing non-standard tex commands like \unit{mm} from siunitx package on $...$ when used for labels (and probably other text elements) does not work when tex rendering by matplotlib is disabled.
Does it still check stuff in $ signs for valid commands?

Code for reproduction

import locale
import matplotlib
import matplotlib.pyplot as plt

matplotlib.use('TkAgg')
plt.style.use('matplotlib_style.mplstyle')  
# text.usetex: False 
# text.latex.preamble:  r'\usepackage{amsmath}\usepackage{siunitx}'
locale.setlocale(locale.LC_ALL, '')

fig, ax = plt.subplots()
ax.plot([0], [0], 'x')

ax.set_xlabel(r'$\Delta x$ \unit{mm}')  # <- does work
ax.set_xlabel(r'$\Delta x \unit{mm}$')  # <- does not work 
plt.savefig('fig.svg')  # plt.show() does not work either

Actual outcome

Traceback (most recent call last):
File "/venv/lib/python3.10/site-packages/matplotlib/_mathtext.py", line 2237, in parse
result = self._expression.parseString(s)
File "/venv/lib/python3.10/site-packages/pyparsing/core.py", line 1134, in parse_string
raise exc.with_traceback(None)
pyparsing.exceptions.ParseFatalException: Unknown symbol: \unit, found '' (at char 0), (line:1, col:1)

...

ValueError:
\unit{mm}
^
Unknown symbol: \unit, found '' (at char 0), (line:1, col:1)

Expected outcome

Label with plain text (not Tex rendered yet) text in .svg file
(rendering those later with latex svg package)

Additional information

Seems to happen when non standard commands are used in math mode (like $\unit{mm}$)
Both $\unit{mm}$ and \unit{mm} work with siunitx package and xetex

Operating system

PopOS 21.10

Matplotlib Version

3.5.1

Matplotlib Backend

TkAgg

Python version

3.10.0 | Make "In console: python --version or python3 --version" in bug report form

Jupyter version

No response

Installation

pip