When I saving a SVG figure is slowed down significantly (10-100x) when I use LaTeX, enabled by the following rc-setting:

text.usetex : true

Note that rendering is somewhat slower, but not much. It is really the saving. I am on macOS 10.12.6 with matplotlib (2.1.0).