This is heavy machinery. I assume we cannot get away cheaply by collecting the requests first and send them to kpsewhich in a single batch run?

I think that would request quite a bit of reworking of the innards of dviread :(

Fair enough. Just wanted to make sure we're not overlooking an easier solution.

I guess the other solution would be to revive @jkseppan's series of PRs #10236, #10238, #10268. From my PoV these PRs (while likely actually implementing a better solution) basically died from the use of sqlite as cache format, which is reputedly something really useful to know but which I know nothing about :-(

Edit: Yet another idea would be to call kpsewhich once to get pdftex.map, and immediately list all the files that are referenced in it, and then just build a gigantic kpsewhich query that locates all the files at once, caching that. From a quick test this is still reasonably fast on linux, but can run into argc length limits on mac :( Or possibly start kpsewhich interactively, feed the whole query to it, and terminate it, which should avoid the buffering issues...
On the other hand some systems seem to have a truly enormous pdftex.map, where doing the whole query is actually significantly slower...

I spent some more time looking at this problem. I can think of two other solutions (which I guess I'm mostly writing for my own reference, but heh :-)):

Solution 1: Plain (e)TeX actually has a way to directly query glyph sizes:

$ latex '\documentclass{article}\begin{document}'
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=latex)
 restricted \write18 enabled.
entering extended mode
LaTeX2e <2020-02-02> patch level 5
L3 programming layer <2020-08-07>
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/article.cls
Document Class: article 2019/12/20 v1.4l Standard LaTeX document class
(/usr/local/texlive/2020/texmf-dist/tex/latex/base/size10.clo))
(/usr/local/texlive/2020/texmf-dist/tex/latex/l3backend/l3backend-dvips.def)
(./texput.aux)
*\font\1=cmr10\message{\the\fontcharwd\1`X}
7.50002pt

and TeX actually appears to flush output, which makes it controllable via subprocess.Popen. The nice thing here is that this completely gets rid of the need to have a tfm parser, as TeX basically does that work for us. We'll need to spend a bit more time grokking the specs to understand the various scale factors involved, but that should hopefully be not too complicated. On the other hand I don't think this handles vf fonts, so we'd need to figure out 1) whether we can detect the need for a .vf without actually kpsearching it (perhaps they are needed only if a glyph slot is empty?), and 2) if they are indeed often needed (if not, then we can fall back to kpsewhich when they are needed, but we'll still have a big speedup in the common case).

Solution 2: luatex embeds and exposes kpathsea, and can likewise be used interactively

In [1]: import subprocess

In [2]: p = subprocess.Popen(["luatex", "\\relax"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0)

In [3]: p.stdout.readline()
Out[3]: b'This is LuaTeX, Version 1.12.0 (TeX Live 2020) \n'

In [4]: p.stdout.readline()
Out[4]: b' restricted system commands enabled.\n'

In [5]: p.stdout.readline()
Out[5]: b'\n'

In [6]: p.stdin.write(b'\\directlua{print(kpse.find_file("pncr8r", "tfm"))}\n')
Out[6]: 99

In [7]: p.stdout.readline()
Out[7]: b'*/usr/local/texlive/2020/texmf-dist/fonts/tfm/adobe/ncntrsbk/pncr8r.tfm\n'

(the * is just the tex prompt; if the file is not found *nil\n is printed.)
The advantage of this method is that it should also work on Windows (where subprocesses are very slow to start and therefore this technique would bring the largest speedup). On the other hand, this means that luatex would become an optional dependency for performance (it doesn't cost much to keep the old direct kpsewhich invocations if luatex is not present).

Is luatex a big burden? I think most major dists have it now, don't they?

See #19551, which implements the luatex-based solution (with some encoding-related wrinkles still left, but it's mostly there).

Mostly superseded by #19558; we can reopen if there's appetite for a solution that specifically doesn't require luatex.