I hope that most of the differences are in small anti-aliasing artifacts so I would expect the svg tests (which do the rasterization locally) should be a bit better than the png tests?

For my understanding:

• Is there an obvious reason that exactly aarch64, ppc64le and s390x need increased tolerances?
• Did you check that one needs the same tolerance increment for all the architectures or did you just copy the value from aarch64 and it worked?

When we have dug into these issues in detail in the past they have either been true broken (because we were doing nasty things an the c-level that assumed byte order and I think we have fixed all of those) or issues with very small variations in float values.

In the past the differences have been single bit flips in a single channel in ~dozen pixels in the image. We are essentially binning full float precision down to 256 (or less) bins as the final step of the rendering (because we save 32bit RGBA images). In most cases changes at the limits of float precision don't really matter (because it all goes down to the same bin), but if we get unlucky a tiny change can flip which side of a boundary the value is on and increase / decrease the value in the in the final output by 1 (which then massively magnifies the change).

I was essentially wondering if it makes sense to pull out the architectures and not define the same machine lookup at every place individually: i.e. instead of

@image_comparison(['fancyarrow_dpi_cor_100dpi.png'], remove_text=True,
                  tol={'aarch64': 0.02, 'ppc64le': 0.02,
                       's390x': 0.02}.get(platform.machine(), 0),

use

                  tol=0.02 if platform.machine() in EXTRA_TOL_MACHINES else 0),

or

                  tol=_add_extra_tol(0.02),

with

def _add_extra_tol(extra_tol, default=0):
    if platform.machine() in ['aarch64', 'ppc64le', 's390x']:
        return default + extra_tol
    else:
        return default

For this to make sense, there should be some comment why the extra tolerance is added and I don't know how to describe it. Maybe it's just "machines that empirically need larger tolerances".

These are values based on actual results, though I did not check all the results directly as I have no access to all those systems. I did set the values to match aarch64, but only in the cases where the RMS difference was already the same. Note, there's at least one case where aarch64 has a special tolerance that the other ones here don't.

Looking a little closer, I think it's actually almost all != x86_64, except that the 32-bit systems are already covered by the additional 0.06, so we could flip the check the other way (0 if x86_64, something else otherwise).

I'm a not clear if @timhoffm is advocating for changing the default threshold on all tests on non-x86_64 or just inverting the logic on the handlful that need it.

I may have over-learned worrying about non-0 tolerances (we used to have a blanket tolerance and then had text tests rendering the wrong glyphs but still come in under the threshold), but I think there is value in only tweaking the tests that need it (presumably because we for what ever reason we get unlucky on them) rather than on all tests.

I'm a not clear if @timhoffm is advocating for changing the default threshold on all tests on non-x86_64 or just inverting the logic on the handlful that need it.

That's not exactly my point. I was just seeing a lot

                  tol={'aarch64': val, 'ppc64le': val,
                       's390x': val}.get(platform.machine(), 0),

where the same val (per case) was always applied to the same 3 platforms. This seems to be a common pattern that could be factored out. The functionality can stay exactly as it is now, the only question is if we want to factor it out.

Sorry, I mis-understood you @timhoffm .

I flipped these all to tol=0 if platform.machine() == 'x86_64' else 'something else'.

Let's wait and give @tacaswell a final say.