-
-
Notifications
You must be signed in to change notification settings - Fork 7.9k
Use test cache for test result images too. #15932
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
If we have a few tests that are not deterministic, will this cause the cache to increase without bound? I think that we cache these folders on at least travis so we would want to make sure we don't start spending more time moving around invalid images than we save by not computing them. |
Fair enough, added a simple (automatic) cache eviction mechanism. |
I don't fully follow this, but it seems useful. But needs a rebase, and probably a review from @tacaswell or @QuLogic |
For image comparison tests in svg/pdf/ps formats, the result images are converted to png for comparison. Previously the conversion results were cached for the baseline images, but not for the test-generated images (because of non-deterministic svg/pdf/etc. results, due to hash-salting, dict ordering, etc.). Now that the test-generated images are generally deterministic, we can enable the cache for baseline images as well. This speeds up `pytest -k '[svg]'` by ~30% (81s initially -> 55s on a seeded cache) and `pytest -k '[pdf]'` by ~10% (62s -> 55s) (there are too few (e)ps image comparison tests to see an effect). Also add logging regarding the cache which may help troubleshooting determinacy problems. A simple cache eviction mechanism prevents the cache from growing without bounds, limiting it to 2x the size of the baseline_images directory. This is a much simpler version of PR7764, which added more sophisticated reporting of cache hits and misses and cache eviction.
rebased |
For image comparison tests in svg/pdf/ps formats, the result images are
converted to png for comparison. Previously the conversion results were
cached for the baseline images, but not for the test-generated images
(because of non-deterministic svg/pdf/etc. results, due to
hash-salting, dict ordering, etc.).
Now that the test-generated images are generally deterministic, we can
enable the cache for baseline images as well. This speeds up
pytest -k '[svg]'
by ~30% (81s initially -> 55s on a seeded cache) andpytest -k '[pdf]'
by ~10% (62s -> 55s) (there are too few (e)ps imagecomparison tests to see an effect). Also add logging regarding the
cache which may help troubleshooting determinacy problems.
This is a much simpler version of PR#7764, which added more sophisticatedreporting of cache hits and misses, as well as cache invalidation (I
think the cache can reasonably be cleaned manually, at least for now).
A simple cache eviction mechanism prevents the cache from growing
without bounds, limiting it to 2x the size of the baseline_images
directory.
This is a much simpler version of PR7764, which added more sophisticated
reporting of cache hits and misses and cache eviction.
Supersedes (mostly) #7764 (@jkseppan).
(Note that the gain in speed is less than reported than in #7764 (comment), because I have also optimized the svg and gs converters in the meantime by running a single process and interacting with it over stdin/stdout, rather that launching a new instance of inkscape/ghostscript for each image.)
PR Summary
PR Checklist