Mnt/multi imageset #25734

tacaswell · 2023-04-20T18:59:20Z

PR Summary

This is not ready for review at all yet, but opening so that I can refer to it in the readme of https://github.com/tacaswell/mpl-imageset-demo. This code is .... sub optimal and needs to be refactored (the compare method should not sometimes write out baseline images and sometimes do comparisons!), but it

The rough scheme is as such:

replace the baseline directory with a text file (currently called image_list.txt where each line is a : separated tuple of (relative path to file, revision number, timestamp)
using git blame we can extract the last commit where any given line was changed
via ENV (and eventually better interfaces) you can re-direct where the image comparison machinery looks for images
in one mode running the test suite will generate the baseline images and write out a json file noting the rev number (from the file), the sha the image was last changed in, and the current version of Matplotlib
in the normal mode the test suite will read the json file and verify that the image in the baseline images is consistent with what we get from image_list.txt

An example of the image list file looks like

test_a/A.pdf:1:1681408722.5270584
test_a/A.png:0:1681408661.3898573
test_a/A.svg:0:1681408661.3898566
test2/B.pdf:0:1681408661.389891
test2/B.png:0:1681408661.3898895

The logic on this format is :

use : as a separator because it forbidden in windows paths the leading paths will be relative (so no drive letters)
information must be in one line so git blame is easy to reason about
version rev number is for humans to read and reason about
the timestamp is to ensure that two people versioning reving the same image in different PRs will cause a merge conflict. There needs to be something semi-unique here. An alternative might be to add a short note justifying why, your initials, asking people to hit random keys, using uuid4, or ... would also work.

The json dumped in the generated baseline directory looks like

{
  "test_basic/image.png": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 0,
    "sha": "8691c2d2bf868a23c80f6ac85ba184a917e49f03"
  },
  "test_basic/line.pdf": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 0,
    "sha": "8691c2d2bf868a23c80f6ac85ba184a917e49f03"
  },
  "test_basic/line.png": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 0,
    "sha": "8691c2d2bf868a23c80f6ac85ba184a917e49f03"
  },
  "test_basic/line.svg": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 0,
    "sha": "8691c2d2bf868a23c80f6ac85ba184a917e49f03"
  },
  "test_other/hist.png": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 1,
    "sha": "0000000000000000000000000000000000000000"
  },
  "test_other/scatter.pdf": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 0,
    "sha": "8691c2d2bf868a23c80f6ac85ba184a917e49f03"
  },
  "test_other/scatter.png": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 0,
    "sha": "8691c2d2bf868a23c80f6ac85ba184a917e49f03"
  },
  "test_other/scatter.svg": {
    "mpl_version": "3.8.0.dev913+g8ce98ade16.d20230420",
    "rev": 0,
    "sha": "8691c2d2bf868a23c80f6ac85ba184a917e49f03"
  }
}

The sha is for the computer, the rev is for the human, and the mpl version is for debugging.

open questions

how do deal with non-git checkouts. As this is mostly going to be around released versions so I think that narrows the problem space quite a bit.
tooling around updating these
how to post diffs when images are changed
how to generate cached version of the baseline on merge to main
how to generate and distribute "blessed" versions on tag

PR Checklist

Documentation and Tests

Has pytest style unit tests (and pytest passes)
Documentation is sphinx and numpydoc compliant (the docs should build without error).
New plotting related features are documented with examples.

Release Notes

New features are marked with a .. versionadded:: directive in the docstring and documented in doc/users/next_whats_new/
API changes are marked with a .. versionchanged:: directive in the docstring and documented in doc/api/next_api_changes/
Release notes conform with instructions in next_whats_new/README.rst or next_api_changes/README.rst

tacaswell · 2023-04-21T00:25:56Z

See tacaswell/mpl-imageset-demo#1 for what a PR would look like

see https://github.com/tacaswell/mpl-imageset-demo#operation for a skeletal user guide but it covers:

how to run tests
how to regenerate the baseline images
how to validate that the baseline set is consistent with the current checkout
tell the system we have changed a test image
tell the system about a new test image

Presumably manage_baseline_images.py will end up in our tools directory and the the envs will be folded into a pytest plugin / some other way to thread that configuration through.

tacaswell · 2023-05-19T03:36:08Z

tacaswell · 2023-05-19T21:11:26Z

The image_lists.txt for Matplotlib are now checked in so people can take a look at what those look like at full scale.

ksunden · 2023-05-19T21:21:50Z

Should the metadata.json files be included in the repo? or are those intended to be local only?

tacaswell · 2023-05-19T22:28:02Z

Should the metadata.json files be included in the repo? or are those intended to be local only?

They are intended to go with the generated sets of images, however given that we are not at the point where we can actually pull the images out of the repo, but I still want the tests to run they need to be checked in for now. The final version of this PR will squash them out of existence.

tacaswell · 2023-05-19T23:36:56Z

I think this is at a point where (modulo the metadata.json files) where it is starting to be reviewable.

The internal names are...not great, but I think I have pulled it apart enough that it is not complete spaghetti code.

MPLTESTIMAGEPATH='/tmp/test_images2' MPLGENERATEBASELINE=1 pytest

should work and generate you a full tree of test images on this branch!

…_neg_coords"" This partially reverts commit 7b71257. Too many tests were removed, restore the extra tests.

Also eliminate an enum only used in one place

Copied from the demo repo

Flagging that we should generate images via the plugin means we do not have access at import time to if we intend to generate the images so remove this check.

The factor of 100 reduces the window of collisions to 10ms which is an acceptable risk.

Force this out of existence eventually

tacaswell · 2023-06-21T01:16:01Z

This now includes a pytest plugin (😱) so that

pytest --image-baseline=/tmp/test_images --generate-images   # generate the baseline images
pytest --image-baseline=/tmp/test_images -n 20               # test against them

works. The $MPLTESTIMAGEPATH and $MPLGENERATEBASELINE also still work (but the flags "win").

All of these names need some feedback.

tacaswell · 2023-06-21T03:29:56Z

This seems to be working (successfully generated test images with a new freetype and it passed against them)!

I'm now sure that this is going to work, but lots of details left.

tacaswell · 2024-03-13T21:02:36Z

not going to get this done in the next few weeks.

tacaswell added this to the v3.8.0 milestone Apr 20, 2023

github-actions bot added the status: needs rebase label May 16, 2023

tacaswell force-pushed the mnt/multi_imageset branch from c047f0b to 8dab81f Compare May 18, 2023 16:20

github-actions bot removed the status: needs rebase label May 18, 2023

tacaswell force-pushed the mnt/multi_imageset branch 2 times, most recently from 26911a3 to 4a0d7ad Compare May 18, 2023 21:20

ksunden mentioned this pull request Jun 15, 2023

Start using Cirrus CI #24597

Closed

6 tasks

tacaswell force-pushed the mnt/multi_imageset branch from 8da8195 to 5afeca3 Compare June 15, 2023 18:58

github-actions bot added the status: needs rebase label Jun 20, 2023

tacaswell force-pushed the mnt/multi_imageset branch from 5afeca3 to 6125208 Compare June 21, 2023 01:10

tacaswell added 14 commits June 20, 2023 21:11

MNT: py312 deprecates pickling objects in itertools

e908a41

FIX: also account for itertools.count used in _AxesStack

7c1e630

CI: skip tk tests on GHA as well

94e0988

Revert "Revert " Merge pull request matplotlib#4019 from myshen/annot…

0fe8d37

…_neg_coords"" This partially reverts commit 7b71257. Too many tests were removed, restore the extra tests.

TST: not clear how this ever worked

9c21fdc

TST: update test images for text baseline changes

803b4c8

WIP: add a ENV to look for the baseline images out-of-tree

54a04c5

TST: add ENV to cause the test suite to generate baseline images

94518d2

DOC: start to sketch documentation

4b8f248

WIP: barely working proof-of-concept

4d13089

MNT: handle creating the metadata json file if needed

b1b8677

MNT: make keys Path objects

69bf1e1

MNT: refactor directory generation functions

d8dc57c

TST: initial checking of list of test images

eadd0ac

tacaswell and others added 15 commits June 20, 2023 21:12

MNT: refactor compare / generate into stand alone methods

79906c5

Also eliminate an enum only used in one place

ENH: add image helper CLI tool

ad48c12

Copied from the demo repo

ENH: add pytest collector to only select image tests to run

d36c6f8

Shorten the flag name

c5b6287

ENH: Added Pytest Plugin to run image generation

bb7d8a0

MNT: remove errors on baseline images missing

cffa655

Flagging that we should generate images via the plugin means we do not have access at import time to if we intend to generate the images so remove this check.

MNT: change the default libpath to match Matplotlib

5516ddc

WIP: improve cli tool

9680166

MNT: switch to using ts*100 as int for image list timestamp

406bc96

The factor of 100 reduces the window of collisions to 10ms which is an acceptable risk.

MNT: add new test image

2d1bb5f

WIP: regenerate json

30f4519

Force this out of existence eventually

MNT: remove images from image_list

cccab00

WIP

db78223

MNT: use - instead of _ in CLI flags

74f69b8

ENH: add pytest flag + restore default values via ENV

cab2a30

tacaswell force-pushed the mnt/multi_imageset branch from 6125208 to cab2a30 Compare June 21, 2023 01:13

github-actions bot removed the status: needs rebase label Jun 21, 2023

tacaswell added 2 commits June 20, 2023 22:22

PRF: add naive caching

39480ab

PERF: skip existing images when generating

639ab9b

tacaswell modified the milestones: v3.8.0, v3.9.0 Jun 21, 2023

github-actions bot added the status: needs rebase label Aug 2, 2023

tacaswell modified the milestones: v3.9.0, v3.10.0 Mar 13, 2024

tacaswell modified the milestones: v3.10.0, v3.11.0 Sep 18, 2024

tacaswell mentioned this pull request Mar 28, 2025

Proposal for the baseline images problem #16447

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Mnt/multi imageset #25734

Mnt/multi imageset #25734

Uh oh!

tacaswell commented Apr 20, 2023

Uh oh!

tacaswell commented Apr 21, 2023

Uh oh!

tacaswell commented May 19, 2023 •

edited

Loading

Uh oh!

tacaswell commented May 19, 2023

Uh oh!

ksunden commented May 19, 2023

Uh oh!

tacaswell commented May 19, 2023

Uh oh!

tacaswell commented May 19, 2023

Uh oh!

tacaswell commented Jun 21, 2023

Uh oh!

tacaswell commented Jun 21, 2023

Uh oh!

tacaswell commented Mar 13, 2024

Uh oh!

Uh oh!

Uh oh!

Mnt/multi imageset #25734

Are you sure you want to change the base?

Mnt/multi imageset #25734

Uh oh!

Conversation

tacaswell commented Apr 20, 2023

PR Summary

open questions

PR Checklist

Uh oh!

tacaswell commented Apr 21, 2023

Uh oh!

tacaswell commented May 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tacaswell commented May 19, 2023

Uh oh!

ksunden commented May 19, 2023

Uh oh!

tacaswell commented May 19, 2023

Uh oh!

tacaswell commented May 19, 2023

Uh oh!

tacaswell commented Jun 21, 2023

Uh oh!

tacaswell commented Jun 21, 2023

Uh oh!

tacaswell commented Mar 13, 2024

Uh oh!

Uh oh!

tacaswell commented May 19, 2023 •

edited

Loading