backend_driver.py overhaul #8069

dstansby · 2017-02-12T18:19:53Z

Whilst doing some documentation changes, I noticed backend_driver.py is very out of date. Annoyingly each example has to be explicitly set in the file for it to be tested. This PR updates backend_driver.py to:

Automatically discover every example file
Correctly strip plt.show calls etc.

In the future this file might be a way provide a good way of benchmarking (#2188) by timing how long all the examples take to run.

examples/tests/backend_driver.py

QuLogic · 2017-02-12T23:46:51Z

Given how you keep hitting this file in your docathon PRs, I think we should merge this sooner rather than later.

dstansby · 2017-02-13T22:27:48Z

Okay, I think this is almost good to go. The only thing I have left to do is to actually run this on a variety of backends and update the files to exclude. If anyone else wants a go, feel free to push to my branch with files to exclude.

examples/tests/backend_driver.py

QuLogic · 2017-02-26T01:41:58Z

Is this complete? Do you want to rebase and squash a bit? There are a quite a few commits for just one file change. In any case, a rebase is needed because of the conflicts.

dstansby · 2017-02-28T14:35:07Z

I think this is almost done, I'll give it a squash/rebase/check sometime soon.

dstansby · 2017-03-13T10:26:17Z

There's a big ol' squash, I will test this on Agg at some point soon, and if it all works I think this is good to merge (after a couple of reviews).

Testing on other backends can probably then be added a 'new contributor friendly' issue.

dstansby · 2017-03-14T19:11:30Z

Right, this is now working fully on Agg. How do people feel about merging this now?

Current todo, which I think could each be opened as new-contributor-friendly issues:

Run on each of the other backends and check that all the examples work
Re-factor so running the script works if not in the examples/tests directory
Output the results to a csv file instead of just printing them to the terminal

QuLogic

LGTM otherwise.

examples/tests/backend_driver.py

NelleV · 2017-03-16T19:36:13Z

This file should probably be moved to the tools folder.

dstansby · 2017-03-17T11:35:48Z

@NelleV would you mind if I added that move to "Re-factor so running the script works if not in the examples/tests directory"? It seems a logical thing to do at the same time.

QuLogic · 2017-03-17T23:28:17Z

Are you referring to a different PR?

dstansby · 2017-03-17T23:53:41Z

Sorry, I'm referring to my mini todo list here in my above comment #8069 (comment)

dstansby · 2017-10-09T19:51:30Z

I have rebased this on to current master if anyone wants to take it for a spin.

tacaswell · 2017-10-22T03:54:32Z

Current output:

Some of these are because the existing __future__ import is split across 2 lines. Might be worth just dropping that logic and trusting that the examples have the proper __future__ imports they need?

Backend agg took 5.23 minutes to complete
  Failures: 
     '../misc/rc_traits_sgskip.py'
     '../misc/svg_filter_line.py'
     '../misc/print_stdout_sgskip.py'
     '../misc/rasterization_demo.py'
     '../misc/agg_buffer.py'
     '../misc/svg_filter_pie.py'
     '../misc/multiprocess_sgskip.py'
     '../userdemo/pgf_preamble_sgskip.py'
     '../text_labels_and_annotations/legend_demo.py'
     '../animation/movie_demo_sgskip.py'
        template ratio 1.127, template residual 0.591
Backend ps took 4.55 minutes to complete
  Failures: 
     '../misc/agg_buffer_to_array.py'
     '../misc/rc_traits_sgskip.py'
     '../misc/svg_filter_line.py'
     '../misc/print_stdout_sgskip.py'
     '../misc/rasterization_demo.py'
     '../misc/agg_buffer.py'
     '../misc/svg_filter_pie.py'
     '../misc/multiprocess_sgskip.py'
     '../userdemo/pgf_preamble_sgskip.py'
     '../text_labels_and_annotations/rainbow_text.py'
     '../text_labels_and_annotations/legend_demo.py'
     '../animation/movie_demo_sgskip.py'
        template ratio 0.979, template residual -0.097
Backend svg took 5.25 minutes to complete
  Failures: 
     '../misc/agg_buffer_to_array.py'
     '../misc/rc_traits_sgskip.py'
     '../misc/svg_filter_line.py'
     '../misc/print_stdout_sgskip.py'
     '../misc/rasterization_demo.py'
     '../misc/agg_buffer.py'
     '../misc/svg_filter_pie.py'
     '../misc/multiprocess_sgskip.py'
     '../userdemo/pgf_preamble_sgskip.py'
     '../text_labels_and_annotations/rainbow_text.py'
     '../text_labels_and_annotations/legend_demo.py'
     '../animation/movie_demo_sgskip.py'
        template ratio 1.130, template residual 0.604
Backend pdf took 4.85 minutes to complete
  Failures: 
     '../misc/agg_buffer_to_array.py'
     '../misc/rc_traits_sgskip.py'
     '../misc/svg_filter_line.py'
     '../misc/print_stdout_sgskip.py'
     '../misc/rasterization_demo.py'
     '../misc/agg_buffer.py'
     '../misc/svg_filter_pie.py'
     '../misc/multiprocess_sgskip.py'
     '../userdemo/pgf_preamble_sgskip.py'
     '../text_labels_and_annotations/rainbow_text.py'
     '../text_labels_and_annotations/legend_demo.py'
     '../animation/movie_demo_sgskip.py'
        template ratio 1.045, template residual 0.210
Backend template took 4.64 minutes to complete
  Failures: 
     '../images_contours_and_fields/quadmesh_demo.py'
     '../misc/agg_buffer_to_array.py'
     '../misc/rc_traits_sgskip.py'
     '../misc/svg_filter_line.py'
     '../misc/print_stdout_sgskip.py'
     '../misc/demo_agg_filter.py'
     '../misc/rasterization_demo.py'
     '../misc/patheffect_demo.py'
     '../misc/agg_buffer.py'
     '../misc/svg_filter_pie.py'
     '../misc/multiprocess_sgskip.py'
     '../mplot3d/wire3d_animation.py'
     '../mplot3d/rotate_axes3d.py'
     '../showcase/xkcd.py'
     '../userdemo/pgf_preamble_sgskip.py'
     '../text_labels_and_annotations/rainbow_text.py'
     '../text_labels_and_annotations/usetex_fonteffects.py'
     '../text_labels_and_annotations/tex_demo.py'
     '../text_labels_and_annotations/usetex_demo.py'
     '../text_labels_and_annotations/legend_demo.py'
     '../text_labels_and_annotations/usetex_baseline_test.py'
     '../axes_grid1/demo_axes_grid2.py'
     '../animation/movie_demo_sgskip.py'
     '../animation/animation_demo.py'
        template ratio 1.000, template residual 0.000

anntzer · 2017-10-22T05:08:54Z

I would just run the file as it is, without trying to play with stripped lines. sys.path can by set by PYTHONPATH and the backend can be set by MPLBACKEND (and if the backend is Agg, then plt.show() is already a noop). If we worry about spurious savefigs (which we probably should indeed, we should just move to a tmpdir before running the examples (and we can append importing savefig and calling savefig). Examples that need to call np.seterr should do so themselves.

Edit: I was worried this would fail if the example explicitly wanted to set a backend itself but it looks like it can actually do that:

$ MPLBACKEND=Agg python -c 'import matplotlib as m; m.use("qt5agg")'

works.

tacaswell

Maybe rethink how we handle the imports and removing show.

tacaswell · 2017-10-30T19:11:50Z

punting to 2.2

dstansby · 2018-10-10T16:14:36Z

Attempting to revive this, so I've rebased. I think it's really quite a useful thing to be able to check all the examples run properly.

examples/tests/backend_driver_sgskip.py

Remove old pylab option Do plt. search for lines to strip PEP8 Print fname if excluding Use pylab_examples as default directory Create file and directory list in same loop Add docstring to end of --help Remove reporting of missing tests By default test all discovered files Don't test user interface examples Properly exclued excluded files Clean up configuration of files to not run dict formatting backend_driver fixes Use argparser Use argparse better Revert wrong changes Remove redundant elif

These are demos that require user interaction

timhoffm

The cleanup itself looks technically good (and it's not critical anyway as this is not part of the shipped library).

I'm wondering though, how realistic this is when we leave out the rendering. Would it make sense to always use savefig? OTOH, this feels like a reimplementation of sphinx-gallery. Can't we use just that for testing?

dstansby · 2019-07-18T21:00:52Z

That's a good point, this PR is so old that it pre-dates sphinx-gallery (at least used by us). Maybe it is obsolete now. On the other hand, probably no harm in merging this and making the script useable (as you say).

QuLogic · 2020-06-20T03:57:24Z

Replaced by #17226.

dstansby · 2020-06-20T09:23:24Z

🎉 happy that this (or a variant on it) finally ended up being merged!

QuLogic reviewed Feb 12, 2017

View reviewed changes

QuLogic reviewed Feb 14, 2017

View reviewed changes

dstansby added the status: needs revision label Mar 1, 2017

dstansby added this to the 2.2 (next next feature release) milestone Mar 1, 2017

dstansby force-pushed the backend-driver branch from 17664d0 to 5a0ed16 Compare March 13, 2017 10:22

dstansby self-assigned this Mar 13, 2017

dstansby removed the status: needs revision label Mar 14, 2017

dstansby changed the title ~~[WIP] backend_driver.py overhaul~~ backend_driver.py overhaul Mar 14, 2017

dstansby modified the milestones: 2.1 (next point release), 2.2 (next next feature release) Mar 14, 2017

QuLogic mentioned this pull request Mar 16, 2017

Removes OldScalarFormatter #7956 #8295

Merged

QuLogic reviewed Mar 16, 2017

View reviewed changes

examples/tests/backend_driver.py Outdated Show resolved Hide resolved

examples/tests/backend_driver.py Outdated Show resolved Hide resolved

dstansby removed their assignment Mar 17, 2017

tacaswell modified the milestones: 2.1 (next point release), 2.2 (next next feature release) Aug 29, 2017

dstansby mentioned this pull request Oct 6, 2017

Restore something like backend_driver #9284

Closed

dstansby force-pushed the backend-driver branch from d22326d to 1d19358 Compare October 9, 2017 19:50

tacaswell force-pushed the backend-driver branch from 28e8c12 to a51cfc8 Compare October 22, 2017 04:25

tacaswell requested changes Oct 30, 2017

View reviewed changes

tacaswell modified the milestones: v2.1.1, v2.2 Oct 30, 2017

dstansby force-pushed the backend-driver branch from fa939b7 to 1c9f576 Compare October 10, 2018 16:14

anntzer reviewed Oct 10, 2018

View reviewed changes

examples/tests/backend_driver_sgskip.py Outdated Show resolved Hide resolved

dstansby and others added 8 commits July 18, 2019 14:18

Exclude some ginput examples

ec8f2b3

Correct helpstrings in argument parser

6fb5899

DOC: update example incantation to match actual filename

ff86975

MNT: update how subprocess is called

1a651a2

MNT: account for renames of files to skip

64126cb

These are demos that require user interaction

ENH: tweak output a bit to be more readable

df8d446

STY: whitespace + remove unused imports

258818f

dstansby force-pushed the backend-driver branch from 1c9f576 to 258818f Compare July 18, 2019 13:18

Remove __future__ strip

1632aa3

timhoffm approved these changes Jul 18, 2019

View reviewed changes

QuLogic added Maintenance status: needs rebase labels Apr 1, 2020

anntzer mentioned this pull request Apr 23, 2020

Replace backend_driver by new example runner. #17226

Merged

6 tasks

QuLogic closed this Jun 20, 2020

dstansby deleted the backend-driver branch June 20, 2020 09:23

story645 removed this from the future releases milestone Oct 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backend_driver.py overhaul #8069

backend_driver.py overhaul #8069

dstansby commented Feb 12, 2017

QuLogic commented Feb 12, 2017

dstansby commented Feb 13, 2017

QuLogic commented Feb 26, 2017

dstansby commented Feb 28, 2017

dstansby commented Mar 13, 2017

dstansby commented Mar 14, 2017 •

edited

Loading

QuLogic left a comment

NelleV commented Mar 16, 2017

dstansby commented Mar 17, 2017

QuLogic commented Mar 17, 2017

dstansby commented Mar 17, 2017

dstansby commented Oct 9, 2017

tacaswell commented Oct 22, 2017

anntzer commented Oct 22, 2017 •

edited

Loading

tacaswell left a comment

tacaswell commented Oct 30, 2017

dstansby commented Oct 10, 2018

timhoffm left a comment

dstansby commented Jul 18, 2019

QuLogic commented Jun 20, 2020

dstansby commented Jun 20, 2020

backend_driver.py overhaul #8069

backend_driver.py overhaul #8069

Conversation

dstansby commented Feb 12, 2017

QuLogic commented Feb 12, 2017

dstansby commented Feb 13, 2017

QuLogic commented Feb 26, 2017

dstansby commented Feb 28, 2017

dstansby commented Mar 13, 2017

dstansby commented Mar 14, 2017 • edited Loading

QuLogic left a comment

Choose a reason for hiding this comment

NelleV commented Mar 16, 2017

dstansby commented Mar 17, 2017

QuLogic commented Mar 17, 2017

dstansby commented Mar 17, 2017

dstansby commented Oct 9, 2017

tacaswell commented Oct 22, 2017

anntzer commented Oct 22, 2017 • edited Loading

tacaswell left a comment

Choose a reason for hiding this comment

tacaswell commented Oct 30, 2017

dstansby commented Oct 10, 2018

timhoffm left a comment

Choose a reason for hiding this comment

dstansby commented Jul 18, 2019

QuLogic commented Jun 20, 2020

dstansby commented Jun 20, 2020

dstansby commented Mar 14, 2017 •

edited

Loading

anntzer commented Oct 22, 2017 •

edited

Loading