Skip to content

DOC: Add JUnit XML parsing example #1451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

user27182
Copy link

@user27182 user27182 commented Apr 8, 2025

The documentation for using junit xml file is helpful but is geared towards integration with tools like CircleCI. For some users, however, it may be easier and simpler (or otherwise preferable) to parse the XML manually (e.g. to generate test cases with pytest for CI).

This PR adds a small code snippet to the docs for parsing the XML.

@user27182 user27182 marked this pull request as draft April 8, 2025 06:26
@user27182 user27182 marked this pull request as ready for review April 8, 2025 06:49
@user27182 user27182 changed the title Add parametrized test cases for execution times from JUnit XML file Add parametrized test cases for execution times from JUnit XML file Apr 8, 2025

.. code-block:: shell

pytest sphinx_gallery/tests/test_execution_times.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite a long example to include in the doc. One option would be to use https://sphinx-design.readthedocs.io/en/latest/ with dropdown but I think it might be better to throw this in a gist and link to that. That way it can be independently updated / adjusted etc.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not familiar with gist, so not sure exactly what that process would look like for this example ... please advise on next steps if this is the way we want to move forward.

Some other ideas:

  1. First add the actual tests, and add the docs in a follow-up PR with a direct link to the repository's new test module. That way the linked test code is always up to date.
  2. Create a new public API for parsing the XML. Then we can shorten the example quite a bit by replacing the parsing code with a simple function call.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go to https://gist.github.com/ it should hopefully make sense how to add code there, and then getting a link to it, etc. But yes if you plan to have it as a test somewhere and want to link to it then that would work, too.

Don't really want to add an XML parsing thing since it seems like a too-general problem

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go to https://gist.github.com/ it should hopefully make sense how to add code there, and then getting a link to it, etc. But yes if you plan to have it as a test somewhere and want to link to it then that would work, too.

Thanks, maybe gist is the way to go after all?
Looks like the tests are currently failing because they only work post-build since they first require the xml file to be generated. So this may require modifying CI a bit to accommodate the tests if you want the tests to actually work for testing the repo's examples, OR we can make a fixture instead to artificially simulate having a pre-generated xml and run the tests on that.

But, IMO this repo doesn't really need these tests, and using a fixture is kind of pointless other than to validate that the tests at least work on some level. So all of this may be too much change/effort for little to no actual benefit for the repo.

Hence, maybe it's best to remove the actual tests from this PR, publish it as a gist as suggested, and keep it all strictly docs-only for this PR. All depends on how reproducible you want this type of example to be, since it's possible the docs example could become out of date, e.g. if the xml path changes for some reason in the future.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, maybe your tests do generate docs by default, and the tests in this PR just need to be tweaked? I.e. maybe the PROJECT_DIR path isn't quite right or differs slightly in CI compared to when I make the docs and run the tests locally.

Basically, I'm not sure if I should try to debug the tests to make them pass or you'd prefer for me to remove them and stick with just having an example in the docs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, the example in this PR is used by PyVista in pyvista/pyvista#7386, and I thought it would be good to document this use-case here for others to use.

So, we could also broadly link to PyVista for an example of using the XML with pytest. Lots of options, not sure what's preferred here by sphinx-gallery

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as tests go you should be able to modify

suite = lxml.etree.fromstring(contents)

In the docs I think a link to a PyVista page or a gist would be better than a big code block inline here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as tests go you should be able to modify

suite = lxml.etree.fromstring(contents)

Thanks for the tip. The key was that a fixure is needed to generate the xml. I tried integrating the tests, but the parametrization does not allow using fixtures, and so things get a bit complicated and then the actual tests would deviate substantially from the ones given as an example in the docs.

And so, I've removed the tests altogether.

In the docs I think a link to a PyVista page or a gist would be better than a big code block inline here

I refactored the code in the docs and the example is much shorter now (23 lines instead of 60). Is this still too long? If so, I'll update it and make use of these other options

@larsoner larsoner mentioned this pull request Apr 8, 2025
@user27182 user27182 changed the title Add parametrized test cases for execution times from JUnit XML file DOC Add pytest example for parametrized tests from JUnit XML file Apr 9, 2025
@user27182 user27182 changed the title DOC Add pytest example for parametrized tests from JUnit XML file DOC: Add JUnit XML parsing example Apr 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants