Split Cygwin CI into non-performance and performance test jobs
#2042
+10
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
One job is for all tests except the `performance` tests, while the other job is for only the `performance` tests. The idea is to decrease the total time it takes for all CI jobs to complete in most cases, by splitting the long-running (currently usually about 13 minute) Cygwin job into two less-long jobs.
In the "Test with pytest" step of the Cygwin test jobs. This is to distinguish the newly split jobs from each other more clearly when glancing at their steps for a run.
This removes an unnecessary trailing slash that I had not used consistently anyway.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One job is for all tests except the
performancetests, while the other job is for only theperformancetests.The idea is to decrease the total time it takes for all CI jobs to complete in most cases, by splitting the long-running (currently usually about 13 minute) Cygwin job into two less-long jobs.
This seems to work well. The performance tests take longer than all the other tests combined. Even with the other steps--mainly setting up Cygwin--taking about 2.5 minutes, this still completes significantly faster than before by allowing the jobs to run in parallel.
A secondary advantage is that incomplete results of tests that are more often of interest can be seen sooner. (I say that's secondary because it could alternatively have been achieved by using a plugin that allow the order of the tests to be specified, and running the performance tests last.)
I'll wait for them to complete here in this PR, and I'll look at the timings, to make sure that wasn't a fluke (also because I would not want to merge a PR whose CI isn't passing, without good reason).
Edit: Yes, the longer of the jobs still completes 4 minutes sooner than the combined test would have completed. (Extrapolating this to future runs presumes that there will not usually be a huge queue of jobs for a Windows runner, or that there will not be a huge queue in the situations where one most wants jobs to complete faster. I think this is a reasonable assumption, but definitely one that should be open to challenge or revision.) All non-xfail tests continue to pass and the output remains readable.
However, before merging this, I'm going to experiment with showing the arguments in the step name, which I suspect may make the jobs easier to distinguish from each other when glancing at their steps.