chore: improve CI reliability #16169
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o pass Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
johnstcn
reviewed
Jan 17, 2025
johnstcn
reviewed
Jan 17, 2025
ThomasK33
reviewed
Jan 17, 2025
johnstcn
approved these changes
Jan 17, 2025
mtojek
reviewed
Jan 17, 2025
matifali
reviewed
Jan 17, 2025
Co-authored-by: Muhammad Atif Ali <atif@coder.com>
mtojek
approved these changes
Jan 17, 2025
hugodutka
reviewed
Jan 17, 2025
Signed-off-by: Danny Kopping <danny@coder.com>
matifali
approved these changes
Jan 20, 2025
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have an effort underway to replace
dbmem(#15109), and consequently we've begun running our full test-suite (with Postgres) on all supported OSs - Windows, MacOS, and Linux, since #15520.Since this change, we've seen a marked decrease in the success rate of our builds on
main(note how the Windows/MacOS failures account for the vast majority of failed builds):We're still investigating why these OSs are a lot less reliable. It's likely that the VMs on which the builds are run have different characteristics from our Ubuntu runners such as disk I/O, network latency, or something else.
In the meantime, we need to start trusting CI failures in
mainagain, as the current failures are too noisy / vague for us to correct.We've also considered hosting our own runners where possible so we can get OS-level observability to rule out some possibilities.
See the meeting notes where we linked into this for more detail.
This PR introduces several changes:
nightly-gauntletworkflowtradeoff: this means that any regressions may be more difficult to discover since we merge to main several times a day
mainon Windows/MacOStest-gois still running the full test-suite against all OSs (including the CLI ones), but will soon be removed once Remove the in-memory database #15109 is completed since it usesdbmemnightly-gauntletto run at 4AM: we've seen several instances of the runner being stopped externally, and we're guessing this may have something to do with the midnight UTC execution time, when other cron jobs may runnightly-gauntletjobs since they haven't passed in a long time, indicating that nobody cares enough to fix them and they don't provide diagnostic value; we can restore them later if necessaryI've manually run both these new workflows successfully:
ci: https://github.com/coder/coder/actions/runs/12825874176/job/35764724907nightly-gauntlet: https://github.com/coder/coder/actions/runs/12825539092