In an effort to both improve stability of our CI (reducing the amount of flakes), we will start gathering statistics on what tests fail and how often.

• 1. Generate statistics from CI runs #6676
  • Ran on branch/commit (main, pr)
  • Platform
  • Test(s) that failed
  • Log per failed test
  • Execution time for successful tests (for regression detection)
• 2. Gather statistics from CI run and store/visualize them #6677
  • Test added date (new test?)
  • First failure date
  • Last failure date
  • Failure count
  • Log/trace
  • Commit (that saw the failure)
  • Execution time (for non-failures)
• 3. Notify of new/repeated failures based on certain thresholds
• 4. Analyze failure (type, rate, origin) and try to assign an owner
• 5. Automatically open GitHub issues from flakes, if issue exists, submit error log as comment
• 6. Automatically detect test speed regressions (consider issue creation or notification, alternatively rely on graphs)
• 7. Daily cron-based stress test for tests -> store statistics
• 8. (Bonus) Reduce CI run time (see statistics to identify slow/problematic test)