Skip to content

ci: execute enterprise and non-enterprise e2e tests concurrently #12872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Apr 10, 2024
Merged

ci: execute enterprise and non-enterprise e2e tests concurrently #12872

merged 6 commits into from
Apr 10, 2024

Conversation

aslilac
Copy link
Member

@aslilac aslilac commented Apr 4, 2024 •
edited
Loading

  • Run the non-enterprise tests and the enterprise tests in a matrix, so that they can be parallelized.
  • We can skip installing a ton of stuff by just running make gen/mark-fresh, further speeding things up. If things are out of date, that'll get caught by the dedicated gen job that we already have.

Skipping go install saves about a minute, and running them concurrently can save like 2-10 minutes depending on lots of factors. Generally gonna be more on the 2 side, but still!

@aslilac aslilac changed the title ci: run enterprise and non-enterprise tests in parallel in ci ci: execute enterprise and non-enterprise tests concurrently Apr 4, 2024
@aslilac aslilac changed the title ci: execute enterprise and non-enterprise tests concurrently ci: execute enterprise and non-enterprise e2e tests concurrently Apr 4, 2024
@aslilac aslilac requested a review from kylecarbs April 5, 2024 17:52
@aslilac
Copy link
Member Author

aslilac commented Apr 5, 2024

CI is failing with...

==> Tests failed
26 passed
4 failed
  site/e2e/tests/app.spec.ts › app
  site/e2e/tests/outdatedAgent.spec.ts › ssh with agent v0.27.0
  site/e2e/tests/outdatedCLI.spec.ts › ssh with client v0.27.0
  site/e2e/tests/webTerminal.spec.ts › web terminal
1 timed out
  site/e2e/tests/updateWorkspace.spec.ts › update workspace, new required, mutable parameter added

...and when I run the tests in dogfood I get...

==> Tests failed
24 passed
4 failed
  site/e2e/tests/groups/removeGroup.spec.ts › remove group
  site/e2e/tests/app.spec.ts › app
  site/e2e/tests/outdatedAgent.spec.ts › ssh with agent v0.27.0
  site/e2e/tests/webTerminal.spec.ts › web terminal
3 timed out
  site/e2e/tests/updateWorkspace.spec.ts › update workspace, new optional, immutable parameter added
  site/e2e/tests/startWorkspace.spec.ts › start workspace with ephemeral parameters
  site/e2e/tests/updateWorkspace.spec.ts › update workspace, new required, mutable parameter added

or sometimes something more like...

==> Tests failed
28 passed
3 failed
  site/e2e/tests/app.spec.ts › app
  site/e2e/tests/outdatedCLI.spec.ts › ssh with client v0.27.0
  site/e2e/tests/outdatedAgent.spec.ts › ssh with agent v0.27.0

... and when I run the tests on macOS...

==> Tests failed
25 passed
2 failed
  site/e2e/tests/outdatedAgent.spec.ts › ssh with agent v0.27.0
  site/e2e/tests/outdatedCLI.spec.ts › ssh with client v0.27.0
4 timed out
  site/e2e/tests/externalAuth.spec.ts › external auth device
  site/e2e/tests/app.spec.ts › app
  site/e2e/tests/updateWorkspace.spec.ts › update workspace, new optional, immutable parameter added
  site/e2e/tests/updateWorkspace.spec.ts › update workspace, new required, mutable parameter added

@aslilac
Copy link
Member Author

aslilac commented Apr 5, 2024

this inconsistency is definitely a problem, so I'm gonna work on ironing out as many of these flakes as I can today 😄

@aslilac aslilac requested a review from Emyrk April 10, 2024 18:20
Emyrk
Emyrk approved these changes Apr 10, 2024
View reviewed changes
Copy link
Member

@Emyrk Emyrk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the secret already set on the github side?

@Emyrk Emyrk mentioned this pull request Apr 10, 2024
Closed
@aslilac
Copy link
Member Author

aslilac commented Apr 10, 2024

Is the secret already set on the github side?

yes!

@aslilac aslilac merged commit 9cf2358 into main Apr 10, 2024
@aslilac aslilac deleted the speed-up-e2e-test-ci branch April 10, 2024 21:42
@github-actions github-actions bot locked and limited conversation to collaborators Apr 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@Emyrk Emyrk Emyrk approved these changes

@kylecarbs kylecarbs Awaiting requested review from kylecarbs

Assignees

@aslilac aslilac

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@aslilac @Emyrk