fix(testing): reduce flakiness in extension e2e tests #5481
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ef74ce8 to
acb6531
Compare
- add new info message when extension loads - prefix vscode_proxy_uri message with "proxyUri:" to make it easier to find in e2e test - activate extension onStartupFinished
Codecov Report
@@ Coverage Diff @@
## main #5481 +/- ##
=======================================
Coverage 72.44% 72.44%
=======================================
Files 30 30
Lines 1673 1673
Branches 366 366
=======================================
Hits 1212 1212
Misses 398 398
Partials 63 63 Continue to review full report at Codecov.
|
This adds two new methods to assist with testing the test extension in our e2e tests, namely these two methods: - isTestExtensionLoaded - reloadUntilTestExtensionIsLoaded Sometimes code-server loads without loading the test extension. These methods allow us to reload and guarantee it's loaded.
This modifies the test extension using our new methods to reduce flakiness by ensuring the test extension has loaded before proceeding with the test.
acb6531 to
1be591c
Compare
code-asher
approved these changes
Aug 22, 2022
test/e2e/models/CodeServer.ts
Outdated
Comment on lines
297
to
299
| await this.page.waitForLoadState("load") | ||
| // Give it an extra second just in case it's feeling extra slow | ||
| await this.page.waitForTimeout(1000) |
There was a problem hiding this comment.
We do the same in reloadUntilEditorIsReady so I wonder if we can omit?
|
Great sleuthing on the flakiness!! |
Co-authored-by: Asher <ash@coder.com>
Co-authored-by: Asher <ash@coder.com>
jsjoeio
commented
Aug 22, 2022
code-asher
approved these changes
Aug 22, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR modifies our e2e test to reduce flakiness in the extension tests.
Fixes #5472