[RFC] test: one snapshot per fixture test #1152
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for the PR, @bradzacher! typescript-eslint is a 100% community driven project, and we are incredibly grateful that you are contributing to that community. The core maintainers work on this in their personal time, so please understand that it may not be possible for them to review your work immediately. Thanks again! 🙏 Please, if you or your company is finding typescript-eslint valuable, help us sustain the project by sponsoring it transparently on https://opencollective.com/typescript-eslint. As a thank you, your profile/company logo will be added to our main README which receives thousands of unique visitors per day. |
128a179 to
c579078
Compare
Codecov Report
@@ Coverage Diff @@
## master #1152 +/- ##
==========================================
- Coverage 94.04% 94.02% -0.02%
==========================================
Files 120 120
Lines 5220 5220
Branches 1445 1445
==========================================
- Hits 4909 4908 -1
Misses 177 177
- Partials 134 135 +1
|
|
Closing in favour of #1491 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1142
With my recent 3.7 changes to the parser, I found it near impossible to use the snapshot test output, because the file was huge, vscode didn't display changes well. Additionally, because the snapshots were sorted alphabetically, adding a new snapshot with a similar filename / structure caused git to generate a messy diff which made it impossible to review.
I did some research into jest, and there's no way to generate multiple snapshot files for a single test file.
So there were two choices:
I didn't want to rewrite the snapshot algorithm, so I decided to codegen.
It worked really well - a single little script generates one test per fixture for both
parserandtypescript-estree, then it's just the standard test pipeline per package.Scary PR due to the changes, but I think it's worth it.
There is a tradeoff to this approach over having them all in one file - a complete test run takes a decent chunk longer (~12s to ~120s for parser, and ~40s to ~120s for typescript-estree). I would hazard a guess that this is due to the extra disk activity from writing more snaps, and jest probably spends some time setting up and tearing down an isolated environment for each test.
I don't see this as a huge problem, as we will only really run the tests on CI, and locally it's easy to exclude them from your test run because they're all within a folder called
generated.Draft to test the pipeline and gather feedback.
How does it work?
shared-fixtures/fixtures.yarn generate-testswithin theshared-fixturesroot.parserandtypescript-estreefolders.parser,typescript-estree.cdinto both theparserandtypescript-estreeroots, and runyarn test, which will pick up the generated tests, and create snapshots.That's it! Because there is one file per fixture, jest outputs one snapshot per fixture.
I was thinking of extending this in future when #704 is ready, so we can make reviews even easier by including a more focused snapshot as well (one without the noise of range/loc).