Hey @JoshuaKGoldberg! I'd be glad to look into this

I looked into this and I've managed to get tests fully green in 4 workspaces, and fell short by just one suite in 2. Main challenges and thoughts:

• Vitest fails to run any test file without test suites defined, and there are some test suites created dynamically.
• process.chwd() can't be used in threads, so unless removed, you'll need to run Vitest without threads, losing most if not all of the performance of the benefits.
• jest-specific-snapshot is not necessary, as there is .toMatchFileSnapshot() in Vitest, which is a huge win, but will require updating thousands of the "specific snapshots" as in Vitest, they don't have metadata, just contents.
• Mocking typescript module doesn't seem to work for me
• Removing require() to package.json of non-existing virtual packages is necessary - it's not possible to mock non-existing packages using Vitest at the moment.

Regarding performance...

Perhaps most interesting, because one of the slowest, is typescript-estree test, one of the ones that "almost pass", which I ran with parse.test.ts deleted.

Here's Jest:

> nx run typescript-estree:test

$ jest --coverage
 PASS  tests/lib/getProjectConfigFiles.test.ts
 PASS  tests/lib/node-utils.test.ts
 PASS  tests/lib/describeFilePath.test.ts
 PASS  tests/lib/convert.test.ts
 PASS  tests/lib/warn-on-unsupported-ts.test.ts
 PASS  tests/lib/semanticInfo-singleRun.test.ts
 PASS  tests/lib/parse.project-true.test.ts
 PASS  tests/lib/createParseSettings.test.ts
 PASS  tests/lib/parse.moduleResolver.placeholder-error.test.ts (6.313 s)
 PASS  tests/lib/parse.moduleResolver.default-program-success.test.ts (5.728 s)
 PASS  tests/lib/parse.moduleResolver.default-program-error.test.ts (5.509 s)
 PASS  tests/lib/parse.moduleResolver.placeholder-success.test.ts (5.608 s)
 PASS  tests/lib/persistentParse.test.ts (10.687 s)
 PASS  tests/lib/semanticInfo.test.ts (14.115 s)
Test Suites: 14 passed, 14 total
Tests:       220 passed, 220 total
Snapshots:   97 passed, 97 total
Time:        14.699 s
Ran all test suites.

Which in Vitest, with threads disabled, performs like this:

> nx run typescript-estree:test

$ vitest run
 RUN  v0.32.2 /Users/wmaj/Projekty/Open source projects/typescript-eslint/packages/typescript-estree
 ✓ tests/lib/semanticInfo.test.ts  (23 tests) 8253ms
 ✓ tests/lib/persistentParse.test.ts  (70 tests) 5093ms
 ✓ tests/lib/parse.moduleResolver.default-program-success.test.ts  (1 test) 1921ms
 ✓ tests/lib/parse.moduleResolver.placeholder-success.test.ts  (1 test) 1932ms
 ✓ tests/lib/parse.moduleResolver.placeholder-error.test.ts  (2 tests) 1538ms
 ✓ tests/lib/parse.moduleResolver.default-program-error.test.ts  (1 test) 1576ms
 ✓ tests/lib/parse.project-true.test.ts  (3 tests) 451ms
 ✓ tests/lib/convert.test.ts  (11 tests) 11ms
 ✓ tests/lib/describeFilePath.test.ts  (84 tests) 5ms
 ✓ tests/lib/warn-on-unsupported-ts.test.ts  (2 tests) 4ms
 ✓ tests/lib/semanticInfo-singleRun.test.ts  (6 tests) 3ms
 ✓ tests/lib/getProjectConfigFiles.test.ts  (10 tests) 1ms
 ✓ tests/lib/node-utils.test.ts  (5 tests) 1ms
 ✓ tests/lib/createParseSettings.test.ts  (1 test) 1ms
 Test Files  14 passed (14)
      Tests  220 passed (220)
   Start at  13:57:52
   Duration  22.37s (transform 182ms, setup 0ms, collect 1.39s, tests 20.79s, environment 0ms, prepare 50ms)

You can see how individual suites are executed in about half the time it takes Jest to execute them. BUT with threads disabled, Vitest can't possibly keep up.

I won't be spending any more time on this, but you're free to continue where I left off if you want to:

https://github.com/wojtekmaj/typescript-eslint/tree/vitest

I also used my script to help me out migrating 40+ repos, although the results were hugely underwhelming because your use cases were vastly different from mine:

https://gist.github.com/wojtekmaj/6defa1f358daae28bd52b7b6dbeb7ab6

Apologies if I'm late to the discussion. I've implemented the codemods that migrate the code from jest to vitest along with the required config files. The tests pass in 8 workspaces present in the project. Below are the results:

>  NX   Ran target test for 11 projects (2m)

   ✔    8/11 succeeded [0 read from cache]

   ✖    3/11 targets failed, including the following:
         - nx run utils:test
         - nx run website:test
         - nx run eslint-plugin:test

threads are enabled for all tests, including typescript-estree, as vitest supports process.chdir and other processess as mentioned in their documentation and the performance seems to be awesome!

Jest

I ran the tests using jest without these changes and received the following results (without cache):

 >  NX   Ran target test for 11 projects (2m)

    ✔    10/11 succeeded [0 read from cache]

    ✖    1/11 targets failed, including the following:
         - nx run website:test

Performance

The results become comparable and vitest easily keeps up with jest. Individual packages also show faster performance with vitest:

    ✔  nx run typescript-estree:test (2s)
    ✔  nx run visitor-keys:test (3s)
    ✔  nx run parser:test (1s)
    
    ✔  nx run scope-manager:test (8s)
    ✔  nx run type-utils:test (5s)
    ✔  nx run ast-spec:test (14s)
    ✔  nx run eslint-plugin-internal:test (13s)
    
    ✔  nx run eslint-plugin-tslint:test (15s)

Test Files changed:

• ast-spec
• eslint-plugin-tslint
• eslint-plugin
• scope-manager
• type-utils
• typescript-estree
• utils
• visitor-keys

Can I interest you in a PR?

Can I interest you in a PR?

@aryaemami59 Absolutely! Anything we mark with the "accepting prs" label is fair game to submit a PR for, no need to ask 🙂 (see contributing docs)

@kirkwaiblinger Awesome, I'll give it a shot. Thanks.

Just a quick update, I am still working on this, it's coming along nicely, I'm just trying to figure out how to minimze the changes necessary and get the integration tests to pass.

I'm not sure if any of you are still working on it, but i've pushed the branch I had for eslint-plugin at least in #10579

if one of you has a better approach, or more work ready to go, feel free to shout up and I'll close mine

we should move one package at a time so this is reviewable, and ideally keep large changes to follow up PRs if possible

@43081j I'm still working on it, should be all done soon https://github.com/aryaemami59/typescript-eslint/tree/migrate-to-vitest.

what's your plan for preparing it for code review?

I doubt we'll be able to PR your whole branch in. it'd be helpful if you could cut it up a bit into one branch per project at least as we could be merging those now while you finish the rest

if you want any help with that, let me know. generally hanging around on discord too

Honestly I'm not sure yet, I would love to get some feedback from the maintainers on how to proceed.

• How much faster would Vitest be?

For the record, here's my local results of the eslint-plugin package tests from #10579 (i.e., time yarn workspace @typescript-eslint/eslint-plugin run test):

Vitest:

real    1m17.117s
user    8m34.753s
sys     0m47.915s


Jest (current):

real    1m43.057s
user    11m19.964s
sys     1m47.260s

how to proceed

First of all, big thanks to everyone in this thread who has already done investigation and POCs to advance this effort!

Taking a look at #10579 and #10582, I think it makes sense to do as much prework as we can to keep the diff minimal when actually cutting over, and keep several things in mind as part of the migration. Seemingly this includes:

• Switch all test() cases to it() in advance, and any similar trivially-avoidable breaking changes
• Restructuring scripts in advance as required in order to work with both test frameworks.
  • chore: update all test scripts to remove coverage flag #10582 addresses the runs with and without coverage. It likely makes sense to move forward with that PR.
  • I think we also need to consider handling single-run/watch-mode in nx/package.json commands. Right now, chore(eslint-plugin): migrate to vitest #10579 activates vitest watch mode locally when running yarn test or yarn workspace @typescript-eslint/eslint-plugin test, so those commands never terminate. While we may want to be able to activate vitest watch mode going forward, I think we want to keep yarn test single-run to match the jest behavior during a migration.
• Using the vitest global API at first, with a plan to switch to explicit imports after migration. (RE chore(eslint-plugin): migrate to vitest #10579 (comment), I definitely acknowledge that the imports don't contribute many lines to the diff, and are certainly dominated by the snapshot changes, but they do add changes to a nonzero number of files that wouldn't otherwise be changed at all! So it's not a huge thing but still preferable)
• Switching one package at a time (as @43081j suggested) for reviewability
• Doing our best (no expectation of perfection!) to ensure we don't run into technical blockers halfway through a migration. While chore(eslint-plugin): migrate to vitest #10579 handles eslint-plugin, which has a lot of tests, its usage of jest/vitest is pretty minimal in a sense. It might be good to prove out first that packages with more complicated mocking function without problems in vitest. For instance, an example of a previous mocking pain point that comes to mind is chore: bump @swc/core to 1.3.1 #5659.

What are people's thoughts of this is as a rough course of action?

@kirkwaiblinger I have pretty much everything done already, I'm just cleaning up to minimize the changes, I can start submitting PRs for each workspace at a time.

I guess the one other important point would be to get a satisfying conclusion to the performance issues brought up in #7112 (comment). I'm confused about this, since the linked doc says that process.chdir() is not supported... Am I missing something?

threads are enabled for all tests, including typescript-estree, as vitest supports process.chdir and other processess as mentioned in their documentation and the performance seems to be awesome!

@kirkwaiblinger process.chdir() used to not be supported but it is now. As far as the performance goes, vitest is definitely better, I'll try to provide some actual numbers as soon as I get a chance.

@kirkwaiblinger process.chdir() used to not be supported but it is now.

Are their docs just out of date? Or is it to do with forks instead of threads??

@kirkwaiblinger The default pool used to be threads but now it is forks, which as mentioned in the docs does support process.chdir().

I see, thanks for the clarification @aryaemami59 !

Alright I think I'm just about ready to start putting up some PRs.

activates vitest watch mode locally when running yarn test or yarn workspace @typescript-eslint/eslint-plugin test, so those commands never terminate

this was a mistake and should've been raised in the PR as part of the review

Using the vitest global API at first

👍

From #10579, running time yarn test -u from packages/eslint-plugin on my M1 Mac Studio:

• main: ~500s user / 60s system
• eslint-plugin-vitest: ~460s user / 40s system

Cool 🙂

All right! We've merged the last of the conversion PRs, meaning every package in typescript-eslint is now onboarded to Vitest. #11068 is now open to clean up the trailing bits of Jest that still exist.

Thanks so much to everyone who participated - in particular @43081j and @aryaemami59 for so much investigation work and PR authoring! 👏 🔥

Kudos @43081j and @aryaemami59!