Skip to content

Enhancement: allow altering the file names that project: true searches for #7056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
4 tasks done
me4502 opened this issue May 24, 2023 · 15 comments
Closed
4 tasks done
Labels
duplicate This issue or pull request already exists

Comments

@me4502
Copy link

me4502 commented May 24, 2023

Before You File a Proposal Please Confirm You Have Done The Following...

Relevant Package

parser

My proposal is suitable for this project

  • I believe my proposal would be useful to the broader TypeScript community (meaning it is not a niche proposal).

Description

Currently, the project: true option allows TypeScript-ESLint to auto-detect the tsconfig.json file within each package of a monorepo, however a relatively common setup is to use a tsconfig.eslint.json file for eslint setup. This means that instead of being able to utilise project: true, each package needs an eslintrc file that points to the project file, or globs need to be used in the base configuration. Neither of these are ideal either for massive duplication reasons or for performance reasons.

Ideally, another option would be added to allow setting the filename that the parser looks for when project is set to true. This would allow utilising a tsconfig.eslint.json file without needing to manually define it in all used locations.

Fail

N/A - enhancement request for configuration option, not related to a code problem

Pass

N/A - enhancement request for configuration option, not related to a code problem

Additional Info

No response

@me4502 me4502 added enhancement New feature or request triage Waiting for team members to take a look labels May 24, 2023
@Josh-Cena
Copy link
Member

The idea is to mimic what tsc does when --project is not provided—both tsc and VS Code looks for tsconfig.json by default. Would project: "**/tsconfig.eslint.json" work for you?

@bradzacher
Copy link
Member

bradzacher commented May 24, 2023

One of the reasons we introduced this mode was to remove/lessen the need for tsconfig.eslint.json.

Previously with the explicit path pattern it was really hard to split up your lint over multiple processes using monorepo tools like nx because you had to use eslint override configs to specifically set like this:

{
  overrides: [
    {
      files: ['package1/**/*.ts'],
      parserOptions: {
        project: ['package1/tsconfig.json'],
      }
    },
    {
      files: ['package2/**/*.ts'],
      parserOptions: {
        project: ['package2/tsconfig.json'],
      }
    },
  ],
}

Which was manual and painful to maintain.

This meant that people would usually just do the config once like project: ['package1/tsconfig.json', 'package2/tsconfig.json'] - which then causes perf problems and possible OOMs. Merging the configs into one using a tsconfig.eslint.json was a possible solution because it worked around the memory issue and did hackily allow people to keep using single-process ESLint to lint their entire codebase.

With project: true you no longer need to do this complicated config and can just do one top-level config with no overrides. When you spin up one lint process per package you automatically have a lint run isolated to that package - which means that there's near-zero chance of OOMs and increased perf due to parallelisation.

This is the recommended way to scale linting on a monorepo rather than relying on the out-of-the-box eslint CLI which is inherently single-threaded and doesn't allow for any memory management.

With all that being said - the big question is "why do you still want to use a tsconfig.eslint.json?" Is there some usecase you're solving for by still using it? Is it still being used because you're doing single-process linting on your monorepo?

@bradzacher bradzacher added awaiting response Issues waiting for a reply from the OP or another party and removed triage Waiting for team members to take a look labels May 24, 2023
@me4502
Copy link
Author

me4502 commented May 24, 2023

The current reason for not doing project: '**/tsconfig.eslint.json' has been performance, as we have a large number of workspaces (~50) setup using yarn 3 workspaces, and many files in some of them. We're currently running linting in 3 ways:

  • In IDE, which suffers relatively bad memory issues when it comes to file parsing
  • In a precommit hook, which runs one process across the entire codebase
  • In per-workspace yarn lint scripts

We have per-project tsconfig.eslint.json files (& .eslintrc.js files) so that they can be run per-workspace, and then a root one that includes all .ts files for the precommit hook. The reason we don't just use tsconfig.json files for these is that those are set to not include test files, whereas we want to also lint test files with eslint

@bradzacher
Copy link
Member

The reason we don't just use tsconfig.json files for these is that those are set to not include test files, whereas we want to also lint test files with eslint

I would actually say that that's the opposite way to do things - you have a tsconfig.build.json which specifies what files are included in your build and then your tsconfig.json specifies all of the "checkable" ts files (including tests and tooling).

The TS LSP also works in the same way we've designed just looking for the nearest tsconfig.json - which is why you should endeavour to use this approach. If you don't use it then the TS LSP will create a "default program" for the code it can't find a tsconfig for and it will make it work, but it won't use the config you probably want it to.

@JoshuaKGoldberg
Copy link
Member

FWIW I've found tsconfig.eslint.json to be useful for root-level config files:

@bradzacher
Copy link
Member

bradzacher commented May 24, 2023

@JoshuaKGoldberg but wouldn't that be solved by doing what I mentioned - inverting the problem?

So you'd have tsconfig.json with include: ['.'] and allowJs, and tsconfig.build.json with just include: ['src']?

Otherwise as I mentioned things might get weird with tools that expect tsconfig.json to cover things.

@JoshuaKGoldberg
Copy link
Member

It hasn't been an issue. The language service seems to always pick up the root-level tsconfig.json file for settings in files like .eslintrc.cjs. And I haven't seen any issues with other tools.

@bradzacher
Copy link
Member

I guess the question is then - why not invert the problem instead to avoid the eslint-only config?

How would your project typecheck tests and such as it stands? Does it use the eslint config to typecheck the non src files?

@JoshuaKGoldberg
Copy link
Member

Inverting feels weird to me 🤔. I'm accustomed to the root-level tsconfig.json being what is used by tsc / building in general. Every variation or other build process is its own, separate thing.

typecheck tests

Tests are treated the same as non-test files. They're type checked and compiled normally. In that style of template, .npmignore / package.json "files" makes them ignored in published outputs. In Astro/Next.js/Remix equivalents, source files aren't transpiled at all.

@bradzacher
Copy link
Member

To me it seems weird to transpile tests - as the project grows this will slow down the build step a lot - which is why we are cautious to exclude them in this project. Imagine if our massive blob of fixtures was included in the transpiration then ignored by the publish - it'd be very slow!

Typechecking them - 100%, but transpiling them - no way!

@me4502
Copy link
Author

me4502 commented May 25, 2023

I guess for inversion IMO what would make the most sense as the "main" tsconfig.json is the main thing they're being used for. While normally I'd consider that builds, if the main tsconfig.json is also used by tsserver for the IDE I could see a reason to have one that touches all files as the main one. I'd consider linting by itself as a weird task to consider as needing the main tsconfig, but in the repo I made this ticket about we do have a lot of libraries that don't build and therefore don't need a main tsconfig file for that purpose. So I moved all of those over and set project: true for them, where the others that still use a tsconfig.eslint.json keep an override in their .eslintrc.js files.

So basically I'm wondering if there's something beyond linting that will make use of a tsconfig.json file when present that would benefit from all files being included, such as IDE-typechecking

@bradzacher
Copy link
Member

bradzacher commented May 25, 2023

From my understanding the IDE typechecking will look for the nearest tsconfig.json that includes the file or it'll use the settings in the root tsconfig if it doesn't find one and sort of "make do".

So it's sort of best effort and the IDE should mostly "just work" no matter your setup.


The thing we're looking into now is if we can use the LSP codebase to power our types and replace the manual algorithm we built. If we can then it'd work the same way the IDE does. I'm not sure if we can customise what tsconfigs it looks for? If we can't then adding the feature in now would mean a hard breaking change down the line when we switch the internals.

It would require investigation to confirm.

@JoshuaKGoldberg
Copy link
Member

JoshuaKGoldberg commented Jul 30, 2023

Following up: in #6754 we added an EXPERIMENTAL_useProjectService flag that acts much more like an IDE (it uses the same "project service" APIs internally). Documenting it is tracked in #7349 and #7350.

Since that flag is experimental -and will be for quite some time-, we still don't have a stable way for situations where tsconfig.json shouldn't include JS files. Which is actually not unusual in the wild - many projects have standardized on explicitly not allowing non-TS files in the root tsconfig.json.

cc @me4502 - thanks for talking with us through it! I've marked the discussion comments as hidden & off-topic so that folks wanting to implement it don't get discouraged by the long thread (cc @bradzacher too). Please yell at me if you think the comments should be surfaced more visibly - maybe we should make a GH Discussion for them?

At any rate, I think we'd want to see an API suggestion before marking this as purely accepting prs. What would this look like in an ESLint config file? In #101 (comment), @bradzacher suggested:

  • project: true - just use the closest tsconfig.json on disk for each linted file.
  • project: { allowedTsconfigNames: FileName[], ignoredTsconfigs: RelativePath[] }
    • allowedTsconfigNames to specify the set of names you want us to search (default ['tsconfig.json']
    • ignoredTsconfigs to ignore specific tsconfigs you know would be useless.

I like that idea. Maybe, allowed: [...] and ignored: [...] for simplicity?

On the other hand, it could get confusing having this object shape have true-style lookups enabled while string and string[] shapes have it disabled. Maybe we should have an explicit relative: boolean flag in there?

project: {
  allowed: ['tsconfig.eslint.json', 'tsconfig.json'],
  ignored: ['./packages/ignore-me/tsconfig.json'],
  relative: true,
}

@JoshuaKGoldberg JoshuaKGoldberg added awaiting response Issues waiting for a reply from the OP or another party and removed awaiting response Issues waiting for a reply from the OP or another party labels Jul 30, 2023
@JoshuaKGoldberg JoshuaKGoldberg changed the title Enhancement: allow altering the file name that project: true searches for Enhancement: allow altering the file names that project: true searches for Jul 30, 2023
@JoshuaKGoldberg
Copy link
Member

...wow. This is an awful way to represent many collapsed comments:

Screenshot of many posts in this thread marked as off-topic and hidden

Sorry for stealing your issue @me4502 but I'm going to just file a new one and un-hide those. I really don't want people who want to discuss or implement this issue to have to wade through all that.

@JoshuaKGoldberg
Copy link
Member

#7383

@JoshuaKGoldberg JoshuaKGoldberg added duplicate This issue or pull request already exists and removed enhancement New feature or request awaiting response Issues waiting for a reply from the OP or another party labels Jul 30, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
duplicate This issue or pull request already exists
Projects
None yet
Development

No branches or pull requests

4 participants