Bug: tsconfig.json not respected in new files when multiple tsconfigs include the same directory with projectService enabled
#8835
Comments
|
I can confirm this behavior of erroring with the My reproduction (see config files below):
Kapture.2024-04-06.at.18.32.20.mp4Error message: Reproduction files:
import eslintTypescript from '@typescript-eslint/eslint-plugin';
import typescriptParser from '@typescript-eslint/parser';
/** @type {import('@typescript-eslint/utils/ts-eslint').FlatConfig.ConfigArray} */
const config = [
{
files: ['**/*.js', '**/*.jsx', '**/*.ts', '**/*.tsx'],
languageOptions: {
parser: typescriptParser,
parserOptions: {
project: './tsconfig.json',
EXPERIMENTAL_useProjectService: true,
warnOnUnsupportedTypeScriptVersion: true,
},
},
plugins: {
'@typescript-eslint': {
rules: eslintTypescript.rules,
},
},
rules: {
'@typescript-eslint/no-unnecessary-condition': 'warn',
},
},
];
export default config;
{
"$schema": "https://json.schemastore.org/tsconfig",
"compilerOptions": {
"lib": ["DOM", "DOM.Iterable", "ESNext"],
"target": "ES2015",
"moduleResolution": "Bundler",
"moduleDetection": "force",
"resolveJsonModule": true,
"esModuleInterop": true,
"isolatedModules": true,
"allowJs": true,
"allowSyntheticDefaultImports": true,
"downlevelIteration": true,
"forceConsistentCasingInFileNames": true,
"noEmit": true,
"noFallthroughCasesInSwitch": true,
"skipLibCheck": true,
"strict": true,
"incremental": true,
"noUncheckedIndexedAccess": true,
"checkJs": true,
"jsx": "preserve",
"strictNullChecks": true,
"module": "esnext"
},
"include": [
"**/*.ts",
"**/*.tsx",
"**/*.js",
"**/*.jsx",
"**/*.cjs",
"**/*.mjs"
],
"exclude": ["node_modules", "build", "public", "scripts"]
} |
|
Hey good news, I tried this out again today and it looks like the problem is fixed! I tried the two mentioned reproductions and a new clean repo going off our Getting Started docs. I haven't dug deep but I my hunch is some combination of #8815 + #9097 would at least be good places to look. @fire332 @karlhorky are you able to confirm whether this issue is still happening for you? 🙏 |
|
Without changing any of the configs and just updating dependencies, the behavior described in the opening comment remains identical. @JoshuaKGoldberg Have you ensured the ESLint extension is installed and ESLint completed initializing in step 4 before moving on to step 5? Versions
|
|
I'm also still experiencing the problem described in my comment, using ESLint v9, the latest v8 alpha and the new
|
|
Hmm. Interesting. I'll take another look, thanks. In case it's useful, I'm only testing with v8 - as of yesterday, |
|
I've just tested on the v8 version with ESLint v9 with the patch from #9205 as well and the same issue occurs. The repro has been updated on the |
|
@fire332 my patch would fix your base compiler options not getting pulled in where you extends in the tsconfig.json. Stuff like this should start working. "paths": {
"@common/*": ["./common/*"],
},Two thoughts: First, are we sure this isn't a tsserver issue? To find the bug for my patch I had to enable logging here: With something like this: logger: {
close: doNothing,
endGroup: doNothing,
getLogFileName: () => undefined,
hasLevel: () => true,
info: (s) => console.info(`INFO: ${s}`),
loggingEnabled: () => true,
msg: (s, type) => console.log(`MSG (${type}): ${s}`),
perftrc: (s) => console.log(`PERF: ${s}`),
startGroup: doNothing,
},I was using the CLI, so if this is an editor only bug you may need to write to a file unless your editor can capture stdout. You can also try the environment variable, but it didn't work for me on the CLI running eslint. You may also get double logs if your editor runs tsserver + eslint language servers that could overwrite themselves so probably best to leave the filename alone to append the PID if this happens to work through vscode. I also ran with the environment variable With logging in eslint and tsserver I was able to trace the issue I found. I raise that it might be tsserver as I see file watchers in my logs like this: @JoshuaKGoldberg would you accept a PR to allow the tsserver logging to be enabled? It was really helpful in running down project service related issues. The second thought is why are we overlapping includes in tsconfigs? The setup I use is below. All A large scale project that follows this method is https://github.com/Effect-TS/effect if you wish to look at an extended example. .
├── packages
│ ├── package-a
│ │ ├── tsconfig.build.json # includes lib (or src) files, has refs to workspace dependencies' tsconfig.build.json
│ │ ├── tsconfig.json # has refs to lib and test
│ │ └── tsconfig.lib.json # includes lib (or src) files, has refs to workspace dependencies' tsconfig.json
│ │ └── tsconfig.test.json # includes tests files, has refs to lib (or src) and workspace dependencies' tsconfig.json
│ ├── package-b
│ │ ├── tsconfig.build.json # includes lib (or src) files,has refs to workspace dependencies' tsconfig.build.json
│ │ ├── tsconfig.json # has refs to lib and test
│ │ └── tsconfig.lib.json # includes lib (or src) files, has refs to workspace dependencies' tsconfig.json
│ │ └── tsconfig.test.json # includes tests files, has refs to lib (or src) and workspace dependencies' tsconfig.json
├── tsconfig.base.json # has shared compiler options, EVERY other tsconfig extends this one
├── tsconfig.build.json # has refs to each package's tsconfig.build.json
├── tsconfig.json # has refs to each package's tsconfig.json and tsconfig.utils.json
├── tsconfig.utils.json # includes root level stuff like my eslint.config.js, esbuild.config.js, etcFor a large project you will need to increase Keep in mind that this is pretty slow as the option indicates. My linting for my work project has gone from a couple minutes to 20 minutes and I suspect there is some race condition as it sometimes goes for an hour before the CI server kills it. Most of that overhead appears to just be tsserver resolving files. My project is a mixed frontend/backend monorepo using vue which doesn't help since it looks, at a glance at least so take with a grain of salt, to send every code block in a .vue file to be resolved (e.g., I think Here is an example of a vue file that seems to keep getting opened triggering resolution in tsserver on a single file lint from the CLI with https://github.com/vuejs/vue-eslint-parser. |
Absolutely, that'd be great. Yes please! I'd consider it a different issue from this: feel free to file a new issue and/or send a PR. If you do only one, I can do the other. 🙂 |
|
Did some log analysis to see if I could find the inflection point between the original file and a copied file as I am able to reproduce on a large repo. It did not work on a small example repo. The last line is where the difference occurs. Original File typescript-eslint:typescript-estree:useProgramFromProjectService Opening project service file for: <REDACTED>/packages/project-a/src/stores/auth.ts at absolute path <REDACTED>/packages/project-a/src/stores/auth.ts
typescript-eslint:tsserver INFO: 'Search path: <REDACTED>/packages/project-a/src/stores'
typescript-eslint:tsserver INFO: 'For info: <REDACTED>/packages/project-a/src/stores/auth.ts :: Config file name: <REDACTED>/packages/project-a/tsconfig.json'
typescript-eslint:tsserver INFO: 'Creating configuration project <REDACTED>/packages/project-a/tsconfig.json'This log message happens here: https://github.com/microsoft/TypeScript/blob/27bcd4cb5a98bce46c9cdd749752703ead021a4b/src/server/editorServices.ts#L2462 Copied File typescript-eslint:typescript-estree:useProgramFromProjectService Opening project service file for: <REDACTED>/packages/project-a/src/stores/auth2.ts at absolute path <REDACTED>/packages/project-a/src/stores/auth2.ts
typescript-eslint:tsserver INFO: 'Search path: <REDACTED>/packages/project-a/src/stores'
typescript-eslint:tsserver INFO: 'For info: <REDACTED>/packages/project-a/src/stores/auth2.ts :: Config file name: <REDACTED>/packages/project-a/tsconfig.json'
typescript-eslint:tsserver INFO: 'FileWatcher:: Added:: WatchInfo: <REDACTED>/packages/project-a/src/stores/tsconfig.json 2000 undefined WatchType: Config file for the inferred project root'That log message is set here when setting up the watch: https://github.com/microsoft/TypeScript/blob/27bcd4cb5a98bce46c9cdd749752703ead021a4b/src/server/editorServices.ts#L2192 I haven't had the time to do a full code level trace or fire up the debugger yet, but at a glance it looks like tsserver treats the new file as an orphan. Since I cannot reproduce on a small repo, it may suggest a race condition. Wrapping up for the night I took a shot in the dark at changing this line in the plugin to use It didn't fix the issue, but this week is also my first venture into the tsserver code, so I may not be using it correctly. To reproduce the logging this is roughly the code I ended up using at const util = require('util');
const debug = require('debug');
const log = debug('typescript-eslint:tsserver');
function createProjectService(optionsRaw, jsDocParsingMode) {
const service = new tsserver.server.ProjectService({
...
logger: {
close: doNothing,
endGroup: doNothing,
getLogFileName: () => undefined,
hasLevel: () => true,
info: (s) => log(`INFO: %s`, util.inspect(s, false, null, false)),
loggingEnabled: () => true,
msg: (s, type) => log(`MSG (%s): %s`, util.inspect(type, false, null, false), util.inspect(s, false, null, false)),
perftrc: (s) => log(`PERF: %s`, util.inspect(s, false, null, false)),
startGroup: doNothing,
},
...
});
...
}I am using neovim but with the vscode eslint LSP. I just started my editor with the |
|
I ran into the same warning but with a different setup.
I don't use I did some debugging. It looks like typescript-eslint/packages/typescript-estree/src/parser.ts Lines 235 to 253 in 63e53e2 The first program is created through The first cycle uses The second cycle uses That in turn explains the warning because Since my problem is caused by the same file being compiled multiple times due to |
|
To expand upon @fluidsonic's findings, using the file watchers (without actually watching the filesystem) looks like it may be a viable solution. I created a small proof-of-concept to try and isolate certain behaviors. It will unfortunately require some state management. I track watches requested by tsserver in a Trie data structure to find the most specific watch. Triggering it after creating a new file fixed the inferred project issue. This means it will use the correct I'm sure more is required to integrate into the plugin efficiently since the plugin is just getting files from eslint without necessarily having access to if the file is new or changed since the last project update. It doesn't really come with much performance benefits. It was slower on my test single file repo, hanging while waiting for the For Expand src/index.ts.../* eslint-disable @typescript-eslint/no-non-null-assertion */
/* eslint-disable @typescript-eslint/no-empty-function */
import ts from 'typescript';
import path from 'node:path';
import fs from 'node:fs';
const eventHandler: ts.server.ProjectServiceEventHandler = (event) => {
console.log(event);
};
const logger: ts.server.Logger = {
close: () => {},
endGroup: () => {},
getLogFileName: (): undefined => undefined,
hasLevel: () => true,
info: (s) => {
console.log('INFO:', s);
},
loggingEnabled: () => true,
msg: (s) => {
console.log('MSG:', s);
},
perftrc: (s) => {
console.log('PERF:', s);
},
startGroup: () => {},
};
class TrieNode<T> {
children: Map<string, TrieNode<T>>;
value: T | null;
constructor(public readonly path: string) {
this.children = new Map();
this.value = null;
}
}
class Trie<T> {
root: TrieNode<T>;
constructor() {
this.root = new TrieNode('');
}
insert(path: string) {
if (!path.startsWith('/')) throw new Error('absolute paths only');
const parts = path.split('/').slice(1); // drop the first empty string
const { currentNode } = parts.reduce(
({ currentNode, rootPath }, part) => {
path = `${rootPath}/${part}`;
if (!currentNode.children.has(part)) {
currentNode.children.set(part, new TrieNode(path));
}
return {
currentNode: currentNode.children.get(part)!,
rootPath: path,
};
},
{
currentNode: this.root,
rootPath: this.root.path,
},
);
return currentNode;
}
get(path: string): TrieNode<T> | null {
const parts = path.split('/');
const { lastNodeWithValue } = parts.reduce(
({ currentNode, lastNodeWithValue }, part) => {
if (!currentNode.children.has(part)) {
return { currentNode: currentNode, lastNodeWithValue };
}
const childNode = currentNode.children.get(part)!;
return {
currentNode: childNode,
lastNodeWithValue: childNode.value !== null ? childNode : lastNodeWithValue,
};
},
{
currentNode: this.root,
lastNodeWithValue: null as TrieNode<T> | null,
},
);
return lastNodeWithValue;
}
}
class Watcher implements ts.FileWatcher {
constructor(
private readonly node: TrieNode<Watcher>,
public readonly callback: () => void,
) {}
close() {
this.node.value = null;
}
}
const watchers = new Trie<Watcher>();
const stubFileWatcher = (
path: string,
callback: ts.FileWatcherCallback,
pollingInterval?: number,
options?: ts.WatchOptions,
): ts.FileWatcher => {
console.log('WATCHFILE:', path, pollingInterval, options);
const node = watchers.insert(path);
if (node.value !== null) {
return node.value;
}
const watcher = new Watcher(node, () => {
callback(path, ts.FileWatcherEventKind.Changed, new Date());
});
node.value = watcher;
return watcher;
};
const stubDirectoryWatcher = (
path: string,
callback: ts.DirectoryWatcherCallback,
recursive?: boolean,
options?: ts.WatchOptions,
): ts.FileWatcher => {
console.log('WATCHDIR:', path, recursive, options);
const node = watchers.insert(path);
if (node.value !== null) {
return node.value;
}
const watcher = new Watcher(node, () => {
callback(path);
});
node.value = watcher;
return watcher;
};
const host: ts.server.ServerHost = {
...ts.sys,
clearImmediate,
clearTimeout,
setImmediate,
setTimeout,
watchFile: stubFileWatcher,
watchDirectory: stubDirectoryWatcher,
};
const options: ts.server.ProjectServiceOptions = {
host, // host capabilities
logger, // logger
cancellationToken: ts.server.nullCancellationToken, // cancel in flight requests
useSingleInferredProject: false, // treat files without a tsconfig as separate projects
useInferredProjectPerProjectRoot: false, // don't lump files without a tsconfig that are in the same directory together
// typingsInstaller: undefined, // use default
eventHandler, // get those diags
canUseWatchEvents: true, // use setup watches
suppressDiagnosticEvents: false, // we want the diagnostics
throttleWaitMilliseconds: 0, // 200, // debounce requests
globalPlugins: [], // no extra global plugins
pluginProbeLocations: [], // just look for plugins in node_modules
allowLocalPluginLoads: true, // allow plugins
// typesMapLocation: undefined, // use default
serverMode: ts.LanguageServiceMode.Semantic, // full type checking
session: undefined, // lsp interface, this PoC has eslint as interface so don't use
jsDocParsingMode: ts.JSDocParsingMode.ParseAll, // parse all jsdoc
};
const projectService = new ts.server.ProjectService(options);
const file = ts.server.toNormalizedPath(`${process.cwd()}/src/index.ts`);
const formatDiagnostic = (diagnostic: ts.Diagnostic) => {
if (diagnostic.file === undefined) return; // TODO: could be whole project
if (diagnostic.start === undefined) return; // TODO: could be whole file
const { line, character } = diagnostic.file.getLineAndCharacterOfPosition(diagnostic.start);
const message = ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n');
// eslint-disable-next-line @typescript-eslint/restrict-template-expressions
console.log(`${diagnostic.file.fileName} (${line + 1},${character + 1}): ${message}`);
};
const printDiagnostic = (file: ts.server.NormalizedPath) => {
console.log('GETTING DIAGNOSTICS FOR:', file);
projectService.openClientFile(file, undefined, ts.ScriptKind.TS, process.cwd());
const scriptInfo = projectService.getScriptInfo(file);
if (!scriptInfo) {
console.log('NO SCRIPT INFO');
process.exit(1);
}
const project = projectService.getDefaultProjectForFile(
scriptInfo.fileName,
true, // finds project or creates inferred project
);
if (project) {
const languageService = project.getLanguageService();
const program = languageService.getProgram();
if (program) {
const source = program.getSourceFile(file);
const diagnostics = program.getSemanticDiagnostics(source);
diagnostics.forEach(formatDiagnostic);
}
}
};
printDiagnostic(file);
const destination = ts.server.toNormalizedPath(path.join(path.dirname(file), 'index2.ts'));
fs.copyFileSync(file, destination);
const watcher = watchers.get(destination);
if (watcher !== null && watcher.value !== null) watcher.value.callback();
printDiagnostic(destination);
fs.rmSync(destination);With this block executed after creating a new file, tsserver finds and associates to the correct project. const watcher = watchers.get(destination);
if (watcher !== null && watcher.value !== null) watcher.value.callback();More log output...If I comment that section out, the new file is assigned to an inferred project. More log output... |
|
So I have a hobbled together set of what I hope are the start of fixes . One note if anyone clones, the First, I track watches set by the This is used by createProjectService.ts to call the watch constructors via the host and useProgramFromProjectService.ts to trigger the watches when a file has not been seen by the I use Also noteworthy is that calling
edit: Been looking through the heap. I'm thinking it might be "okay". We use a lot of complex types which is likely inflating the memory usage. The |
|
If anyone wants to test, I've moved to this branch: https://github.com/higherorderfunctor/typescript-eslint/tree/patches2upstream. There are a lot of modifications, so I'll work to get these up-streamed over several PRs. I've confirmed the I've included the package.json{
...
"pnpm": {
"patchedDependencies": {
"eslint-plugin-import-x@0.5.1": "patches/eslint-plugin-import-x@0.5.1.patch",
},
// forces dependencies to use the same version
"overrides": {
"typescript": "$typescript",
"typescript-eslint": "$typescript-eslint",
"@typescript-eslint/eslint-plugin": "$typescript-eslint",
"@typescript-eslint/parser": "$typescript-eslint",
"@typescript-eslint/rule-schema-to-typescript-types": "$typescript-eslint",
"@typescript-eslint/scope-manager": "$typescript-eslint",
"@typescript-eslint/type-utils": "$typescript-eslint",
"@typescript-eslint/utils": "$typescript-eslint",
"@typescript-eslint/visitor-keys": "$typescript-eslint",
// my custom fork
"@typescript-eslint/types": "higherorderfunctor/typescript-eslint#patches2upstream&path:packages/types",
"@typescript-eslint/typescript-estree": "higherorderfunctor/typescript-eslint#patches2upstream&path:packages/typescript-estree"
}
},
"devDependencies": {
"eslint": "^8.57.0",
"typescript": "rc",
"typescript-eslint": "rc-v8",
// just for reference on what I have tested against
"@eslint/eslintrc": "^3.1.0",
"@eslint/js": "^8.57.0",
"@stylistic/eslint-plugin": "^2.1.0",
"@types/eslint": "^8.56.10",
"@typescript-eslint/utils": "rc-v8", // uses some types in my eslint.config.js for type checking
"@vue/eslint-config-prettier": "^9.0.0",
"@vue/eslint-config-typescript": "^13.0.0",
"eslint-config-airbnb-base": "^15.0.0",
"eslint-config-airbnb-typescript": "^18.0.0",
"eslint-config-prettier": "^9.1.0",
"eslint-import-resolver-alias": "^1.1.2",
"eslint-import-resolver-typescript": "^3.6.1",
"eslint-plugin-codegen": "0.28.0",
"eslint-plugin-deprecation": "^3.0.0",
"eslint-plugin-import-x": "^0.5.1",
"eslint-plugin-jsonc": "^2.16.0",
"eslint-plugin-markdown": "^5.0.0",
"eslint-plugin-prefer-arrow-functions": "^3.3.2",
"eslint-plugin-prettier": "^5.1.3",
"eslint-plugin-promise": "^6.2.0",
"eslint-plugin-security": "^3.0.0",
"eslint-plugin-simple-import-sort": "^12.1.0",
"eslint-plugin-sonar": "^0.14.1",
"eslint-plugin-sonarjs": "^1.0.3",
"eslint-plugin-sort-destructure-keys": "^2.0.0",
"eslint-plugin-tsdoc": "^0.3.0",
"eslint-plugin-vue": "^9.26.0",
"eslint-plugin-vuejs-accessibility": "^2.3.0",
"eslint-plugin-vuetify": "^2.4.0",
"eslint_d": "^13.1.2",
"jsonc-eslint-parser": "^2.4.0",
"vue-eslint-parser": "^9.4.3",
...
},
...
}My RAM utilization is way down today... not sure if it is something I did, something upstream in the If you do get OOM, the heap can be increased. NODE_OPTIONS="--max-old-space-size=8184" eslint ...Known Issues With the flat config and Additional Options I've added two new options with their defaults below. {
languageOptions: {
parserOptions: {
projectService: {
incremental: false, // when true, only sends changes to tsserver instead of whole files
maximumOpenFiles: 16, // auto closes open files with tsserver based on access order
}
}
}
} |
|
Marking as accepting PRs since I'm conferencing much of this month. Caveats:
|
|
I mentioned this in another thread, but I have a busy week this week in case someone else needs to pick up the PR. Dropping my notes in case anyone else does pick it up. The working implementation is https://github.com/higherorderfunctor/typescript-eslint/tree/patches2upstream, the draft PR is not currently in a working state. While I have a working implementation, it duplicates some of the existing logic in
Exporting or proving an access function for
const filePathMatchedByConfiguredProject = (
service: ts.server.ProjectService,
filePath: string,
): boolean => {
for (const project of service.configuredProjects.values()) {
if (project.containsFile(filePath as ts.server.NormalizedPath)) {
return true;
}
}
return false;
}; |
tsconfig.json not respected in new files when multiple tsconfigs include the same directory with useProjectService enabledtsconfig.json not respected in new files when multiple tsconfigs include the same directory with projectService enabled
|
👋 @higherorderfunctor now that #9306 is merged (thanks again!), do you think you'll have time to un-draft #9353 in the next few days? For context, we plan on releasing v8 in three days: midday Wednesday EST. I'm also wondering if this is just not something we can reasonably be tackling internally in typescript-eslint. ESLint's architecture right now doesn't give us any indication as to whether it's single-run or when files are closed in an editor (eslint/rfcs#102). We have existing issues with file changes in editors (see #3536 & its links)... |
|
@JoshuaKGoldberg I can probably get a minimum viable implementation done by Monday. Basically, is the file in the project service? If no, trigger the best matching watcher. Recheck and if still no add to default project or treat as an isolated file. This will deal with new source files requiring the LSP to be restarted. Some of the more complex decision making I don't think I'd be able to have ready by release day. Examples:
|
|
Yeah, these are a lot of open questions. I really really appreciate your offer to hustle but long-term I don't think it's a given that we'd want to try to handle these cases internally (#8835 (comment)). I'm going to go ahead and remove this from the v8 milestone.
That's one of the things I'm very hopeful for with the project service! |
|
Also experiencing this issue with strict is actually enabled on each config in monorepo My eslint.config.js: import eslint from '@eslint/js'
import prettier from 'eslint-config-prettier'
import ts from 'typescript-eslint'
export default ts.config(
eslint.configs.recommended,
prettier,
...ts.configs.stylisticTypeChecked,
...ts.configs.strictTypeChecked,
{
languageOptions: { parserOptions: { projectService: true, tsconfigRootDir: import.meta.dirname }
}
)
|
Before You File a Bug Report Please Confirm You Have Done The Following...
Issue Description
When
EXPERIMENTAL_useProjectService: trueand at least twotsconfig.jsons include a non-empty common directory, typescript-eslint will not respecttsconfig.jsonfor new/renamed files until ESLint is reloaded.Possibly related: #8206 #7435
Reproduction Repository Link
https://github.com/fire332/typescript-eslint-bug-repro/
Repro Steps
pnpm install.prettierrc.jsoreslint.config.jsand wait for ESLint to initialize.tsfile in./srcor./src-altDeveloper: Reload Windowfrom the command paletteVersions
typescript-eslint7.5.0TypeScript5.4.3ESLint8.57.0node>=20.11.1The text was updated successfully, but these errors were encountered: