Skip to content

Watcher improvements #4509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 19, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
275 changes: 171 additions & 104 deletions ci/dev/watch.ts
Original file line number Diff line number Diff line change
@@ -1,139 +1,206 @@
import * as cp from "child_process"
import { spawn, fork, ChildProcess } from "child_process"
import del from "del"
import { promises as fs } from "fs"
import * as path from "path"
import { onLine } from "../../src/node/util"

async function main(): Promise<void> {
try {
const watcher = new Watcher()
await watcher.watch()
} catch (error: any) {
console.error(error.message)
process.exit(1)
}
import { CompilationStats, onLine, OnLineCallback, VSCodeCompileStatus } from "../../src/node/util"

interface DevelopmentCompilers {
[key: string]: ChildProcess | undefined
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to help non-generic functions like Object.entries play nice with the type signature.

vscode: ChildProcess
vscodeWebExtensions: ChildProcess
codeServer: ChildProcess
plugins: ChildProcess | undefined
}

class Watcher {
private readonly rootPath = path.resolve(__dirname, "../..")
private readonly vscodeSourcePath = path.join(this.rootPath, "vendor/modules/code-oss-dev")
private rootPath = path.resolve(process.cwd())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: since we moved vscodeSourcePath any reason not to move rootPath as well? That way paths has all paths.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately JS doesn’t let you reference earlier object properties while still defining entries.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhhh got it! Disregard 👍

private readonly paths = {
/** Path to uncompiled VS Code source. */
vscodeDir: path.join(this.rootPath, "vendor", "modules", "code-oss-dev"),
compilationStatsFile: path.join(this.rootPath, "out", "watcher.json"),
pluginDir: process.env.PLUGIN_DIR,
}

//#region Web Server

private static log(message: string, skipNewline = false): void {
process.stdout.write(message)
if (!skipNewline) {
process.stdout.write("\n")
/** Development web server. */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 for comments

private webServer: ChildProcess | undefined

private reloadWebServer = (): void => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Basically this reloadWebServer kills the webServer if it exists, forks the process (not sure what process.argv.slice(2) is here? (might be nice to name it or add a comment?) and starts up a new child process, logging a message to let us know it started

(and also adding a log for when the process exits)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process.argv.slice(2) is a bit of a node convention. Since this array contains all the CLI arguments, the first two entries are usually the node runtime and the script name, e.g. node entry.js

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL! Thanks for the explainer!

if (this.webServer) {
this.webServer.kill()
}

// Pass CLI args, save for `node` and the initial script name.
const args = process.argv.slice(2)
this.webServer = fork(path.join(this.rootPath, "out/node/entry.js"), args)
const { pid } = this.webServer

this.webServer.on("exit", () => console.log("[Code Server]", `Web process ${pid} exited`))

console.log("\n[Code Server]", `Spawned web server process ${pid}`)
}

public async watch(): Promise<void> {
let server: cp.ChildProcess | undefined
const restartServer = (): void => {
if (server) {
server.kill()
}
const s = cp.fork(path.join(this.rootPath, "out/node/entry.js"), process.argv.slice(2))
console.log(`[server] spawned process ${s.pid}`)
s.on("exit", () => console.log(`[server] process ${s.pid} exited`))
server = s
}
//#endregion

const vscode = cp.spawn("yarn", ["watch"], { cwd: this.vscodeSourcePath })
//#region Compilers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this pattern you've started introducing 👏 Does it have a name? Comment regions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish I could take credit for it but it’s built into VS Code’s region collapsing.


const vscodeWebExtensions = cp.spawn("yarn", ["watch-web"], { cwd: this.vscodeSourcePath })
private readonly compilers: DevelopmentCompilers = {
codeServer: spawn("tsc", ["--watch", "--pretty", "--preserveWatchOutput"], { cwd: this.rootPath }),
vscode: spawn("yarn", ["watch"], { cwd: this.paths.vscodeDir }),
vscodeWebExtensions: spawn("yarn", ["watch-web"], { cwd: this.paths.vscodeDir }),
plugins: this.paths.pluginDir ? spawn("yarn", ["build", "--watch"], { cwd: this.paths.pluginDir }) : undefined,
}

const tsc = cp.spawn("tsc", ["--watch", "--pretty", "--preserveWatchOutput"], { cwd: this.rootPath })
const plugin = process.env.PLUGIN_DIR
? cp.spawn("yarn", ["build", "--watch"], { cwd: process.env.PLUGIN_DIR })
: undefined
private vscodeCompileStatus = VSCodeCompileStatus.Loading

const cleanup = (code?: number | null): void => {
Watcher.log("killing vs code watcher")
vscode.removeAllListeners()
vscode.kill()
public async initialize(): Promise<void> {
for (const event of ["SIGINT", "SIGTERM"]) {
process.on(event, () => this.dispose(0))
}

Watcher.log("killing vs code web extension watcher")
vscodeWebExtensions.removeAllListeners()
vscodeWebExtensions.kill()
if (!this.hasVerboseLogging) {
console.log("\n[Watcher]", "Compiler logs will be minimal. Pass --log to show all output.")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helps quite a bit when dealing with the extension compilation which is often quite verbose and inactionable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes! Basically this hides it unless --log is passed when starting yarn watch? Fantastic DX improvement 🔥

}

Watcher.log("killing tsc")
tsc.removeAllListeners()
tsc.kill()
this.cleanFiles()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this section looks a lot cleaner 🔥


if (plugin) {
Watcher.log("killing plugin")
plugin.removeAllListeners()
plugin.kill()
}
for (const [processName, devProcess] of Object.entries(this.compilers)) {
if (!devProcess) continue

if (server) {
Watcher.log("killing server")
server.removeAllListeners()
server.kill()
devProcess.on("exit", (code) => {
this.log(`[${processName}]`, "Terminated unexpectedly")
this.dispose(code)
})

if (devProcess.stderr) {
devProcess.stderr.on("data", (d: string | Uint8Array) => process.stderr.write(d))
}
}

onLine(this.compilers.vscode, this.parseVSCodeLine)
onLine(this.compilers.codeServer, this.parseCodeServerLine)

Watcher.log("killing watch")
process.exit(code || 0)
if (this.compilers.plugins) {
onLine(this.compilers.plugins, this.parsePluginLine)
}
}

//#endregion

process.on("SIGINT", () => cleanup())
process.on("SIGTERM", () => cleanup())
//#region Line Parsers

vscode.on("exit", (code) => {
Watcher.log("vs code watcher terminated unexpectedly")
cleanup(code)
})
private parseVSCodeLine: OnLineCallback = (strippedLine, originalLine) => {
if (!strippedLine.includes("watch-extensions") || this.hasVerboseLogging) {
console.log("[VS Code]", originalLine)
}

vscodeWebExtensions.on("exit", (code) => {
Watcher.log("vs code extension watcher terminated unexpectedly")
cleanup(code)
})
switch (this.vscodeCompileStatus) {
case VSCodeCompileStatus.Loading:
// Wait for watch-client since "Finished compilation" will appear multiple
// times before the client starts building.
if (strippedLine.includes("Starting 'watch-client'")) {
console.log("[VS Code] 🚧 Compiling 🚧", "(This may take a moment!)")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥🔥 love the emojis in the logs :D

this.vscodeCompileStatus = VSCodeCompileStatus.Compiling
}
break
case VSCodeCompileStatus.Compiling:
if (strippedLine.includes("Finished compilation")) {
console.log("[VS Code] ✨ Finished compiling! ✨", "(Refresh your web browser ♻️)")
this.vscodeCompileStatus = VSCodeCompileStatus.Compiled

this.emitCompilationStats()
this.reloadWebServer()
}
break
case VSCodeCompileStatus.Compiled:
console.log("[VS Code] 🔔 Finished recompiling! 🔔", "(Refresh your web browser ♻️)")
this.emitCompilationStats()
this.reloadWebServer()
break
}
}

tsc.on("exit", (code) => {
Watcher.log("tsc terminated unexpectedly")
cleanup(code)
})
private parseCodeServerLine: OnLineCallback = (strippedLine, originalLine) => {
if (!strippedLine.length) return

if (plugin) {
plugin.on("exit", (code) => {
Watcher.log("plugin terminated unexpectedly")
cleanup(code)
})
console.log("[Compiler][Code Server]", originalLine)

if (strippedLine.includes("Watching for file changes")) {
console.log("[Compiler][Code Server]", "Finished compiling!", "(Refresh your web browser ♻️)")

this.reloadWebServer()
}
}

private parsePluginLine: OnLineCallback = (strippedLine, originalLine) => {
if (!strippedLine.length) return

vscodeWebExtensions.stderr.on("data", (d) => process.stderr.write(d))
vscode.stderr.on("data", (d) => process.stderr.write(d))
tsc.stderr.on("data", (d) => process.stderr.write(d))
console.log("[Compiler][Plugin]", originalLine)

if (plugin) {
plugin.stderr.on("data", (d) => process.stderr.write(d))
if (strippedLine.includes("Watching for file changes...")) {
this.reloadWebServer()
}
}

onLine(vscode, (line, original) => {
console.log("[vscode]", original)
if (line.includes("Finished compilation")) {
restartServer()
}
})
//#endregion

onLine(tsc, (line, original) => {
// tsc outputs blank lines; skip them.
if (line !== "") {
console.log("[tsc]", original)
}
if (line.includes("Watching for file changes")) {
restartServer()
}
})
//#region Utilities

if (plugin) {
onLine(plugin, (line, original) => {
// tsc outputs blank lines; skip them.
if (line !== "") {
console.log("[plugin]", original)
}
if (line.includes("Watching for file changes")) {
restartServer()
}
})
/**
* Cleans files from previous builds.
*/
private cleanFiles(): Promise<string[]> {
console.log("[Watcher]", "Cleaning files from previous builds...")

return del([
"out/**/*",
// Included because the cache can sometimes enter bad state when debugging compiled files.
".cache/**/*",
])
}

/**
* Emits a file containing compilation data.
* This is especially useful when Express needs to determine if VS Code is still compiling.
*/
private emitCompilationStats(): Promise<void> {
const stats: CompilationStats = {
status: this.vscodeCompileStatus,
lastCompiledAt: new Date(),
}

this.log("Writing watcher stats...")
return fs.writeFile(this.paths.compilationStatsFile, JSON.stringify(stats, null, 2))
}

private log(...entries: string[]) {
process.stdout.write(entries.join(" "))
}

private dispose(code: number | null): void {
for (const [processName, devProcess] of Object.entries(this.compilers)) {
this.log(`[${processName}]`, "Killing...\n")
devProcess?.removeAllListeners()
devProcess?.kill()
}
process.exit(typeof code === "number" ? code : 0)
}

private get hasVerboseLogging() {
return process.argv.includes("--log")
}

//#endregion
}

async function main(): Promise<void> {
try {
const watcher = new Watcher()
await watcher.initialize()
} catch (error: any) {
console.error(error.message)
process.exit(1)
}
}

Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
"@typescript-eslint/parser": "^5.0.0",
"audit-ci": "^5.0.0",
"codecov": "^3.8.3",
"del": "^6.0.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should this be a devDependency?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already is 😄
Screen Shot 2021-11-19 at 11 32 34

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOL does this mean my eyesight is going? Apologies!

"doctoc": "^2.0.0",
"eslint": "^7.7.0",
"eslint-config-prettier": "^8.1.0",
Expand Down
20 changes: 14 additions & 6 deletions src/node/routes/vscode.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { WebsocketRequest } from "../../../typings/pluginapi"
import { logError } from "../../common/util"
import { isDevMode } from "../constants"
import { ensureAuthenticated, authenticated, redirect } from "../http"
import { loadAMDModule } from "../util"
import { loadAMDModule, readCompilationStats } from "../util"
import { Router as WsRouter } from "../wsRouter"
import { errorHandler } from "./errors"

Expand Down Expand Up @@ -40,7 +40,6 @@ export class CodeServerRouteWrapper {
if (error instanceof Error && ["EntryNotFound", "FileNotFound", "HttpError"].includes(error.message)) {
next()
}

errorHandler(error, req, res, next)
}

Expand All @@ -62,9 +61,21 @@ export class CodeServerRouteWrapper {
*/
private ensureCodeServerLoaded: express.Handler = async (req, _res, next) => {
if (this._codeServerMain) {
// Already loaded...
return next()
}

if (isDevMode) {
// Is the development mode file watcher still busy?
const compileStats = await readCompilationStats()

if (!compileStats || !compileStats.lastCompiledAt) {
return next(new Error("VS Code may still be compiling..."))
}
}

// Create the server...

const { args } = req

/**
Expand All @@ -84,10 +95,7 @@ export class CodeServerRouteWrapper {
})
} catch (createServerError) {
logError(logger, "CodeServerRouteWrapper", createServerError)

const loggedError = isDevMode ? new Error("VS Code may still be compiling...") : createServerError

return next(loggedError)
return next(createServerError)
}

return next()
Expand Down
Loading