Skip to content

coder ssh nil pointer dereference under heavy load / agent disconnected #9884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mafredri opened this issue Sep 27, 2023 · 0 comments · Fixed by #9892
Closed

coder ssh nil pointer dereference under heavy load / agent disconnected #9884

mafredri opened this issue Sep 27, 2023 · 0 comments · Fixed by #9892
Assignees
@mafredri
Labels
cli Area: CLI

Comments

@mafredri
Copy link
Member

During scaletesting where the coder server was under heavy load from multiple agents and API responses very slow, we observed the following nil pointer dereference:

❯ coder ssh scaletest/runner
==> ⧗ The workspace agent lost connection
2023-09-27 10:49:47.956Z Wait for it to reconnect or restart your workspace.
2023-09-27 10:49:47.956Z For more information and troubleshooting, see https://coder.com/docs/v2/latest/templates#agent-connection-issues and https://coder.com/docs/coder-oss/latest/templates#troubleshooting-templates
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12ef680]

goroutine 1 [running]:
github.com/coder/coder/v2/cli/cliui.Agent({0x1c60878?, 0xc00070d1d0?}, {0x1c51540?, 0xc000130010}, {0x62, 0xf6, 0x9c, 0x9a, 0xd8, 0xac, ...}, ...)
	/home/runner/actions-runner/_work/coder/coder/cli/cliui/agent.go:233 +0xb00
github.com/coder/coder/v2/cli.(*RootCmd).ssh.func1(0xc0003b1e00)
	/home/runner/actions-runner/_work/coder/coder/cli/ssh.go:186 +0x113e
github.com/coder/coder/v2/cli.(*RootCmd).Command.func6.1(0xc0001c36c0?)
	/home/runner/actions-runner/_work/coder/coder/cli/root.go:277 +0xd1
github.com/coder/coder/v2/cli.(*RootCmd).initClientInternal.func1.1(0xc0003b1e00)
	/home/runner/actions-runner/_work/coder/coder/cli/root.go:596 +0x882
github.com/coder/coder/v2/cli/clibase.RequireRangeArgs.func1.1(0x0?)
	/home/runner/actions-runner/_work/coder/coder/cli/clibase/cmd.go:574 +0x362
github.com/coder/coder/v2/cli/clibase.(*Invocation).run(0xc0003b1e00, 0xc0002ade00)
	/home/runner/actions-runner/_work/coder/coder/cli/clibase/cmd.go:388 +0x10b2
github.com/coder/coder/v2/cli/clibase.(*Invocation).run(0xc0003b1b00, 0xc0002ade00)
	/home/runner/actions-runner/_work/coder/coder/cli/clibase/cmd.go:325 +0xb5e
github.com/coder/coder/v2/cli/clibase.(*Invocation).Run(0xc0002adeb8?)
	/home/runner/actions-runner/_work/coder/coder/cli/clibase/cmd.go:478 +0xe5
github.com/coder/coder/v2/cli.(*RootCmd).RunMain(0xc000547e30?, {0xc00053ae00, 0x25, 0x40})
	/home/runner/actions-runner/_work/coder/coder/cli/root.go:134 +0x1a5
main.main()
	/home/runner/actions-runner/_work/coder/coder/enterprise/cmd/coder/main.go:11 +0x45

The coder client version used was the latest (at the time of writing): v2.1.5+66019ef

The nil pointer deref happens at:

coder/cli/cliui/agent.go

Line 245 in c67db6e

sw.Complete(stage, agent.LastConnectedAt.Sub(*agent.DisconnectedAt))

This suggests last connected or disconnected may be nil.
@mafredri mafredri added cli Area: CLI bug labels Sep 27, 2023
@mafredri mafredri changed the title coder ssh nil pointer dereference under heavy load or agent disconnected coder ssh nil pointer dereference under heavy load / agent disconnected Sep 27, 2023
@mafredri mafredri self-assigned this Sep 27, 2023
@mafredri mafredri mentioned this issue Sep 27, 2023
Merged
mafredri added a commit that referenced this issue Sep 27, 2023
@mafredri
fix(cli/cliui): avoid nil pointer deref when agent disconnects
bd59fba 
Fixes #9884
mafredri added a commit that referenced this issue Sep 27, 2023
@mafredri
fix(cli/cliui): avoid nil pointer deref when agent disconnects (#9892)
0878381 
Fixes #9884
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mafredri mafredri

Labels
cli Area: CLI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(cli/cliui): avoid nil pointer deref when agent disconnects coder/coder
1 participant
@mafredri