Upgrading coder `v0.19.2` -> `v0.20.0` results in code-server becoming inaccessible (blank page on load) #6746

edmondsiu0 · 2023-03-23T08:11:18Z

Hi Coder team, first of all, thanks for Coder!

I have noticed an issue with the latest version of Coder v0.20.0, after upgrade the following features results in (near) blank page stuck forever loading:

code-server
Terminal

No network activities were captured by Chrome while it sits there waiting to load (screenshot below shows code-server).

With that being said, some activities were seen on the container with code-server running, indicating "invalid initiation message".

2023-03-23 08:06:24.838 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake initiation
2023-03-23 08:06:25.034 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Received handshake initiation
2023-03-23 08:06:25.034 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - ConsumeMessageInitiation: handshake replay @ 2023-03-23 08:06:25.016777216 +0000 UTC
2023-03-23 08:06:25.034 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake response
2023-03-23 08:06:25.034 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] Received invalid initiation message from ee673a36eec77a8bb35c94328b0e331b35ed66eb3d300493831ea31d8bf91d05
2023-03-23 08:06:30.051 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Handshake did not complete after 5 seconds, retrying (try 2)
2023-03-23 08:06:30.051 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake initiation
2023-03-23 08:06:30.303 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Received handshake initiation
2023-03-23 08:06:30.303 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake response
2023-03-23 08:06:30.304 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Received handshake initiation
2023-03-23 08:06:30.304 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake response
2023-03-23 08:06:35.166 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Handshake did not complete after 5 seconds, retrying (try 3)
2023-03-23 08:06:35.572 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Received handshake initiation
2023-03-23 08:06:35.572 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake response
2023-03-23 08:06:35.572 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - ConsumeMessageInitiation: handshake replay @ 2023-03-23 08:06:35.536870912 +0000 UTC
2023-03-23 08:06:35.572 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] Received invalid initiation message from ee673a36eec77a8bb35c94328b0e331b35ed66eb3d300493831ea31d8bf91d05
2023-03-23 08:06:36.728 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/userspace.go:1254>	(*userspaceEngine).Ping	ping(fd7a:115c:a1e0:4ae8:a38a:2d92:c807:8a7c): sending disco ping to [7mc6N]  ...
2023-03-23 08:06:36.894 [DEBUG]	<./codersdk/agentsdk/agentsdk.go:196>	(*Client).Listen.func1	got coordinate pong	{"took": "41.230589ms"}
2023-03-23 08:06:40.792 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Received handshake initiation
2023-03-23 08:06:40.792 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Received handshake initiation
2023-03-23 08:06:40.792 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake response
2023-03-23 08:06:40.792 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [v2] [7mc6N] - Sending handshake response
2023-03-23 08:06:40.792 [DEBUG]	(tailnet.wgengine)	<./../../../tailscale.com/wgengine/wglog/wglog.go:81>	NewLogger.func1	wg: [7mc6N] - Failed to create response message: handshake initiation must be consumed first

I was able to replicate this issue on both my existing coder installation, as well as a fresh upgrade from a clean v0.19.2.

The text was updated successfully, but these errors were encountered:

matifali · 2023-03-23T09:49:36Z

Can you reproduce this on a fresh, clean install of 0.20.0?

edmondsiu0 · 2023-03-23T09:53:30Z

Can you reproduce this on a fresh, clean install of 0.20.0?

Was just about to post an update on this - I can reproduce the same issue with a fresh clean v0.20.0 (while fresh clean v0.19.2 works).

renesas-brandon-hussey · 2023-03-23T11:19:16Z

I have this same issue. I spent most of yesterday trying to narrow this down and find and issue but could not find one. The issue does not always happen. Some new Workspaces will work after initially being created. If I then go back and use them some time later, I get the blank screen. Even when I receive a blank screen I can still SSH into the Workspace and use it with no issues. The only issue I've seen is in using the coder_app's through the web browser.

I did check in the logs and I have the same Received invalid initiation message message as shown in the original post.

matifali · 2023-03-23T11:22:51Z

@bpmct Is this is a regression?

bpmct · 2023-03-23T11:35:38Z

Thanks for reporting, and sorry this happened! We'll start investigating.

It'd be helpful to see both the Coder agent logs (/tmp/coder-agent.log inside the container/VM/pod) and the coder server logs. You can email them to me if you'd prefer (email in my GitHub profile).

edmondsiu0 · 2023-03-23T12:24:46Z

@bpmct Logs sent via email! :)

renesas-brandon-hussey · 2023-03-23T13:29:51Z

@bpmct I can provide logs too if needed.

bpmct · 2023-03-23T13:47:28Z

It wouldn't hurt :)

kylecarbs · 2023-03-23T14:02:21Z

Humph, @edmondsiu0 @renesas-brandon-hussey I'm not sure why it's not working, but I know what the problematic change is...

#6595

I'll revert this for now and we can do a patch, but I'm not sure why it'd seemingly be dropping some of the traffic. Are you using a proxy in front of your Coder deployment?

kylecarbs · 2023-03-23T14:04:22Z

@edmondsiu0 @renesas-brandon-hussey could y'all try running Coder with verbose logs and send me the output? That's where the failing connection is coming from.

Not sure why this is breaking deployments, but it seems likely to be the cause. See #6746.

This was causing boatloads of connects to reestablish every time... See #6746

edmondsiu0 · 2023-03-23T14:14:38Z

@kylecarbs I am fronting coder with Traefik (worked until v0.19.2), I tried looking at Traefik earlier for obvious connection errors but nothing stood out to me.

happy to supply verbose logs, but I can't seem to find info on turning on verbose logging.
Is there an environment variable, or CMD override that I can reference?

kylecarbs · 2023-03-23T14:15:15Z

@edmondsiu0 actually, I think I found the issue! It only seems to effect some environments...

Could you add me from our Coder Discord so we can debug further? https://discord.gg/coder

This was causing boatloads of connects to reestablish every time... See #6746

renesas-brandon-hussey · 2023-03-23T14:23:23Z

I am also using Traefik. Same thing as @edmondsiu0 I looked for obvious issues but did not see any. Coder is running in docker and my Templates are docker-in-docker based.

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs · 2023-03-23T14:59:05Z

@renesas-brandon-hussey @edmondsiu0 v0.20.1 is coming out, and it should fix the problem for y'all!

kylecarbs · 2023-03-23T15:09:46Z

It's out now!

edmondsiu0 · 2023-03-23T15:21:08Z

🥳 v0.20.1 fixed it for me!
Thanks for fixing!

kylecarbs · 2023-03-23T15:21:57Z

Sick! Happy to hear it :)

renesas-brandon-hussey · 2023-03-23T17:06:03Z

I have updated to v0.20.1 and in my initial testing the issue is fixed. Thanks for the quick turnaround!

kylecarbs · 2023-03-23T19:11:16Z

Of course! Apologies that it happened in the first place, but now our code is improved, so win win.

bpmct added bug s1 Bugs that break core workflows. Only humans may set this. labels Mar 23, 2023

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: revert "direct embedded derp traffic directly to the server"

4974777

Not sure why this is breaking deployments, but it seems likely to be the cause. See #6746.

kylecarbs mentioned this issue Mar 23, 2023

fix: revert "direct embedded derp traffic directly to the server" #6749

Closed

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections

8db208a

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs mentioned this issue Mar 23, 2023

fix: use a background context when piping derp connections #6750

Merged

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections

105c88f

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections

139f905

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections

e7a63dd

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections

7798533

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections

680677e

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections

ead3e05

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs added a commit that referenced this issue Mar 23, 2023

fix: use a background context when piping derp connections (#6750)

ed9a3b9

This was causing boatloads of connects to reestablish every time... See #6746

kylecarbs closed this as completed Mar 23, 2023

bpmct added s0 Major regression, all-hands-on-deck to fix and removed s1 Bugs that break core workflows. Only humans may set this. labels Apr 17, 2023

mtojek mentioned this issue Apr 18, 2023

[Retrospective] Use E2E test for release acceptance #7154

Closed

bpmct assigned kylecarbs Apr 26, 2023

matifali added the bug label Jun 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading coder `v0.19.2` -> `v0.20.0` results in code-server becoming inaccessible (blank page on load) #6746

Upgrading coder `v0.19.2` -> `v0.20.0` results in code-server becoming inaccessible (blank page on load) #6746

edmondsiu0 commented Mar 23, 2023

matifali commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023

renesas-brandon-hussey commented Mar 23, 2023 •

edited

Loading

matifali commented Mar 23, 2023

bpmct commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023

renesas-brandon-hussey commented Mar 23, 2023

bpmct commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023 •

edited

Loading

kylecarbs commented Mar 23, 2023 •

edited

Loading

renesas-brandon-hussey commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

renesas-brandon-hussey commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

Upgrading coder v0.19.2 -> v0.20.0 results in code-server becoming inaccessible (blank page on load) #6746

Upgrading coder v0.19.2 -> v0.20.0 results in code-server becoming inaccessible (blank page on load) #6746

Comments

edmondsiu0 commented Mar 23, 2023

matifali commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023

renesas-brandon-hussey commented Mar 23, 2023 • edited Loading

matifali commented Mar 23, 2023

bpmct commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023

renesas-brandon-hussey commented Mar 23, 2023

bpmct commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023 • edited Loading

kylecarbs commented Mar 23, 2023 • edited Loading

renesas-brandon-hussey commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

edmondsiu0 commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

renesas-brandon-hussey commented Mar 23, 2023

kylecarbs commented Mar 23, 2023

Upgrading coder `v0.19.2` -> `v0.20.0` results in code-server becoming inaccessible (blank page on load) #6746

Upgrading coder `v0.19.2` -> `v0.20.0` results in code-server becoming inaccessible (blank page on load) #6746

renesas-brandon-hussey commented Mar 23, 2023 •

edited

Loading

edmondsiu0 commented Mar 23, 2023 •

edited

Loading

kylecarbs commented Mar 23, 2023 •

edited

Loading