Skip to content

test flake: TestAgent_UpdatedDERP #9018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
spikecurtis opened this issue Aug 10, 2023 · 6 comments · Fixed by #9079
Closed

test flake: TestAgent_UpdatedDERP #9018

spikecurtis opened this issue Aug 10, 2023 · 6 comments · Fixed by #9079
Assignees
@spikecurtis

Comments

@spikecurtis
Copy link
Contributor

c.f. https://github.com/coder/coder/actions/runs/5817048961/job/15771103773?pr=8939

It looks like it might be an issue closing down the agent:

t.go:85: 2023-08-10 04:48:10.494 [debu]  agent: reporting lifecycle state  payload={"state":"off","changed_at":"2023-08-10T04:48:08.492411Z"}
    t.go:85: 2023-08-10 04:48:10.494 [debu]  agent.client: post lifecycle  req={"state":"off","changed_at":"2023-08-10T04:48:08.492411Z"}
    t.go:85: 2023-08-10 04:48:10.494 [info]  agent.ssh-server: stopped serving listener  listen_addr={} ...
        error= tailnet:
                   github.com/coder/coder/tailnet.(*listener).Accept
                       /Users/runner/work/coder/coder/tailnet/conn.go:898
                 - use of closed network connection
    t.go:85: 2023-08-10 04:48:10.495 [debu]  unable to read agent update, connection may be closed  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159 ...
        error= read json:
                   github.com/coder/coder/tailnet.(*coordinator).handleNextAgentMessage
                       /Users/runner/work/coder/coder/tailnet/coordinator.go:552
                 - EOF
    t.go:85: 2023-08-10 04:48:10.495 [debu]  deleted agent socket and node  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159
    t.go:85: 2023-08-10 04:48:10.495 [debu]  agent.net.tailnet.net.wgengine: magicsock: disco: node [Mlrfg] d:0866c35abd9250bc now using DERP only (reset)
    t.go:85: 2023-08-10 04:48:10.495 [debu]  agent.net.tailnet.net.wgengine: magicsock: disco: node [EnmtN] d:be40005ba46ecaf5 now using DERP only (reset)
    t.go:85: 2023-08-10 04:48:10.495 [debu]  agent.net.tailnet.net.wgengine: magicsock: closing connection to derp-2 (conn-close), age 2s
    t.go:85: 2023-08-10 04:48:10.495 [debu]  agent.net.tailnet.net.wgengine: magicsock: 0 active derp conns
    t.go:85: 2023-08-10 04:48:10.495 [debu]  done sending updates  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159
    t.go:85: 2023-08-10 04:48:10.495 [debu]  agent.net.tailnet.net.wgengine: [v1] warning: fakeRouter.Close: not implemented.
    t.go:85: 2023-08-10 04:48:10.496 [debu]  agent.net.tailnet.net.wgengine: wg: [v2] Routine: receive incoming mkReceiveFunc - stopped
    t.go:85: 2023-08-10 04:48:10.496 [debu]  agent.net.tailnet.net.wgengine: wg: [v2] Device closing
    t.go:85: 2023-08-10 04:48:10.496 [debu]  agent.net.tailnet.net.wgengine: wg: [v2] Routine: receive incoming receiveDERP - stopped
    t.go:85: 2023-08-10 04:48:10.496 [debu]  agent.net.tailnet.net.wgengine: wg: [v2] Routine: receive incoming mkReceiveFunc - stopped
    t.go:85: 2023-08-10 04:48:10.496 [debu]  agent.net.tailnet.net.wgengine: wg: [v2] Device closed
    t.go:85: 2023-08-10 04:48:10.496 [debu]  client.net.wgengine: wg: [v2] Received message with unknown type
    t.go:85: 2023-08-10 04:48:10.496 [debu]  client.net.wgengine: wgengine: idle peer [FqI/N] now active, reconfiguring WireGuard
    t.go:85: 2023-08-10 04:48:10.496 [debu]  unable to read client update, connection may be closed  client_id=dca19e91-0640-47f4-a094-4352b089da63  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159 ...
        error= read json:
                   github.com/coder/coder/tailnet.(*coordinator).handleNextClientMessage
                       /Users/runner/work/coder/coder/tailnet/coordinator.go:350
                 - io: read/write on closed pipe
    t.go:85: 2023-08-10 04:48:10.496 [debu]  deleted client node  client_id=dca19e91-0640-47f4-a094-4352b089da63  agent_id=00000000-0000-0000-0000-000000000000
    t.go:85: 2023-08-10 04:48:10.496 [debu]  client.net.wgengine: wgengine: Reconfig: configuring userspace WireGuard config (with 1/1 peers)
    t.go:85: 2023-08-10 04:48:10.496 [debu]  deleted client connectionSocket from map  client_id=dca19e91-0640-47f4-a094-4352b089da63  agent_id=00000000-0000-0000-0000-000000000000  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159
    t.go:85: 2023-08-10 04:48:10.496 [debu]  deleted client agents  client_id=dca19e91-0640-47f4-a094-4352b089da63  agent_id=00000000-0000-0000-0000-000000000000
    t.go:85: 2023-08-10 04:48:10.496 [debu]  done sending updates  client_id=b0236ad0-3498-4097-8fe0-9bf5965914ad  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159
    t.go:85: 2023-08-10 04:48:10.496 [debu]  client: wireguard status  status=<nil>  error="engine closing; no status"
    t.go:85: 2023-08-10 04:48:10.496 [debu]  unable to read client update, connection may be closed  client_id=b0236ad0-3498-4097-8fe0-9bf5965914ad  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159 ...
        error= read json:
                   github.com/coder/coder/tailnet.(*coordinator).handleNextClientMessage
                       /Users/runner/work/coder/coder/tailnet/coordinator.go:350
                 - io: read/write on closed pipe
    t.go:85: 2023-08-10 04:48:10.497 [debu]  deleted client node  client_id=b0236ad0-3498-4097-8fe0-9bf5965914ad  agent_id=00000000-0000-0000-0000-000000000000
    t.go:85: 2023-08-10 04:48:10.497 [debu]  deleted client connectionSocket from map  client_id=b0236ad0-3498-4097-8fe0-9bf5965914ad  agent_id=00000000-0000-0000-0000-000000000000  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159
    t.go:85: 2023-08-10 04:48:10.497 [debu]  deleted last client connectionSocket from map  client_id=b0236ad0-3498-4097-8fe0-9bf5965914ad  agent_id=00000000-0000-0000-0000-000000000000  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159
    t.go:85: 2023-08-10 04:48:10.497 [debu]  deleted client agents  client_id=b0236ad0-3498-4097-8fe0-9bf5965914ad  agent_id=00000000-0000-0000-0000-000000000000
    t.go:85: 2023-08-10 04:48:10.496 [debu]  done sending updates  client_id=dca19e91-0640-47f4-a094-4352b089da63  agent_id=1069f8c2-da4d-44ed-a791-8efc0a562159
    t.go:85: 2023-08-10 04:48:10.497 [debu]  client.net.wgengine: magicsock: disco: node [FqI/N] d:81df9814602fd43b now using DERP only (reset)
    t.go:85: 2023-08-10 04:48:10.497 [debu]  client.net.wgengine: magicsock: closing connection to derp-2 (conn-close), age 2s
    t.go:85: 2023-08-10 04:48:10.497 [debu]  client.net.wgengine: magicsock: 0 active derp conns
    t.go:85: 2023-08-10 04:48:10.497 [debu]  client.net.wgengine: [v1] warning: fakeRouter.Close: not implemented.
    t.go:85: 2023-08-10 04:48:10.497 [debu]  client.net.wgengine: wg: [v2] Device closing
    t.go:85: 2023-08-10 04:48:10.497 [debu]  client.net.wgengine: wg: [v2] Routine: receive incoming mkReceiveFunc - stopped
    t.go:85: 2023-08-10 04:48:10.497 [debu]  client.net.wgengine: wg: [v2] Routine: receive incoming mkReceiveFunc - stopped
    t.go:85: 2023-08-10 04:48:28.953 [debu]  client.net.wgengine: netcheck: netcheck.runProbe: got STUN response for t2 from 127.0.0.1:[644](https://github.com/coder/coder/actions/runs/5817048961/job/15771103773?pr=8939#step:5:645)14 (8cd36014440b4f9da1c2a5e1) in 310.658µs
    t.go:85: 2023-08-10 04:48:28.953 [debu]  client.net.wgengine: netcheck: netcheck.runProbe: got STUN response for t2 from 127.0.0.1:64414 (371434d60f1821e3d2dda7f9) in 443.353µs
    t.go:85: 2023-08-10 04:48:28.953 [debu]  client.net.wgengine: netcheck: netcheck.runProbe: got STUN response for t2 from 127.0.0.1:64414 (e2933482dfad8aa8e393e4ad) in 461.689µs
    t.go:85: 2023-08-10 04:48:29.004 [debu]  client.net.wgengine: netcheck: UDP is blocked, trying ICMP
    t.go:85: 2023-08-10 04:48:29.004 [debu]  client.net.wgengine: netcheck: [v1] measuring ICMP latency of test (2): listen ip4:icmp 0.0.0.0: socket: operation not permitted
    t.go:85: 2023-08-10 04:48:29.009 [debu]  client.net.wgengine: netcheck: [v1] netcheck: measuring HTTP(S) latency of test (2): unexpected status code: 426 (426 Upgrade Required)
    t.go:85: 2023-08-10 04:48:29.009 [debu]  client.net.wgengine: netcheck: [v1] report: udp=true v6=false v6os=true mapvarydest=false hair= portmap= v4a=127.0.0.1:64414 derp=0
    t.go:85: 2023-08-10 04:48:52.324 [debu]  client.net.wgengine: netcheck: netcheck.runProbe: got STUN response for t2 from 127.0.0.1:64414 (463dc37097c59ca7ccc3835d) in 434.664µs
    t.go:85: 2023-08-10 04:48:52.324 [debu]  client.net.wgengine: netcheck: netcheck.runProbe: got STUN response for t2 from 127.0.0.1:64414 (b6c87218e125f861b3d9a844) in 434.951µs
    t.go:85: 2023-08-10 04:48:52.324 [debu]  client.net.wgengine: netcheck: netcheck.runProbe: got STUN response for t2 from 127.0.0.1:64414 (ac66530e3ab80a1472a7bb32) in 451.426µs
    t.go:85: 2023-08-10 04:48:52.374 [debu]  client.net.wgengine: netcheck: UDP is blocked, trying ICMP
    t.go:85: 2023-08-10 04:48:52.374 [debu]  client.net.wgengine: netcheck: [v1] measuring ICMP latency of test (2): listen ip4:icmp 0.0.0.0: socket: operation not permitted
    t.go:85: 2023-08-10 04:48:52.379 [debu]  client.net.wgengine: netcheck: [v1] netcheck: measuring HTTP(S) latency of test (2): unexpected status code: 426 (426 Upgrade Required)
    t.go:85: 2023-08-10 04:48:52.379 [debu]  client.net.wgengine: netcheck: [v1] report: udp=true v6=false v6os=true mapvarydest=false hair= portmap= v4a=127.0.0.1:64414 derp=0
2023-08-10 04:48:55.501 [info]  stdlib: ...
    msg= 2023/08/10 04:48:55 wgengine watchdog stacks:
         goroutine profile: total 168
         36 @ 0x103b356 0x1006dfd 0x1006938 0x1a30859 0x1070241
         #	0x1a30858	github.com/tailscale/wireguard-go/device.(*Device).RoutineDecryption+0x198	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:245

         36 @ 0x103b356 0x1006dfd 0x1006938 0x1a30cf4 0x1070241
         #	0x1a30cf3	github.com/tailscale/wireguard-go/device.(*Device).RoutineHandshake+0x193	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:279

         36 @ 0x103b356 0x1006dfd 0x1006938 0x1a34c14 0x1070241
         #	0x1a34c13	github.com/tailscale/wireguard-go/device.(*Device).RoutineEncryption+0x1b3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/send.go:451

         12 @ 0x103b356 0x1909e25 0x1909e0d 0x1909f31 0x1927370 0x1927361 0x1070241
         #	0x1909e24	gvisor.dev/gvisor/pkg/sync.Gopark+0x84					/Users/runner/go/pkg/mod/github.com/coder/gvisor@v0.0.0-20230714132058-be2e4ac102c3/pkg/sync/runtime_unsafe.go:41
         #	0x1909e0c	gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker+0x6c			/Users/runner/go/pkg/mod/github.com/coder/gvisor@v0.0.0-20230714132058-be2e4ac102c3/pkg/sleep/sleep_unsafe.go:209
         #	0x1909f30	gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch+0x30			/Users/runner/go/pkg/mod/github.com/coder/gvisor@v0.0.0-20230714132058-be2e4ac102c3/pkg/sleep/sleep_unsafe.go:256
         #	0x192736f	gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch+0xaf			/Users/runner/go/pkg/mod/github.com/coder/gvisor@v0.0.0-20230714132058-be2e4ac102c3/pkg/sleep/sleep_unsafe.go:279
         #	0x1927360	gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start+0xa0	/Users/runner/go/pkg/mod/github.com/coder/gvisor@v0.0.0-20230714132058-be2e4ac102c3/pkg/tcpip/transport/tcp/dispatcher.go:287

         3 @ 0x103b356 0x1034257 0x1069c89 0x10aa932 0x10ac74f 0x10ac73d 0x1209329 0x1229b05 0x1227c31 0x1f594f3 0x1070241
         #	0x1069c88	internal/poll.runtime_pollWait+0x88		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306
         #	0x10aa931	internal/poll.(*pollDesc).wait+0x31		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84
         #	0x10ac74e	internal/poll.(*pollDesc).waitRead+0x24e	/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
         #	0x10ac73c	internal/poll.(*FD).ReadFromInet4+0x23c		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:250
         #	0x1209328	net.(*netFD).readFromInet4+0x28			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/fd_posix.go:66
         #	0x1229b04	net.(*UDPConn).readFromAddrPort+0x144		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/udpsock_posix.go:78
         #	0x1227c30	net.(*UDPConn).ReadFromUDPAddrPort+0x30		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/udpsock.go:175
         #	0x1f594f2	tailscale.com/net/stun/stuntest.runSTUN+0xd2	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/stun/stuntest/stuntest.go:60

         3 @ 0x103b356 0x1034257 0x1069c89 0x10aa932 0x10b023d 0x10b022b 0x120b295 0x1226cc5 0x1225ddd 0x134238d 0x13bd285 0x1a63f8a 0x1070241
         #	0x1069c88	internal/poll.runtime_pollWait+0x88		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306
         #	0x10aa931	internal/poll.(*pollDesc).wait+0x31		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84
         #	0x10b023c	internal/poll.(*pollDesc).waitRead+0x2bc	/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
         #	0x10b022a	internal/poll.(*FD).Accept+0x2aa		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:614
         #	0x120b294	net.(*netFD).accept+0x34			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/fd_unix.go:172
         #	0x1226cc4	net.(*TCPListener).accept+0x24			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/tcpsock_posix.go:148
         #	0x1225ddc	net.(*TCPListener).Accept+0x3c			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/tcpsock.go:297
         #	0x134238c	crypto/tls.(*listener).Accept+0x2c		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/tls.go:66
         #	0x13bd284	net/http.(*Server).Serve+0x384			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/http/server.go:3059
         #	0x1a63f89	net/http/httptest.(*Server).goServe.func1+0x69	/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/http/httptest/server.go:310

         3 @ 0x103b356 0x104b97e 0x1a20158 0x1070241
         #	0x1a20157	github.com/tailscale/wireguard-go/ratelimiter.(*Ratelimiter).Init.func1+0x97	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/ratelimiter/ratelimiter.go:68

         3 @ 0x103b356 0x104ca6f 0x104ca46 0x106b7e7 0x107cc8b 0x1a22e0b 0x1070241
         #	0x106b7e6	sync.runtime_Semacquire+0x26						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:62
         #	0x107cc8a	sync.(*WaitGroup).Wait+0x4a						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/waitgroup.go:116
         #	0x1a22e0a	github.com/tailscale/wireguard-go/device.newOutboundQueue.func1+0x2a	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/channels.go:32

         3 @ 0x103b356 0x104ca6f 0x104ca46 0x106b7e7 0x107cc8b 0x1a22f4b 0x1070241
         #	0x106b7e6	sync.runtime_Semacquire+0x26						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:62
         #	0x107cc8a	sync.(*WaitGroup).Wait+0x4a						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/waitgroup.go:116
         #	0x1a22f4a	github.com/tailscale/wireguard-go/device.newInboundQueue.func1+0x2a	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/channels.go:50

         3 @ 0x103b356 0x104ca6f 0x104ca46 0x106b7e7 0x107cc8b 0x1a2308b 0x1070241
         #	0x106b7e6	sync.runtime_Semacquire+0x26						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:62
         #	0x107cc8a	sync.(*WaitGroup).Wait+0x4a						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/waitgroup.go:116
         #	0x1a2308a	github.com/tailscale/wireguard-go/device.newHandshakeQueue.func1+0x2a	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/channels.go:68

         2 @ 0x103b356 0x104ca6f 0x104ca46 0x106b7e7 0x107cc8b 0x1a27425 0x1a28036 0x1a254d3 0x1a26f75 0x1baf025 0x1bb429b 0x1bb4049 0x1070241
         #	0x106b7e6	sync.runtime_Semacquire+0x26						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:62
         #	0x107cc8a	sync.(*WaitGroup).Wait+0x4a						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/waitgroup.go:116
         #	0x1a27424	github.com/tailscale/wireguard-go/device.closeBindLocked+0x84		/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:434
         #	0x1a28035	github.com/tailscale/wireguard-go/device.(*Device).BindClose+0x35	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:541
         #	0x1a254d2	github.com/tailscale/wireguard-go/device.(*Device).downLocked+0x32	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:196
         #	0x1a26f74	github.com/tailscale/wireguard-go/device.(*Device).Close+0x154		/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:382
         #	0x1baf024	tailscale.com/wgengine.(*userspaceEngine).Close+0x284			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:1078
         #	0x1bb429a	tailscale.com/wgengine.(*watchdogEngine).watchdog.func1+0x1a		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:117
         #	0x1bb4048	tailscale.com/wgengine.(*watchdogEngine).watchdogErr.func2+0x28		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:83

         1 @ 0x1030876 0x1069805 0x11d1435 0x11d124d 0x11cdfcb 0x1bb3af8 0x1bb423c 0x1bb57aa 0x1e3ca62 0x1f685e5 0x11318ff 0x1132442 0x1132ee9 0x1132e75 0x1070241
         #	0x1069804	runtime/pprof.runtime_goroutineProfileWithLabels+0x24			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/mprof.go:844
         #	0x11d1434	runtime/pprof.writeRuntimeProfile+0xb4					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/pprof/pprof.go:734
         #	0x11d124c	runtime/pprof.writeGoroutine+0x4c					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/pprof/pprof.go:694
         #	0x11cdfca	runtime/pprof.(*Profile).WriteTo+0x14a					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/pprof/pprof.go:329
         #	0x1bb3af7	tailscale.com/wgengine.(*watchdogEngine).watchdogErr+0x3d7		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:92
         #	0x1bb423b	tailscale.com/wgengine.(*watchdogEngine).watchdog+0x7b			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:116
         #	0x1bb57a9	tailscale.com/wgengine.(*watchdogEngine).Close+0x89			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:175
         #	0x1e3ca61	github.com/coder/coder/tailnet.(*Conn).Close+0x1e1			/Users/runner/work/coder/coder/tailnet/conn.go:676
         #	0x1f685e4	github.com/coder/coder/agent_test.TestAgent_UpdatedDERP.func3.1+0x64	/Users/runner/work/coder/coder/agent/agent_test.go:1771
         #	0x11318fe	testing.(*common).Cleanup.func1+0x11e					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1150
         #	0x1132441	testing.(*common).runCleanup+0xe1					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1328
         #	0x1132ee8	testing.tRunner.func2+0x28						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1570
         #	0x1132e74	testing.tRunner+0x134							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1582

         1 @ 0x103b356 0x1006dfd 0x10068f8 0x11333a5 0x1132e84 0x1070241
         #	0x11333a4	testing.tRunner.func1+0x4a4	/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1542
         #	0x1132e83	testing.tRunner+0x143		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1582

         1 @ 0x103b356 0x1006dfd 0x10068f8 0x11333a5 0x1132e84 0x1135da9 0x113471a 0x1f1dcab 0x1f76d25 0x1f76d0f 0x103af27 0x1070241
         #	0x11333a4	testing.tRunner.func1+0x4a4				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1542
         #	0x1132e83	testing.tRunner+0x143					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1582
         #	0x1135da8	testing.runTests+0x488					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:2034
         #	0x1134719	testing.(*M).Run+0x639					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1906
         #	0x1f1dcaa	go.uber.org/goleak.VerifyTestMain+0x6a			/Users/runner/go/pkg/mod/go.uber.org/goleak@v1.2.1/testmain.go:53
         #	0x1f76d24	github.com/coder/coder/agent_test.TestMain+0x1e4	/Users/runner/work/coder/coder/agent/agent_test.go:59
         #	0x1f76d0e	main.main+0x1ce						_testmain.go:127
         #	0x103af26	runtime.main+0x206					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/proc.go:250

         1 @ 0x103b356 0x1006dfd 0x10068f8 0x1a4845f 0x1a49a28 0x1070241
         #	0x1a4845e	tailscale.com/net/tstun.(*fakeTUN).Read+0x1e		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/tstun/fake.go:37
         #	0x1a49a27	tailscale.com/net/tstun.(*Wrapper).pollVector+0x247	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/tstun/wrap.go:416

         1 @ 0x103b356 0x1006dfd 0x1006938 0x1a31a39 0x1070241
         #	0x1a31a38	github.com/tailscale/wireguard-go/device.(*Peer).RoutineSequentialReceiver+0x198	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:443

         1 @ 0x103b356 0x1006dfd 0x1006938 0x1a35188 0x1070241
         #	0x1a35187	github.com/tailscale/wireguard-go/device.(*Peer).RoutineSequentialSender+0x187	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/send.go:492

         1 @ 0x103b356 0x1006dfd 0x1006938 0x1a36cc5 0x1070241
         #	0x1a36cc4	github.com/tailscale/wireguard-go/device.(*Device).RoutineTUNEventReader+0x84	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/tun.go:19

         1 @ 0x103b356 0x1006dfd 0x1006938 0x1a4b7ca 0x1a33937 0x1070241
         #	0x1a4b7c9	tailscale.com/net/tstun.(*Wrapper).Read+0xc9					/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/tstun/wrap.go:724
         #	0x1a33936	github.com/tailscale/wireguard-go/device.(*Device).RoutineReadFromTUN+0x2d6	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/send.go:247

         1 @ 0x103b356 0x1006dfd 0x1006938 0x1b4ff7c 0x1a2fa04 0x1070241
         #	0x1b4ff7b	tailscale.com/wgengine/magicsock.(*connBind).receiveDERP+0x11b			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:638
         #	0x1a2fa03	github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming+0x3c3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:111

         1 @ 0x103b356 0x1006dfd 0x1006938 0x1ba968b 0x1070241
         #	0x1ba968a	tailscale.com/wgengine.NewUserspaceEngine.func6+0x4a	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:394

         1 @ 0x103b356 0x1034257 0x1069c89 0x10aa932 0x10abd19 0x10abd07 0x1209109 0x121c525 0x130d77d 0x1115818 0x130d965 0x130ae56 0x1310d4f 0x1310d50 0x113a6bf 0x113ae8c 0x1afde1e 0x1b00887 0x1b18685 0x1b18672 0x1b4e769 0x1070241
         #	0x1069c88	internal/poll.runtime_pollWait+0x88				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306
         #	0x10aa931	internal/poll.(*pollDesc).wait+0x31				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84
         #	0x10abd18	internal/poll.(*pollDesc).waitRead+0x298			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
         #	0x10abd06	internal/poll.(*FD).Read+0x286					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:167
         #	0x1209108	net.(*netFD).Read+0x28						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/fd_posix.go:55
         #	0x121c524	net.(*conn).Read+0x44						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/net.go:183
         #	0x130d77c	crypto/tls.(*atLeastReader).Read+0x3c				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:788
         #	0x1115817	bytes.(*Buffer).ReadFrom+0x97					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/bytes/buffer.go:202
         #	0x130d964	crypto/tls.(*Conn).readFromUntil+0xe4				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:810
         #	0x130ae55	crypto/tls.(*Conn).readRecordOrCCS+0x115			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:617
         #	0x1310d4e	crypto/tls.(*Conn).readRecord+0x16e				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:583
         #	0x1310d4f	crypto/tls.(*Conn).Read+0x16f					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:1316
         #	0x113a6be	bufio.(*Reader).fill+0xfe					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/bufio/bufio.go:106
         #	0x113ae8b	bufio.(*Reader).ReadByte+0x2b					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/bufio/bufio.go:265
         #	0x1afde1d	tailscale.com/derp.readFrameHeader+0x1d				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp.go:172
         #	0x1b00886	tailscale.com/derp.(*Client).recvTimeout+0x1c6			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp_client.go:476
         #	0x1b18684	tailscale.com/derp.(*Client).Recv+0x84				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp_client.go:450
         #	0x1b18671	tailscale.com/derp/derphttp.(*Client).RecvDetail+0x71		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derphttp/derphttp_client.go:1128
         #	0x1b4e768	tailscale.com/wgengine/magicsock.(*Conn).runDerpReader+0x488	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:491

         1 @ 0x103b356 0x1034257 0x1069c89 0x10aa932 0x10abd19 0x10abd07 0x1209109 0x121c525 0x130d77d 0x1115818 0x130d965 0x130ae56 0x1310d4f 0x1310d50 0x13b23d1 0x113a6bf 0x113ae8c 0x1afde1e 0x1b05fae 0x1b05b8b 0x1b03185 0x1b1928c 0x13b970f 0x13bcd16 0x13b8232 0x1070241
         #	0x1069c88	internal/poll.runtime_pollWait+0x88		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306
         #	0x10aa931	internal/poll.(*pollDesc).wait+0x31		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84
         #	0x10abd18	internal/poll.(*pollDesc).waitRead+0x298	/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
         #	0x10abd06	internal/poll.(*FD).Read+0x286			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:167
         #	0x1209108	net.(*netFD).Read+0x28				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/fd_posix.go:55
         #	0x121c524	net.(*conn).Read+0x44				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/net.go:183
         #	0x130d77c	crypto/tls.(*atLeastReader).Read+0x3c		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:788
         #	0x1115817	bytes.(*Buffer).ReadFrom+0x97			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/bytes/buffer.go:202
         #	0x130d964	crypto/tls.(*Conn).readFromUntil+0xe4		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:810
         #	0x130ae55	crypto/tls.(*Conn).readRecordOrCCS+0x115	/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:617
         #	0x1310d4e	crypto/tls.(*Conn).readRecord+0x16e		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:583
         #	0x1310d4f	crypto/tls.(*Conn).Read+0x16f			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/crypto/tls/conn.go:1316
         #	0x13b23d0	net/http.(*connReader).Read+0x170		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/http/server.go:782
         #	0x113a6be	bufio.(*Reader).fill+0xfe			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/bufio/bufio.go:106
         #	0x113ae8b	bufio.(*Reader).ReadByte+0x2b			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/bufio/bufio.go:265
         #	0x1afde1d	tailscale.com/derp.readFrameHeader+0x1d		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp.go:172
         #	0x1b05fad	tailscale.com/derp.(*sclient).run+0x20d		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp_server.go:769
         #	0x1b05b8a	tailscale.com/derp.(*Server).accept+0xbaa	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp_server.go:734
         #	0x1b03184	tailscale.com/derp.(*Server).Accept+0x204	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp_server.go:439
         #	0x1b1928b	tailscale.com/derp/derphttp.Handler.func1+0x34b	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derphttp/derphttp_server.go:58
         #	0x13b970e	net/http.HandlerFunc.ServeHTTP+0x2e		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/http/server.go:2122
         #	0x13bcd15	net/http.serverHandler.ServeHTTP+0x315		/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/http/server.go:2936
         #	0x13b8231	net/http.(*conn).serve+0x611			/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/http/server.go:1995

         1 @ 0x103b356 0x1034257 0x1069c89 0x10aa932 0x10ac74f 0x10ac73d 0x1209329 0x1229b05 0x1227c31 0x1b6d96a 0x1b6de65 0x1b60455 0x1a2fa04 0x1070241
         #	0x1069c88	internal/poll.runtime_pollWait+0x88						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306
         #	0x10aa931	internal/poll.(*pollDesc).wait+0x31						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84
         #	0x10ac74e	internal/poll.(*pollDesc).waitRead+0x24e					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
         #	0x10ac73c	internal/poll.(*FD).ReadFromInet4+0x23c						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:250
         #	0x1209328	net.(*netFD).readFromInet4+0x28							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/fd_posix.go:66
         #	0x1229b04	net.(*UDPConn).readFromAddrPort+0x144						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/udpsock_posix.go:78
         #	0x1227c30	net.(*UDPConn).ReadFromUDPAddrPort+0x30						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/udpsock.go:175
         #	0x1b6d969	tailscale.com/wgengine/magicsock.(*RebindingUDPConn).readFromWithInitPconn+0x89	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/rebinding_conn.go:57
         #	0x1b6de64	tailscale.com/wgengine/magicsock.(*RebindingUDPConn).ReadBatch+0x124		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/rebinding_conn.go:105
         #	0x1b60454	tailscale.com/wgengine/magicsock.(*Conn).mkReceiveFunc.func1+0x1d4		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/magicsock.go:1163
         #	0x1a2fa03	github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming+0x3c3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:111

         1 @ 0x103b356 0x1034257 0x1069c89 0x10aa932 0x10acb0f 0x10acafd 0x1209449 0x1229a47 0x1227c31 0x1b6d96a 0x1b6de65 0x1b60455 0x1a2fa04 0x1070241
         #	0x1069c88	internal/poll.runtime_pollWait+0x88						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/netpoll.go:306
         #	0x10aa931	internal/poll.(*pollDesc).wait+0x31						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:84
         #	0x10acb0e	internal/poll.(*pollDesc).waitRead+0x24e					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_poll_runtime.go:89
         #	0x10acafc	internal/poll.(*FD).ReadFromInet6+0x23c						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/internal/poll/fd_unix.go:277
         #	0x1209448	net.(*netFD).readFromInet6+0x28							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/fd_posix.go:72
         #	0x1229a46	net.(*UDPConn).readFromAddrPort+0x86						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/udpsock_posix.go:85
         #	0x1227c30	net.(*UDPConn).ReadFromUDPAddrPort+0x30						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/udpsock.go:175
         #	0x1b6d969	tailscale.com/wgengine/magicsock.(*RebindingUDPConn).readFromWithInitPconn+0x89	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/rebinding_conn.go:57
         #	0x1b6de64	tailscale.com/wgengine/magicsock.(*RebindingUDPConn).ReadBatch+0x124		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/rebinding_conn.go:105
         #	0x1b60454	tailscale.com/wgengine/magicsock.(*Conn).mkReceiveFunc.func1+0x1d4		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/magicsock.go:1163
         #	0x1a2fa03	github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming+0x3c3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:111

         1 @ 0x103b356 0x104b97e 0x10a6f11 0x10a76c5 0x113d8f6 0x1a45a13 0x1a44f92 0x1a4535a 0x1bab51f 0x1baaa92 0x1b52db4 0x1b50514 0x1b50074 0x1a2fa04 0x1070241
         #	0x10a6f10	io.(*pipe).read+0xb0								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:57
         #	0x10a76c4	io.(*PipeReader).Read+0x24							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:136
         #	0x113d8f5	bufio.(*Scanner).Scan+0x875							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/bufio/scan.go:214
         #	0x1a45a12	tailscale.com/wgengine/wgcfg.FromUAPI+0xd2					/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/parser.go:76
         #	0x1a44f91	tailscale.com/wgengine/wgcfg.DeviceConfig+0x1d1					/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/device.go:31
         #	0x1a45359	tailscale.com/wgengine/wgcfg.ReconfigDevice+0xb9				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/device.go:52
         #	0x1bab51e	tailscale.com/wgengine.(*userspaceEngine).maybeReconfigWireguardLocked+0x85e	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:687
         #	0x1baaa91	tailscale.com/wgengine.(*userspaceEngine).noteRecvActivity+0x291		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:565
         #	0x1b52db3	tailscale.com/wgengine/magicsock.(*endpoint).noteRecvActivity+0x93		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/endpoint.go:229
         #	0x1b50513	tailscale.com/wgengine/magicsock.(*Conn).processDERPReadResult+0x273		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:683
         #	0x1b50073	tailscale.com/wgengine/magicsock.(*connBind).receiveDERP+0x213			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:642
         #	0x1a2fa03	github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming+0x3c3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:111

         1 @ 0x103b356 0x104b97e 0x10a730b 0x10a77e5 0x10fa55b 0x1a4804e 0x1a47eaa 0x1a47646 0x1a45545 0x1bab51f 0x1baaa92 0x1b52db4 0x1b50514 0x1b50074 0x1a2fa04 0x1070241
         #	0x10a730a	io.(*pipe).write+0x20a								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:86
         #	0x10a77e4	io.(*PipeWriter).Write+0x24							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:165
         #	0x10fa55a	fmt.Fprintf+0x9a								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/fmt/print.go:225
         #	0x1a4804d	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI.func1+0xcd			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:29
         #	0x1a47ea9	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI.func3+0xc9			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:38
         #	0x1a47[645](https://github.com/coder/coder/actions/runs/5817048961/job/15771103773?pr=8939#step:5:646)	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI+0x6c5				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:74
         #	0x1a45544	tailscale.com/wgengine/wgcfg.ReconfigDevice+0x2a4				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/device.go:64
         #	0x1bab51e	tailscale.com/wgengine.(*userspaceEngine).maybeReconfigWireguardLocked+0x85e	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:687
         #	0x1baaa91	tailscale.com/wgengine.(*userspaceEngine).noteRecvActivity+0x291		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:565
         #	0x1b52db3	tailscale.com/wgengine/magicsock.(*endpoint).noteRecvActivity+0x93		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/endpoint.go:229
         #	0x1b50513	tailscale.com/wgengine/magicsock.(*Conn).processDERPReadResult+0x273		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:683
         #	0x1b50073	tailscale.com/wgengine/magicsock.(*connBind).receiveDERP+0x213			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:642
         #	0x1a2fa03	github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming+0x3c3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:111

         1 @ 0x103b356 0x104b97e 0x1220a17 0x1220765 0x1186ee8 0x1186adb 0x11867f8 0x1e3f176 0x1070241
         #	0x1220a16	net.(*pipe).read+0x156						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/pipe.go:159
         #	0x1220764	net.(*pipe).Read+0x24						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/pipe.go:142
         #	0x1186ee7	encoding/json.(*Decoder).refill+0x187				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:165
         #	0x1186ada	encoding/json.(*Decoder).readValue+0xba				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:140
         #	0x11867f7	encoding/json.(*Decoder).Decode+0x77				/Users/runner/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:63
         #	0x1e3f175	github.com/coder/coder/tailnet.ServeCoordinator.func2+0xb5	/Users/runner/work/coder/coder/tailnet/coordinator.go:95

         1 @ 0x103b356 0x104b97e 0x1220a17 0x1220765 0x1186ee8 0x1186adb 0x11867f8 0x1e416c5 0x1e40533 0x1f6ddda 0x1070241
         #	0x1220a16	net.(*pipe).read+0x156								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/pipe.go:159
         #	0x1220764	net.(*pipe).Read+0x24								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/net/pipe.go:142
         #	0x1186ee7	encoding/json.(*Decoder).refill+0x187						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:165
         #	0x1186ada	encoding/json.(*Decoder).readValue+0xba						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:140
         #	0x11867f7	encoding/json.(*Decoder).Decode+0x77						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/encoding/json/stream.go:63
         #	0x1e416c4	github.com/coder/coder/tailnet.(*coordinator).handleNextClientMessage+0x304	/Users/runner/work/coder/coder/tailnet/coordinator.go:348
         #	0x1e40532	github.com/coder/coder/tailnet.(*coordinator).ServeClient+0x832			/Users/runner/work/coder/coder/tailnet/coordinator.go:291
         #	0x1f6ddd9	github.com/coder/coder/agent_test.setupAgent.func4+0xd9				/Users/runner/work/coder/coder/agent/agent_test.go:2154

         1 @ 0x103b356 0x104b97e 0x198f2f8 0x1070241
         #	0x198f2f7	tailscale.com/net/netmon.(*Monitor).debounce+0xd7	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/netmon/netmon.go:291

         1 @ 0x103b356 0x104b97e 0x1a494b8 0x1070241
         #	0x1a494b7	tailscale.com/net/tstun.(*Wrapper).pumpEvents+0x117	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/tstun/wrap.go:345

         1 @ 0x103b356 0x104b97e 0x1b0a5b1 0x1b0[662](https://github.com/coder/coder/actions/runs/5817048961/job/15771103773?pr=8939#step:5:663)5 0x14cb384 0x1070241
         #	0x1b0a5b0	tailscale.com/derp.(*sclient).sendLoop+0x[670](https://github.com/coder/coder/actions/runs/5817048961/job/15771103773?pr=8939#step:5:671)		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp_server.go:1480
         #	0x1b06624	tailscale.com/derp.(*sclient).run.func1+0x24		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/derp/derp_server.go:755
         #	0x14cb383	golang.org/x/sync/errgroup.(*Group).Go.func1+0x63	/Users/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75

         1 @ 0x103b356 0x104b97e 0x1b4fbae 0x1070241
         #	0x1b4fbad	tailscale.com/wgengine/magicsock.(*Conn).runDerpWriter+0x1ad	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:619

         1 @ 0x103b356 0x104b97e 0x1bb3a28 0x1bb423c 0x1bb57aa 0x1e3ca62 0x1e4edc5 0x1f6dc7d 0x11318ff 0x1132442 0x1132ee9 0x1132e75 0x1070241
         #	0x1bb3a27	tailscale.com/wgengine.(*watchdogEngine).watchdogErr+0x307		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:86
         #	0x1bb423b	tailscale.com/wgengine.(*watchdogEngine).watchdog+0x7b			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:116
         #	0x1bb57a9	tailscale.com/wgengine.(*watchdogEngine).Close+0x89			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:175
         #	0x1e3ca61	github.com/coder/coder/tailnet.(*Conn).Close+0x1e1			/Users/runner/work/coder/coder/tailnet/conn.go:[676](https://github.com/coder/coder/actions/runs/5817048961/job/15771103773?pr=8939#step:5:677)
         #	0x1e4edc4	github.com/coder/coder/codersdk.(*WorkspaceAgentConn).Close+0xc4	/Users/runner/work/coder/coder/codersdk/workspaceagentconn.go:193
         #	0x1f6dc7c	github.com/coder/coder/agent_test.setupAgent.func6+0x1c			/Users/runner/work/coder/coder/agent/agent_test.go:2164
         #	0x11318fe	testing.(*common).Cleanup.func1+0x11e					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1150
         #	0x1132441	testing.(*common).runCleanup+0xe1					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1328
         #	0x1132ee8	testing.tRunner.func2+0x28						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1570
         #	0x1132e74	testing.tRunner+0x134							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/testing/testing.go:1582

         1 @ 0x103b356 0x104b97e 0x1bb9ba9 0x1bb9b49 0x1c835fc 0x1070241
         #	0x1bb9ba8	gvisor.dev/gvisor/pkg/tcpip/link/channel.(*queue).ReadContext+0x88	/Users/runner/go/pkg/mod/github.com/coder/gvisor@v0.0.0-20230714132058-be2e4ac102c3/pkg/tcpip/link/channel/channel.go:71
         #	0x1bb9b48	gvisor.dev/gvisor/pkg/tcpip/link/channel.(*Endpoint).ReadContext+0x28	/Users/runner/go/pkg/mod/github.com/coder/gvisor@v0.0.0-20230714132058-be2e4ac102c3/pkg/tcpip/link/channel/channel.go:177
         #	0x1c835fb	tailscale.com/wgengine/netstack.(*Impl).inject+0x3b			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/netstack/netstack.go:475

         1 @ 0x103b356 0x104b97e 0x1e48a9d 0x1070241
         #	0x1e48a9c	github.com/coder/coder/tailnet.(*TrackedConn).SendUpdates+0xbc	/Users/runner/work/coder/coder/tailnet/trackedconn.go:99

         1 @ 0x103b356 0x104ca6f 0x104ca46 0x106b906 0x107b185 0x107c7d6 0x107c7b5 0x1a3854b 0x1a45716 0x1070241
         #	0x106b905	sync.runtime_SemacquireMutex+0x25					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:77
         #	0x107b184	sync.(*Mutex).lockSlow+0x164						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/mutex.go:171
         #	0x107c7d5	sync.(*Mutex).Lock+0x35							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/mutex.go:90
         #	0x107c7b4	sync.(*RWMutex).Lock+0x14						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/rwmutex.go:147
         #	0x1a3854a	github.com/tailscale/wireguard-go/device.(*Device).IpcSetOperation+0x6a	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/uapi.go:143
         #	0x1a45715	tailscale.com/wgengine/wgcfg.ReconfigDevice.func2+0x35			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/device.go:60

         1 @ 0x103b356 0x104ca6f 0x104ca46 0x106b966 0x1a37278 0x1a37253 0x1a45236 0x1070241
         #	0x106b965	sync.runtime_SemacquireRWMutexR+0x25					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:82
         #	0x1a37277	sync.(*RWMutex).RLock+0x77						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/rwmutex.go:71
         #	0x1a37252	github.com/tailscale/wireguard-go/device.(*Device).IpcGetOperation+0x52	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/uapi.go:52
         #	0x1a45235	tailscale.com/wgengine/wgcfg.DeviceConfig.func1+0x35			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/device.go:28

         1 @ 0x106c9bf 0x1494709 0x198fec5 0x198fea0 0x198ecca 0x1070241
         #	0x106c9be	syscall.syscall+0x7e					/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sys_darwin.go:23
         #	0x1494708	golang.org/x/sys/unix.read+0x48				/Users/runner/go/pkg/mod/golang.org/x/sys@v0.11.0/unix/zsyscall_darwin_amd64.go:1813
         #	0x198fec4	golang.org/x/sys/unix.Read+0x44				/Users/runner/go/pkg/mod/golang.org/x/sys@v0.11.0/unix/syscall_unix.go:159
         #	0x198fe9f	tailscale.com/net/netmon.(*darwinRouteMon).Receive+0x1f	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/netmon/netmon_darwin.go:55
         #	0x198ecc9	tailscale.com/net/netmon.(*Monitor).pump+0x89		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/net/netmon/netmon.go:250
2023-08-10 04:48:55.501 [info]  stdlib: ...
    msg= 2023/08/10 04:48:55 wgengine watchdog in-flight:
         in-flight[12]: name=Close duration=45.004s start=2023-08-10T04:48:10.49[683](https://github.com/coder/coder/actions/runs/5817048961/job/15771103773?pr=8939#step:5:684)8Z
2023-08-10 04:48:55.501 [info]  stdlib: 2023/08/10 04:48:55 wgengine: watchdog timeout on Close
@spikecurtis spikecurtis self-assigned this Aug 10, 2023
@spikecurtis
Copy link
Contributor Author

Ok, what's happening is a deadlock in tailscale while shutting down.

tailscale calls Close() on the wireguard device, which holds (among other locks) the ipcMutex while shutting down. It then attempts to close its Bind -- which is the interface it uses to receive packets from the network. This device is the tailscale magic socket. It waits for all receive functions to close.

However, the DERP receive function has side effects beyond just reading from DERP, including sometimes triggering a wireguard update.

         1 @ 0x103b356 0x104b97e 0x10a730b 0x10a77e5 0x10fa55b 0x1a4804e 0x1a47eaa 0x1a47646 0x1a45545 0x1bab51f 0x1baaa92 0x1b52db4 0x1b50514 0x1b50074 0x1a2fa04 0x1070241
         #	0x10a730a	io.(*pipe).write+0x20a								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:86
         #	0x10a77e4	io.(*PipeWriter).Write+0x24							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:165
         #	0x10fa55a	fmt.Fprintf+0x9a								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/fmt/print.go:225
         #	0x1a4804d	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI.func1+0xcd			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:29
         #	0x1a47ea9	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI.func3+0xc9			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:38
         #	0x1a47645	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI+0x6c5				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:74
         #	0x1a45544	tailscale.com/wgengine/wgcfg.ReconfigDevice+0x2a4				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/device.go:64
         #	0x1bab51e	tailscale.com/wgengine.(*userspaceEngine).maybeReconfigWireguardLocked+0x85e	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:687
         #	0x1baaa91	tailscale.com/wgengine.(*userspaceEngine).noteRecvActivity+0x291		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:565
         #	0x1b52db3	tailscale.com/wgengine/magicsock.(*endpoint).noteRecvActivity+0x93		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/endpoint.go:229
         #	0x1b50513	tailscale.com/wgengine/magicsock.(*Conn).processDERPReadResult+0x273		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:683
         #	0x1b50073	tailscale.com/wgengine/magicsock.(*connBind).receiveDERP+0x213			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:642
         #	0x1a2fa03	github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming+0x3c3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:111

The wireguard update blocks on (you guessed it!) acquiring the ipcMutex.

@spikecurtis
Copy link
Contributor Author

Looks like this was introduced recently on the upstream wireguard-go and pulled into our fork: coder/wireguard-go@b7cd547

However, it has not yet made it to tailscale's fork, so opening a bug against tailscale seems like poor form.

@spikecurtis
Copy link
Contributor Author

Since the upstream commit claims that allowing IPC updates while shutting down is a problem, I think we shouldn't revert the commit. (It would also just be tech debt preventing us from taking upstream updates.)

I don't see any reason why the noteRecvActivity() call has to happen synchronously with receiving the packet and delivering it to wireguard. It's just a little kick to the wgengine that stuff has been happening and maybe we need a reconfiguration. We could move it to a dedicated goroutine and kick it off via a channel.

@coadler
Copy link
Contributor

coadler commented Aug 10, 2023

Here's the original issue I opened against Tailscale a few months ago: tailscale/tailscale#8059

The commit I pulled in was originally meant to address this issue, but I don't think it fully solves it. My last attempt at upgrading Tailscale around then was fully stopped by this issue as CI was failing ~90% of the time with this issue, but I was only able to get the deadlock a handful of times this time.

This was referenced Aug 11, 2023
Open
Closed
Closed
@raggi
Copy link

raggi commented Aug 12, 2023

I see multiple goroutines inside maybeReconfigWireguardLocked, it appears this must be being called without holding wgLock.
In the stack these are both being called from noteRecvActivity which takes the lock at the top: https://github.com/tailscale/tailscale/blob/7a9c6c71e16cd399aba511a8b5b46951a4fa04be/wgengine/userspace.go#L531

I can't find an instance where this lock isn't taken, nor can I find something spuriously unlocking this mutex.

Is the attached goroutine stack coming from a process running tests with t.Parallel() ?

@spikecurtis spikecurtis mentioned this issue Aug 14, 2023
Merged
@spikecurtis
Copy link
Contributor Author

@raggi

There are multiple instances of the userspace engine running (it's an integration test, so multiple speakers are present), so presence of multiple calls doesn't indicate that it isn't taking the lock.

Closing the userspace engine closes the wgdev without holding the wgLock:

e.g.

2 @ 0x103b356 0x104ca6f 0x104ca46 0x106b7e7 0x107cc8b 0x1a27425 0x1a28036 0x1a254d3 0x1a26f75 0x1baf025 0x1bb429b 0x1bb4049 0x1070241
         #	0x106b7e6	sync.runtime_Semacquire+0x26						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/runtime/sema.go:62
         #	0x107cc8a	sync.(*WaitGroup).Wait+0x4a						/Users/runner/hostedtoolcache/go/1.20.7/x64/src/sync/waitgroup.go:116
         #	0x1a27424	github.com/tailscale/wireguard-go/device.closeBindLocked+0x84		/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:434
         #	0x1a28035	github.com/tailscale/wireguard-go/device.(*Device).BindClose+0x35	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:541
         #	0x1a254d2	github.com/tailscale/wireguard-go/device.(*Device).downLocked+0x32	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:196
         #	0x1a26f74	github.com/tailscale/wireguard-go/device.(*Device).Close+0x154		/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/device.go:382
         #	0x1baf024	tailscale.com/wgengine.(*userspaceEngine).Close+0x284			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:1078
         #	0x1bb429a	tailscale.com/wgengine.(*watchdogEngine).watchdog.func1+0x1a		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:117
         #	0x1bb4048	tailscale.com/wgengine.(*watchdogEngine).watchdogErr.func2+0x28		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/watchdog.go:83

But, it's definitely not as simple as just taking the lock while closing wgdev, since it's the DERP packet receive function that is calling maybeReconfigWireguardLocked, and that function is what device.closeBindLocked is waiting for.

         1 @ 0x103b356 0x104b97e 0x10a730b 0x10a77e5 0x10fa55b 0x1a4804e 0x1a47eaa 0x1a47646 0x1a45545 0x1bab51f 0x1baaa92 0x1b52db4 0x1b50514 0x1b50074 0x1a2fa04 0x1070241
         #	0x10a730a	io.(*pipe).write+0x20a								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:86
         #	0x10a77e4	io.(*PipeWriter).Write+0x24							/Users/runner/hostedtoolcache/go/1.20.7/x64/src/io/pipe.go:165
         #	0x10fa55a	fmt.Fprintf+0x9a								/Users/runner/hostedtoolcache/go/1.20.7/x64/src/fmt/print.go:225
         #	0x1a4804d	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI.func1+0xcd			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:29
         #	0x1a47ea9	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI.func3+0xc9			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:38
         #	0x1a47645	tailscale.com/wgengine/wgcfg.(*Config).ToUAPI+0x6c5				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/writer.go:74
         #	0x1a45544	tailscale.com/wgengine/wgcfg.ReconfigDevice+0x2a4				/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/wgcfg/device.go:64
         #	0x1bab51e	tailscale.com/wgengine.(*userspaceEngine).maybeReconfigWireguardLocked+0x85e	/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:687
         #	0x1baaa91	tailscale.com/wgengine.(*userspaceEngine).noteRecvActivity+0x291		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/userspace.go:565
         #	0x1b52db3	tailscale.com/wgengine/magicsock.(*endpoint).noteRecvActivity+0x93		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/endpoint.go:229
         #	0x1b50513	tailscale.com/wgengine/magicsock.(*Conn).processDERPReadResult+0x273		/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:683
         #	0x1b50073	tailscale.com/wgengine/magicsock.(*connBind).receiveDERP+0x213			/Users/runner/go/pkg/mod/github.com/coder/tailscale@v1.1.1-0.20230809183309-7a9c6c71e16c/wgengine/magicsock/derp.go:642
         #	0x1a2fa03	github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming+0x3c3	/Users/runner/go/pkg/mod/github.com/coder/wireguard-go@v0.0.0-20230807234434-d825b45ccbf5/device/receive.go:111

so, also holding wglock in Close() will just move the deadlock higher in the stack at tailscale.com/wgengine.(*userspaceEngine).noteRecvActivity.

I'm currently testing a fix that has tailscale.com/wgengine.(*userspaceEngine).noteRecvActivity checking for e.closed before processing receive activity. But, it's tricky to comply with the locking order... I'll give you what I have once I'm satisfied it's not deadlocking in our tests.

This was referenced Aug 14, 2023
Open
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spikecurtis spikecurtis

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@raggi @spikecurtis @coadler