Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit c109fc7

Browse files
ethanndicksonethanndickson
committedSep 4, 2024
docs: add networking troubleshooting page
1 parent 093d243 commit c109fc7

File tree

5 files changed

+122
-37
lines changed

5 files changed

+122
-37
lines changed
 

‎cli/cliui/agent.go

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,7 @@ func PeerDiagnostics(w io.Writer, d tailnet.PeerDiagnostics) {
309309
_, _ = fmt.Fprint(w, "✘ not connected to DERP\n")
310310
}
311311
if d.SentNode {
312-
_, _ = fmt.Fprint(w, "✔ sent local data to Coder networking coodinator\n")
312+
_, _ = fmt.Fprint(w, "✔ sent local data to Coder networking coordinator\n")
313313
} else {
314314
_, _ = fmt.Fprint(w, "✘ have not sent local data to Coder networking coordinator\n")
315315
}
@@ -381,6 +381,9 @@ func (d ConnDiags) Write(w io.Writer) {
381381
_, _ = fmt.Fprintf(w, " - %s\n\n", msg)
382382
}
383383
}
384+
if !d.PingP2P || d.Verbose || len(client) > 0 || len(agent) > 0 || len(general) > 1 {
385+
_, _ = fmt.Fprintln(w, "For more information, see https://coder.com/docs/networking/troubleshooting")
386+
}
384387
}
385388

386389
func (d ConnDiags) splitDiagnostics() (general, client, agent []string) {

‎cli/cliui/agent_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -533,7 +533,7 @@ func TestPeerDiagnostics(t *testing.T) {
533533
LastWireguardHandshake: time.Time{},
534534
},
535535
want: []*regexp.Regexp{
536-
regexp.MustCompile(`^✔ sent local data to Coder networking coodinator$`),
536+
regexp.MustCompile(`^✔ sent local data to Coder networking coordinator$`),
537537
},
538538
},
539539
{

‎docs/manifest.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -351,6 +351,11 @@
351351
"title": "STUN and NAT",
352352
"description": "Learn how Coder establishes direct connections",
353353
"path": "./networking/stun.md"
354+
},
355+
{
356+
"title": "Troubleshooting",
357+
"description": "Troubleshoot networking issues in Coder",
358+
"path": "./networking/troubleshooting.md"
354359
}
355360
]
356361
},

‎docs/networking/index.md

Lines changed: 1 addition & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -169,41 +169,7 @@ with security policies. In these cases, pass the `--browser-only` flag to
169169
With browser-only connections, developers can only connect to their workspaces
170170
via the web terminal and [web IDEs](../ides/web-ides.md).
171171

172-
## Troubleshooting
173-
174-
The `coder ping -v <workspace>` will ping a workspace and return debug logs for
175-
the connection. We recommend running this command and inspecting the output when
176-
debugging SSH connections to a workspace. For example:
177-
178-
```console
179-
$ coder ping -v my-workspace
180-
181-
2023-06-21 17:50:22.412 [debu] wgengine: ping(fd7a:115c:a1e0:49d6:b259:b7ac:b1b2:48f4): sending disco ping to [cFYPo] ...
182-
pong from my-workspace proxied via DERP(Denver) in 90ms
183-
2023-06-21 17:50:22.503 [debu] wgengine: magicsock: closing connection to derp-13 (conn-close), age 5s
184-
2023-06-21 17:50:22.503 [debu] wgengine: magicsock: 0 active derp conns
185-
2023-06-21 17:50:22.504 [debu] wgengine: wg: [v2] Routine: receive incoming v6 - stopped
186-
2023-06-21 17:50:22.504 [debu] wgengine: wg: [v2] Device closed
187-
```
188-
189-
The `coder speedtest <workspace>` command measures user <-> workspace
190-
throughput. E.g.:
191-
192-
```
193-
$ coder speedtest dev
194-
29ms via coder
195-
Starting a 5s download test...
196-
INTERVAL TRANSFER BANDWIDTH
197-
0.00-1.00 sec 630.7840 MBits 630.7404 Mbits/sec
198-
1.00-2.00 sec 913.9200 MBits 913.8106 Mbits/sec
199-
2.00-3.00 sec 943.1040 MBits 943.0399 Mbits/sec
200-
3.00-4.00 sec 933.3760 MBits 933.2143 Mbits/sec
201-
4.00-5.00 sec 848.8960 MBits 848.7019 Mbits/sec
202-
5.00-5.02 sec 13.5680 MBits 828.8189 Mbits/sec
203-
----------------------------------------------------
204-
0.00-5.02 sec 4283.6480 MBits 853.8217 Mbits/sec
205-
```
206-
207172
## Up next
208173

209174
- Learn about [Port Forwarding](./port-forwarding.md)
175+
- Troubleshoot [Networking Issues](./troubleshooting.md)

‎docs/networking/troubleshooting.md

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Troubleshooting
2+
3+
`coder ping <workspace>` will ping the workspace agent and print diagnostics on
4+
the state of the connection. These diagnostics are created by inspecting both
5+
the client and agent network configurations, and provide insights into why a
6+
direct connection may be impeded, or why the quality of one might be degraded.
7+
8+
The `-v/--verbose` flag can be appended to the command to print client debug
9+
logs.
10+
11+
```console
12+
$ coder ping dev
13+
pong from workspace proxied via DERP(Council Bluffs, Iowa) in 42ms
14+
pong from workspace proxied via DERP(Council Bluffs, Iowa) in 41ms
15+
pong from workspace proxied via DERP(Council Bluffs, Iowa) in 39ms
16+
✔ preferred DERP region: 999 (Council Bluffs, Iowa)
17+
✔ sent local data to Coder networking coordinator
18+
✔ received remote agent data from Coder networking coordinator
19+
preferred DERP region: 999 (Council Bluffs, Iowa)
20+
endpoints: x.x.x.x:46433, x.x.x.x:46433, x.x.x.x:46433
21+
✔ Wireguard handshake 11s ago
22+
23+
❗ You are connected via a DERP relay, not directly (p2p)
24+
Possible client-side issues with direct connection:
25+
- Network interface utun0 has MTU 1280, (less than 1378), which may degrade the quality of direct connections
26+
27+
Possible agent-side issues with direct connection:
28+
- Agent is potentially behind a hard NAT, as multiple endpoints were retrieved from different STUN servers
29+
- Agent IP address is within an AWS range (AWS uses hard NAT)
30+
```
31+
32+
## Common Problems
33+
34+
### Direct Connections Disabled Deployment-Wide
35+
36+
Direct connections can be disabled at the deployment level by setting the
37+
`CODER_BLOCK_DIRECT` environment variable or the `--block-direct-connections`
38+
flag on the server. When set, this will be reflected in the output of
39+
`coder ping`.
40+
41+
### Firewall Blocks UDP
42+
43+
Some corporate firewalls block UDP traffic. Direct connections require UDP
44+
traffic to be allowed between the client and agent, as well as between the
45+
client/agent and STUN servers in most cases. `coder ping` will indicate if
46+
either the Coder agent or client had issues sending or receiving UDP packets to
47+
STUN servers.
48+
49+
If this is the case, you may need to add exceptions to the firewall to allow UDP
50+
for Coder workspaces, clients, and STUN servers.
51+
52+
### Endpoint-Dependent NAT (Hard NAT)
53+
54+
Hard NATs prevent public endpoints gathered from STUN servers from being used by
55+
the peer to establish a direct connection. Typically, if only one side of the
56+
connection is behind a hard NAT, direct connections can still be established
57+
easily. However, if both sides are behind hard NATs, direct connections may take
58+
longer to establish or may not be possible at all.
59+
60+
Hard NATs may also be simulated by firewalls that block ingress traffic from
61+
unknown 5-tuples (Protocol, Source IP, Source Port, Destination IP, Destination
62+
Port).
63+
64+
`coder ping` will indicate if it's possible the client or agent is behind a hard
65+
NAT or if a firewall is exhibiting this behavior.
66+
67+
Learn more about [STUN and NAT](./stun.md).
68+
69+
### VPNs
70+
71+
If a VPN is the default route for all IP traffic, it may interfere with the
72+
ability for clients and agents to form direct connections. This happens if the
73+
NAT does not permit traffic to be
74+
['hairpinned'](./stun.md#3-direct-connections-with-vpn-and-nat-hairpinning) from
75+
the public IP address of the NAT (determined via STUN) to the internal IP
76+
address of the agent.
77+
78+
If this is the case, you may need to add exceptions to the VPN for Coder, modify
79+
the NAT configuration, or deploy an internal STUN server.
80+
81+
### Low MTU
82+
83+
If a network interface on the side of either the client or agent has an MTU
84+
smaller than 1378, any direct connections form may have degraded quality or
85+
performance, as IP packets are fragmented. `coder ping` will indicate if this is
86+
the case by inspecting network interfaces on both the client and the workspace
87+
agent.
88+
89+
If another interface cannot be used, and the MTU cannot be changed, you may need
90+
to disable direct connections, and relay all traffic via DERP instead, which
91+
will not be affected by the low MTU.
92+
93+
## Throughput
94+
95+
The `coder speedtest <workspace>` command measures the throughput between the
96+
client and the workspace agent.
97+
98+
```console
99+
$ coder speedtest workspace
100+
29ms via coder
101+
Starting a 5s download test...
102+
INTERVAL TRANSFER BANDWIDTH
103+
0.00-1.00 sec 630.7840 MBits 630.7404 Mbits/sec
104+
1.00-2.00 sec 913.9200 MBits 913.8106 Mbits/sec
105+
2.00-3.00 sec 943.1040 MBits 943.0399 Mbits/sec
106+
3.00-4.00 sec 933.3760 MBits 933.2143 Mbits/sec
107+
4.00-5.00 sec 848.8960 MBits 848.7019 Mbits/sec
108+
5.00-5.02 sec 13.5680 MBits 828.8189 Mbits/sec
109+
----------------------------------------------------
110+
0.00-5.02 sec 4283.6480 MBits 853.8217 Mbits/sec
111+
```

0 commit comments

Comments
 (0)
Failed to load comments.