Filter out devices in tailnet that have an MTU that is too small #13327
Comments
|
gVisor is nominally permitted to send packets up to 1280. So, I think it really needs to be 1378 to be totally safe |
|
Unfortunately, we can't filter out the devices at the We could potentially intercept and squash Disco packets coming from these interfaces, or artificially inflate Disco packets to the maximum size we expect from Wireguard (1310), such that only paths that handle packets that big can get discovered for p2p. |
|
Disco doesn't have any length parameter built in, so if we padded the messages we'd have to change the protocol. That's a nightmare from back-compatibility standpoint. Or, figure out how to predict the length of the ciphertext from the plaintext, and pad before we encrypt with NaCl. |
|
Intercepting and squashing Disco packets from particular interfaces is quite difficult because of the way UDP sockets work. For incoming UDP packets we can get their source IP (where they are from), but don't normally get the destination IP or interface. There are some low-level system calls you can make to get this information, but they're not portable, so we'd be handling multiple OS implementations. The next idea I had was to figure out which interface we'd be sending replies via by looking at the routing table. But again, there is no portable implementation. We'd have to do multiple OS implementations. |
|
Looks like we can pad out Disco pings and pongs prior to encrypting with NaCl. (CallMeMaybe is only ever sent over DERP.) This has some performance implications, but Disco packets don't get sent that often, so it's probably fine. |
|
Testing on Linux, I've discovered that with the inflated ping/pong packets, the Don't Fragment (DF) flag only gets set if the packet is smaller than the interface MTU. If the interface MTU is smaller than the packet, then it fragments the packet.
So, possible next steps: There is a DONT_FRAGMENT socket option we could try setting. Not sure how portable it is, but in theory it might convince Linux to drop too-large packets instead of fragmenting them. Setting that on the tailscale UDP socket is basically us saying that we should always prefer DERP to a fragmented direct path, which may or may not be faster. This whole thing is honestly turning into a boondoggle, IMO. We thought it would be easy, and prevent some customer heartburn by automatically preferring DERP over paths with "bad" MTU, but it's turning out to be very difficult and involved. Customers can already work around the issue by setting DISABLE_DIRECT, so maybe we should cut our losses: drop an MTU check into |
re: #13327 Adds local interfaces to `coder netcheck` and checks their MTUs for potential problems. This is mostly relevant for end-user systems where VPNs are common. We _could_ also add it to coderd healthcheck, but until I see coderd connecting to workspaces over a VPN in the wild, I don't think its worth the UX effort. Netcheck results get the following: ``` "interfaces": { "error": null, "severity": "ok", "warnings": null, "dismissed": false, "interfaces": [ { "name": "lo0", "mtu": 16384, "addresses": [ "127.0.0.1/8", "::1/128", "fe80::1/64" ] }, { "name": "en8", "mtu": 1500, "addresses": [ "192.168.50.217/24", "fe80::c13:1a92:3fa5:dd7e/64" ] } ] } ``` _Technically_ not back compatible if anyone is parsing `coder netcheck` output as JSON, since the original output is now under `"derp"` in the output.
On devices with too small of an MTU, direct connections may be established (i.e. disco packets fit) but will drop every packet containing app data which renders the connection unusable.
This can happen on Cloudflare WARP VPN.
We should block these devices from being used for direct connections:
Background
Relates to https://github.com/coder/customers/issues/592
Inside the Wireguard tunnel to the workspace, we use IPv6 as the network layer protocol. IPv6 sets the minimum MTU for devices that carry it to 1280 octets. Thus, we set the virtual Wireguard TUN to an MTU of 1280 octets, and gVisor's tcpip stack will keep the inner IPv6 packets smaller than 1280 octets. However, these packets then need to get encapsulated in Wireguard, and in the case of direct connections, further encapsulated in UDP & IP and sent over the outer network interface. With this overhead, the typical TCP packets are 1338 octets. Thus, if the outer network interface has an MTU smaller than this, the packets are dropped.
It is not possible to reconfigure the inner protocols to use a smaller MTU because 1280 is the minimum for IPv6 according to the IPv6 specs.
gVisor is actually a bit conservative in its choice of segment size, leaving some extra room for TCP & IP options, so we may need greater than 1338 minimum to ensure packets aren't dropped.
Additional Requirements:
netcheckcommand to highlight devices which are being skippedThe text was updated successfully, but these errors were encountered: