From aff688e31e7ba08e1a7eb89455c884bdc0c28e31 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Tue, 6 Feb 2024 12:43:10 +0000 Subject: [PATCH 01/18] chore(docs): add requirements re ports and stun server to docs --- docs/networking/index.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/networking/index.md b/docs/networking/index.md index f5d94b10b70e6..c808572f94077 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -13,6 +13,37 @@ user <-> workspace connections are end-to-end encrypted. [Tailscale's open source](https://tailscale.com) backs our networking logic. +## Requirements + +In order for clients and workspaces to be able to connect: + +- All clients and agents must be able to establish a connection to the Coder + server (`CODER_ACCESS_URL`) over HTTPS (`tcp/443`). +- Any reverse proxy or ingress between the Coder control plane and clients must + support WebSockets. + +In order for clients to be able to establish direct connections: + +> **Note:** Direct connections via the web browser are not supported. To improve +> latency for browser-based applications running inside Coder workspaces, +> consider deploying one or more +> [workspace proxies](../admin/workspace-proxies.md). + +- The client is connecting using the CLI (e.g. `coder ssh` or + `coder port-forward`). +- The client and workspace agent are both able to connect to a specific STUN + server. + > The STUN server needs to tell the client and workspace their respective + > `address:port` pairs from its perspective so that they can establish a + > direct connection with each other. If the client and agent are only able to + > connect to STUN servers on different networks, then a direct connection will + > not be possible. For an in-depth technical explanation, see + > [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). +- Outbound UDP traffic must be allowed for both the client and the agent from + source ports `udp/3478` and `udp/41641` to all destination ports. + > For more detailed information, see + > [What firewall ports should I open to use Tailscale? (tailscale.com)](https://tailscale.com/kb/1082/firewall-ports). + ## coder server Workspaces connect to the coder server via the server's external address, set From efb639a2a7bd37f1a7f1e84e50f3f3dffcc56fb8 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Tue, 6 Feb 2024 15:41:09 +0000 Subject: [PATCH 02/18] Update docs/networking/index.md Co-authored-by: Dean Sheather --- docs/networking/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index c808572f94077..55113b131a823 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -18,7 +18,7 @@ user <-> workspace connections are end-to-end encrypted. In order for clients and workspaces to be able to connect: - All clients and agents must be able to establish a connection to the Coder - server (`CODER_ACCESS_URL`) over HTTPS (`tcp/443`). + server (`CODER_ACCESS_URL`) over HTTP/HTTPS. - Any reverse proxy or ingress between the Coder control plane and clients must support WebSockets. From 4e95495aeeb437c84cccbd244325f182fb61efa8 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Tue, 6 Feb 2024 16:57:42 +0000 Subject: [PATCH 03/18] add section on stun and nats --- docs/networking/index.md | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index 55113b131a823..87fdf94129ee8 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -31,18 +31,33 @@ In order for clients to be able to establish direct connections: - The client is connecting using the CLI (e.g. `coder ssh` or `coder port-forward`). -- The client and workspace agent are both able to connect to a specific STUN - server. - > The STUN server needs to tell the client and workspace their respective - > `address:port` pairs from its perspective so that they can establish a - > direct connection with each other. If the client and agent are only able to - > connect to STUN servers on different networks, then a direct connection will - > not be possible. For an in-depth technical explanation, see - > [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). -- Outbound UDP traffic must be allowed for both the client and the agent from - source ports `udp/3478` and `udp/41641` to all destination ports. - > For more detailed information, see - > [What firewall ports should I open to use Tailscale? (tailscale.com)](https://tailscale.com/kb/1082/firewall-ports). +- The client and workspace agent are both able to connect to [a specific STUN server](#stun-in-a-natshell). +- All outbound UDP traffic must be allowed for both the client and the agent + on **all ports** to each others' respective networks. + - To establish a direct connection, both agent and client use STUN. This involves sending UDP packets outbound on `udp/3478` to the configured [STUN server](../cli/server.md#--derp-server-stun-addresses). If either the agent or the client are unable to send and receive UDP packets to a STUN server, then direct connections will not be possible. + - Both agents and clients will then establish a [WireGuard](https://www.wireguard.com/)️ tunnel and send UDP traffic on ephemeral (high) ports. If a firewall between the client and the agent blocks this UDP traffic, direct connections will not be possible. + +### STUN in a NATshell + +> [Session Traversal Utilities for NAT (STUN)](https://www.rfc-editor.org/rfc/rfc8489.html) is a protocol used to assist applications in establishing peer-to-peer communications across Network Address Translations (NATs) or firewalls. +> +> [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) is commonly used in private networks to allow mulitple devices to share a single public IP address. The vast majority of ISPs today use at least one level of NAT. + +Normally, both Coder agent and client will be running in different _private_ networks (e.g. `192.168.1.0/24`). In order for them to communicate with each other, they will each need their counterpart's public IP address and port. + +Inside of that network, packets from the agent or client will show up as having source address `192.168.1.X:12345`. Howver, outside of this private network, the source address will show up differently (for example, `12.3.4.56:54321`). In order for the Coder client and agent to establish a direct connection with each other, one of them needs to know the `ip:port` pair under which their counterpart can be reached. + +In order to accomplish this, both client and agent can use a STUN server: + +> The below glosses over a lot of the complexity of traversing NATs. +> For a more in-depth technical explanation, see [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). + +- **Discovery:** Both the client and agent will send UDP traffic to a configured STUN server. This STUN server is generally located on the public internet, and responds with the public IP address and port from which the request came. +- **Port Mapping:** The client and agent then exchange this information through the Coder server. They will then construct packets that should be able to successfully traverse their counterpart's NATs successfully. +- **NAT Traversal:** The client and agent then send these crafted packets to their counterpart's public addresses. If all goes well, the NATs on the other end should route these packets to the correct internal address. +- **Connection:** Once the packets reach the other side, they send a response back to the source `ip:port` from the packet. Again, the NATs should recognize these responses as belonging to an ongoing communication, and forward them accordingly. + +At this point, both the client and agent should be able to send traffic directly to each other. ## coder server From c18787541192cc892d36bc20cecad16729ec1ca6 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Tue, 6 Feb 2024 17:05:39 +0000 Subject: [PATCH 04/18] fmt --- docs/networking/index.md | 71 +++++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 20 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index 87fdf94129ee8..562b8a8645f13 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -31,33 +31,64 @@ In order for clients to be able to establish direct connections: - The client is connecting using the CLI (e.g. `coder ssh` or `coder port-forward`). -- The client and workspace agent are both able to connect to [a specific STUN server](#stun-in-a-natshell). -- All outbound UDP traffic must be allowed for both the client and the agent - on **all ports** to each others' respective networks. - - To establish a direct connection, both agent and client use STUN. This involves sending UDP packets outbound on `udp/3478` to the configured [STUN server](../cli/server.md#--derp-server-stun-addresses). If either the agent or the client are unable to send and receive UDP packets to a STUN server, then direct connections will not be possible. - - Both agents and clients will then establish a [WireGuard](https://www.wireguard.com/)️ tunnel and send UDP traffic on ephemeral (high) ports. If a firewall between the client and the agent blocks this UDP traffic, direct connections will not be possible. +- The client and workspace agent are both able to connect to + [a specific STUN server](#stun-in-a-natshell). +- All outbound UDP traffic must be allowed for both the client and the agent on + **all ports** to each others' respective networks. + - To establish a direct connection, both agent and client use STUN. This + involves sending UDP packets outbound on `udp/3478` to the configured + [STUN server](../cli/server.md#--derp-server-stun-addresses). If either the + agent or the client are unable to send and receive UDP packets to a STUN + server, then direct connections will not be possible. + - Both agents and clients will then establish a + [WireGuard](https://www.wireguard.com/)️ tunnel and send UDP traffic on + ephemeral (high) ports. If a firewall between the client and the agent + blocks this UDP traffic, direct connections will not be possible. ### STUN in a NATshell -> [Session Traversal Utilities for NAT (STUN)](https://www.rfc-editor.org/rfc/rfc8489.html) is a protocol used to assist applications in establishing peer-to-peer communications across Network Address Translations (NATs) or firewalls. +> [Session Traversal Utilities for NAT (STUN)](https://www.rfc-editor.org/rfc/rfc8489.html) +> is a protocol used to assist applications in establishing peer-to-peer +> communications across Network Address Translations (NATs) or firewalls. > -> [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) is commonly used in private networks to allow mulitple devices to share a single public IP address. The vast majority of ISPs today use at least one level of NAT. - -Normally, both Coder agent and client will be running in different _private_ networks (e.g. `192.168.1.0/24`). In order for them to communicate with each other, they will each need their counterpart's public IP address and port. - -Inside of that network, packets from the agent or client will show up as having source address `192.168.1.X:12345`. Howver, outside of this private network, the source address will show up differently (for example, `12.3.4.56:54321`). In order for the Coder client and agent to establish a direct connection with each other, one of them needs to know the `ip:port` pair under which their counterpart can be reached. +> [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) +> is commonly used in private networks to allow mulitple devices to share a +> single public IP address. The vast majority of ISPs today use at least one +> level of NAT. + +Normally, both Coder agent and client will be running in different _private_ +networks (e.g. `192.168.1.0/24`). In order for them to communicate with each +other, they will each need their counterpart's public IP address and port. + +Inside of that network, packets from the agent or client will show up as having +source address `192.168.1.X:12345`. Howver, outside of this private network, the +source address will show up differently (for example, `12.3.4.56:54321`). In +order for the Coder client and agent to establish a direct connection with each +other, one of them needs to know the `ip:port` pair under which their +counterpart can be reached. In order to accomplish this, both client and agent can use a STUN server: -> The below glosses over a lot of the complexity of traversing NATs. -> For a more in-depth technical explanation, see [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). - -- **Discovery:** Both the client and agent will send UDP traffic to a configured STUN server. This STUN server is generally located on the public internet, and responds with the public IP address and port from which the request came. -- **Port Mapping:** The client and agent then exchange this information through the Coder server. They will then construct packets that should be able to successfully traverse their counterpart's NATs successfully. -- **NAT Traversal:** The client and agent then send these crafted packets to their counterpart's public addresses. If all goes well, the NATs on the other end should route these packets to the correct internal address. -- **Connection:** Once the packets reach the other side, they send a response back to the source `ip:port` from the packet. Again, the NATs should recognize these responses as belonging to an ongoing communication, and forward them accordingly. - -At this point, both the client and agent should be able to send traffic directly to each other. +> The below glosses over a lot of the complexity of traversing NATs. For a more +> in-depth technical explanation, see +> [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). + +- **Discovery:** Both the client and agent will send UDP traffic to a configured + STUN server. This STUN server is generally located on the public internet, and + responds with the public IP address and port from which the request came. +- **Port Mapping:** The client and agent then exchange this information through + the Coder server. They will then construct packets that should be able to + successfully traverse their counterpart's NATs successfully. +- **NAT Traversal:** The client and agent then send these crafted packets to + their counterpart's public addresses. If all goes well, the NATs on the other + end should route these packets to the correct internal address. +- **Connection:** Once the packets reach the other side, they send a response + back to the source `ip:port` from the packet. Again, the NATs should recognize + these responses as belonging to an ongoing communication, and forward them + accordingly. + +At this point, both the client and agent should be able to send traffic directly +to each other. ## coder server From fb8a887ccdf2b1882f06381c8316593faae94d0a Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Tue, 6 Feb 2024 17:25:36 +0000 Subject: [PATCH 05/18] fix typos --- docs/networking/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index 562b8a8645f13..a7c4890d9c1f2 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -52,7 +52,7 @@ In order for clients to be able to establish direct connections: > communications across Network Address Translations (NATs) or firewalls. > > [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) -> is commonly used in private networks to allow mulitple devices to share a +> is commonly used in private networks to allow multiple devices to share a > single public IP address. The vast majority of ISPs today use at least one > level of NAT. @@ -61,8 +61,8 @@ networks (e.g. `192.168.1.0/24`). In order for them to communicate with each other, they will each need their counterpart's public IP address and port. Inside of that network, packets from the agent or client will show up as having -source address `192.168.1.X:12345`. Howver, outside of this private network, the -source address will show up differently (for example, `12.3.4.56:54321`). In +source address `192.168.1.X:12345`. However, outside of this private network, +the source address will show up differently (for example, `12.3.4.56:54321`). In order for the Coder client and agent to establish a direct connection with each other, one of them needs to know the `ip:port` pair under which their counterpart can be reached. From 9abe9eb6b09e29b74c45688796a8b95ae981f66b Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Wed, 7 Feb 2024 13:47:25 +0000 Subject: [PATCH 06/18] Apply suggestions from code review Co-authored-by: Spike Curtis --- docs/networking/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index a7c4890d9c1f2..778480c8bee05 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -19,13 +19,13 @@ In order for clients and workspaces to be able to connect: - All clients and agents must be able to establish a connection to the Coder server (`CODER_ACCESS_URL`) over HTTP/HTTPS. -- Any reverse proxy or ingress between the Coder control plane and clients must +- Any reverse proxy or ingress between the Coder control plane and clients/agents must support WebSockets. In order for clients to be able to establish direct connections: > **Note:** Direct connections via the web browser are not supported. To improve -> latency for browser-based applications running inside Coder workspaces, +> latency for browser-based applications running inside Coder workspaces in regions far from the Coder control plane, > consider deploying one or more > [workspace proxies](../admin/workspace-proxies.md). From 05dcbad4d382ee8ebda6c9800a618a8e346770aa Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Wed, 7 Feb 2024 13:59:24 +0000 Subject: [PATCH 07/18] add note re: plugins/ssh --- docs/networking/index.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index 778480c8bee05..90dbc5aff07e3 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -19,18 +19,22 @@ In order for clients and workspaces to be able to connect: - All clients and agents must be able to establish a connection to the Coder server (`CODER_ACCESS_URL`) over HTTP/HTTPS. -- Any reverse proxy or ingress between the Coder control plane and clients/agents must - support WebSockets. +- Any reverse proxy or ingress between the Coder control plane and + clients/agents must support WebSockets. In order for clients to be able to establish direct connections: > **Note:** Direct connections via the web browser are not supported. To improve -> latency for browser-based applications running inside Coder workspaces in regions far from the Coder control plane, -> consider deploying one or more +> latency for browser-based applications running inside Coder workspaces in +> regions far from the Coder control plane, consider deploying one or more > [workspace proxies](../admin/workspace-proxies.md). - The client is connecting using the CLI (e.g. `coder ssh` or - `coder port-forward`). + `coder port-forward`). Note that the + [VSCode extension](https://marketplace.visualstudio.com/items?itemName=coder.coder-remote) + and [JetBrains Plugin](https://plugins.jetbrains.com/plugin/19620-coder/), and + [`ssh coder.`](../cli/config-ssh.md) all utilize the CLI to + establish a workspace connection. - The client and workspace agent are both able to connect to [a specific STUN server](#stun-in-a-natshell). - All outbound UDP traffic must be allowed for both the client and the agent on From 90405b48d53040e1e06302ce63c3e9182a5ba9ec Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Wed, 7 Feb 2024 14:40:10 +0000 Subject: [PATCH 08/18] clarify requirements regarding direct connections --- docs/networking/index.md | 47 ++++++++++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index 90dbc5aff07e3..07bc690794834 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -35,8 +35,15 @@ In order for clients to be able to establish direct connections: and [JetBrains Plugin](https://plugins.jetbrains.com/plugin/19620-coder/), and [`ssh coder.`](../cli/config-ssh.md) all utilize the CLI to establish a workspace connection. -- The client and workspace agent are both able to connect to - [a specific STUN server](#stun-in-a-natshell). +- Either the client or workspace agent are able to discover a reachable + `ip:port` of their counterpart. If the agent and client are able to + communicate with each other using their locally assigned IP addresses, then a + direct connection can be established immediately. Otherwise, the client and + agent will contact + [the configured STUN servers](../cli/server.md#derp-server-stun-addresses) to + try and determine which `ip:port` can be used to communicate with their + counterpart. See [below](#stun-in-a-natshell) for more details on how this + process works. - All outbound UDP traffic must be allowed for both the client and the agent on **all ports** to each others' respective networks. - To establish a direct connection, both agent and client use STUN. This @@ -51,6 +58,19 @@ In order for clients to be able to establish direct connections: ### STUN in a NATshell +In order for one application to connect to another across a network, the +connecting application needs to know the IP address and port under which the +target application is reachable. If both applications reside on the same +network, then they can most likely connect directly to each other. In the +context of a Coder workspace agent and client, this is generally not the case, +as both agent and client will most likely be running in different _private_ +networks (e.g. `192.168.1.0/24`). In this case, at least one of the two will +need to know an IP address and port under which they can reach their +counterpart. + +This problem is often referred to as NAT traversal, and Coder uses a standard +protocol named STUN to address this. + > [Session Traversal Utilities for NAT (STUN)](https://www.rfc-editor.org/rfc/rfc8489.html) > is a protocol used to assist applications in establishing peer-to-peer > communications across Network Address Translations (NATs) or firewalls. @@ -60,26 +80,25 @@ In order for clients to be able to establish direct connections: > single public IP address. The vast majority of ISPs today use at least one > level of NAT. -Normally, both Coder agent and client will be running in different _private_ -networks (e.g. `192.168.1.0/24`). In order for them to communicate with each -other, they will each need their counterpart's public IP address and port. - Inside of that network, packets from the agent or client will show up as having source address `192.168.1.X:12345`. However, outside of this private network, the source address will show up differently (for example, `12.3.4.56:54321`). In order for the Coder client and agent to establish a direct connection with each other, one of them needs to know the `ip:port` pair under which their -counterpart can be reached. +counterpart can be reached. Once communication succeeds in one direction, we can +inspect the source address of the received packet to determine the return +address. -In order to accomplish this, both client and agent can use a STUN server: +At a high level, STUN works like this: > The below glosses over a lot of the complexity of traversing NATs. For a more > in-depth technical explanation, see > [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). -- **Discovery:** Both the client and agent will send UDP traffic to a configured - STUN server. This STUN server is generally located on the public internet, and - responds with the public IP address and port from which the request came. +- **Discovery:** Both the client and agent will send UDP traffic to one or more + configured STUN servers. These STUN servers are generally located on the + public internet, and respond with the public IP address and port from which + the request came. - **Port Mapping:** The client and agent then exchange this information through the Coder server. They will then construct packets that should be able to successfully traverse their counterpart's NATs successfully. @@ -133,6 +152,12 @@ Direct connections are a straight line between the user and workspace, so there is no special geo-distribution configuration. To speed up direct connections, move the user and workspace closer together. +Establishing a direct connection can be an involved process because both the +client and workspace agent will likely be behind at least one level of NAT, +meaning that we need to use STUN to learn the IP address and port under which +the client and agent can both contact each other. See +[above](#stun-in-a-natshell) for more information on how this process works. + If a direct connection is not available (e.g. client or server is behind NAT), Coder will use a relayed connection. By default, [Coder uses Google's public STUN server](../cli/server.md#--derp-server-stun-addresses), From 9d9759ba12c768e395093767cf14c59057874dfa Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Wed, 7 Feb 2024 20:27:16 +0000 Subject: [PATCH 09/18] add one example scenario --- docs/networking/index.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/networking/index.md b/docs/networking/index.md index 07bc690794834..03eca5e85317e 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -113,6 +113,22 @@ At a high level, STUN works like this: At this point, both the client and agent should be able to send traffic directly to each other. +Below are some example scenarios: + +1. Direct connections without NAT or STUN + + ```mermaid + flowchart LR + A[Client Workstation\n192.168.21.47:38297] + B((Private Network\ne.g. Corp. LAN)) + C[Workspace Agent\n192.168.21.147:41563] + A <--> B + B <--> C + ``` + + In this example, both the client and agent are located on the network `192.168.21.0/24`. Assuming no firewalls are blocking packets in either direction, both client and agent are able to communicate directly with each other's locally assigned IP address. + + ## coder server Workspaces connect to the coder server via the server's external address, set From ad2d95304b085f2e7c9addc33ba1d9d118dd62bd Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Thu, 8 Feb 2024 15:53:07 +0000 Subject: [PATCH 10/18] add more scenarios and diagrams --- docs/networking/index.md | 95 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 3 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index 03eca5e85317e..552a840cb4324 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -119,15 +119,104 @@ Below are some example scenarios: ```mermaid flowchart LR + subgraph corpnet["Private Network\ne.g. Corp. LAN"] A[Client Workstation\n192.168.21.47:38297] - B((Private Network\ne.g. Corp. LAN)) C[Workspace Agent\n192.168.21.147:41563] - A <--> B - B <--> C + A <--> C + end ``` In this example, both the client and agent are located on the network `192.168.21.0/24`. Assuming no firewalls are blocking packets in either direction, both client and agent are able to communicate directly with each other's locally assigned IP address. +2. Direct connections with one layer of NAT + + ```mermaid + flowchart LR + subgraph homenet["Network A"] + client["Client workstation\n192.168.1.101:38297"] + homenat["NAT\n??.??.??.??:?????"] + end + subgraph internet["Public Internet"] + stun1["STUN server"] + end + subgraph corpnet["Network B"] + agent["Workspace agent\n10.21.43.241:56812"] + corpnat["NAT\n??.??.??.??:?????"] + end + + client --- homenat + agent --- corpnat + corpnat -- "[I see 12.34.56.7:41563]" --> stun1 + homenat -- "[I see 65.4.3.21:29187]" --> stun1 + ``` + + In this example, client and agent are located on different networks and connect to each other over the public Internet. Both client and agent connect to a configured STUN server located on the public Internet to determine the public IP address and port on which they can be reached. They then exchange this information through Coder server, and can then communicate directly with each other through their respective NATs. + + ```mermaid + flowchart LR + subgraph homenet["Home Network"] + direction LR + client["Client workstation\n192.168.1.101:38297"] + homenat["Home Router/NAT\n65.4.3.21:29187"] + end + subgraph corpnet["Corp Network"] + direction LR + agent["Workspace agent\n10.21.43.241:56812"] + corpnat["Corp Router/NAT\n12.34.56.7:41563"] + end + + subgraph internet["Public Internet"] + end + + client -- "[12.34.56.7:41563]" --- homenat + homenat -- "[12.34.56.7:41563]" --- internet + internet -- "[12.34.56.7:41563]" --- corpnat + corpnat -- "[10.21.43.241:56812]" --> agent + ``` + +3. Direct connections with VPN. + + In this example, the client workstation must use a VPN to connect to the corporate network. All traffic from the client will enter through the VPN entry node and exit at the VPN exit node inside the corporate network. Traffic from the client inside the corporate network will appear to be coming from the IP address of the VPN exit node `172.16.1.2`. Traffic from the client to the public internet will appear to have the public IP address of the corporate router `12.34.56.7`. + + The workspace agent is running on a Kubernetes cluster inside the corporate network, which is behind its own layer of NAT. To anyone inside the corporate network but outside the cluster network, its traffic will appear to be coming from `172.16.1.254`. However, traffic from the agent to services on the public Internet will also see traffic originating from the public IP address assigned to the corporate router. + + If the client and agent both use the public STUN server, the addresses discoverd by STUN will both be the public IP address of the corporate router. To correctly route the traffic backwards, the corporate router must correctly map packets sent from the client to the cluster router, and from the agent to the VPN exit node. This behaviour is known as "hairpinning", and does not work in all cases. + + In this configuration, deploying an internal STUN server can aid establishing direct connections between client and agent. When the agent and client query this internal STUN server, they will be able to determine the addresses on the corporate network from which their traffic appears to originate. Using these internal addresses is much more likely to result in a successful direct connection. + + ```mermaid + flowchart TD + subgraph homenet["Home Network"] + client["Client workstation\n192.168.1.101"] + homenat["Home Router/NAT\n65.4.3.21"] + end + + subgraph internet["Public Internet"] + stun1["Public STUN"] + vpn1["VPN entry node"] + end + + subgraph corpnet["Corp Network 172.16.1.0/24"] + corpnat["Corp Router/NAT\n172.16.1.1\n12.34.56.7"] + vpn2["VPN exit node\n172.16.1.2"] + stun2["Private STUN"] + + subgraph cluster["Cluster Network 10.11.12.0/16"] + clusternat["Cluster Router/NAT\n10.11.12.1\n172.16.1.254"] + agent["Workspace agent\n10.11.12.34"] + end + end + + vpn1 === vpn2 + vpn2 --> stun2 + client === homenat + homenat === vpn1 + homenat x-.-x stun1 + agent --- clusternat + clusternat --- corpnat + corpnat --> stun1 + corpnat --> stun2 + ``` ## coder server From 7d2b50f70fda1d045dff7e93db89245b55e063c7 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Thu, 8 Feb 2024 15:57:55 +0000 Subject: [PATCH 11/18] move stun docs to separate page --- docs/networking/index.md | 166 +------------------------------------- docs/networking/stun.md | 168 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 170 insertions(+), 164 deletions(-) create mode 100644 docs/networking/stun.md diff --git a/docs/networking/index.md b/docs/networking/index.md index 552a840cb4324..3b3c8201851c0 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -42,7 +42,7 @@ In order for clients to be able to establish direct connections: agent will contact [the configured STUN servers](../cli/server.md#derp-server-stun-addresses) to try and determine which `ip:port` can be used to communicate with their - counterpart. See [below](#stun-in-a-natshell) for more details on how this + counterpart. See [STUN and NAT](./stun.md) for more details on how this process works. - All outbound UDP traffic must be allowed for both the client and the agent on **all ports** to each others' respective networks. @@ -56,168 +56,6 @@ In order for clients to be able to establish direct connections: ephemeral (high) ports. If a firewall between the client and the agent blocks this UDP traffic, direct connections will not be possible. -### STUN in a NATshell - -In order for one application to connect to another across a network, the -connecting application needs to know the IP address and port under which the -target application is reachable. If both applications reside on the same -network, then they can most likely connect directly to each other. In the -context of a Coder workspace agent and client, this is generally not the case, -as both agent and client will most likely be running in different _private_ -networks (e.g. `192.168.1.0/24`). In this case, at least one of the two will -need to know an IP address and port under which they can reach their -counterpart. - -This problem is often referred to as NAT traversal, and Coder uses a standard -protocol named STUN to address this. - -> [Session Traversal Utilities for NAT (STUN)](https://www.rfc-editor.org/rfc/rfc8489.html) -> is a protocol used to assist applications in establishing peer-to-peer -> communications across Network Address Translations (NATs) or firewalls. -> -> [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) -> is commonly used in private networks to allow multiple devices to share a -> single public IP address. The vast majority of ISPs today use at least one -> level of NAT. - -Inside of that network, packets from the agent or client will show up as having -source address `192.168.1.X:12345`. However, outside of this private network, -the source address will show up differently (for example, `12.3.4.56:54321`). In -order for the Coder client and agent to establish a direct connection with each -other, one of them needs to know the `ip:port` pair under which their -counterpart can be reached. Once communication succeeds in one direction, we can -inspect the source address of the received packet to determine the return -address. - -At a high level, STUN works like this: - -> The below glosses over a lot of the complexity of traversing NATs. For a more -> in-depth technical explanation, see -> [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). - -- **Discovery:** Both the client and agent will send UDP traffic to one or more - configured STUN servers. These STUN servers are generally located on the - public internet, and respond with the public IP address and port from which - the request came. -- **Port Mapping:** The client and agent then exchange this information through - the Coder server. They will then construct packets that should be able to - successfully traverse their counterpart's NATs successfully. -- **NAT Traversal:** The client and agent then send these crafted packets to - their counterpart's public addresses. If all goes well, the NATs on the other - end should route these packets to the correct internal address. -- **Connection:** Once the packets reach the other side, they send a response - back to the source `ip:port` from the packet. Again, the NATs should recognize - these responses as belonging to an ongoing communication, and forward them - accordingly. - -At this point, both the client and agent should be able to send traffic directly -to each other. - -Below are some example scenarios: - -1. Direct connections without NAT or STUN - - ```mermaid - flowchart LR - subgraph corpnet["Private Network\ne.g. Corp. LAN"] - A[Client Workstation\n192.168.21.47:38297] - C[Workspace Agent\n192.168.21.147:41563] - A <--> C - end - ``` - - In this example, both the client and agent are located on the network `192.168.21.0/24`. Assuming no firewalls are blocking packets in either direction, both client and agent are able to communicate directly with each other's locally assigned IP address. - -2. Direct connections with one layer of NAT - - ```mermaid - flowchart LR - subgraph homenet["Network A"] - client["Client workstation\n192.168.1.101:38297"] - homenat["NAT\n??.??.??.??:?????"] - end - subgraph internet["Public Internet"] - stun1["STUN server"] - end - subgraph corpnet["Network B"] - agent["Workspace agent\n10.21.43.241:56812"] - corpnat["NAT\n??.??.??.??:?????"] - end - - client --- homenat - agent --- corpnat - corpnat -- "[I see 12.34.56.7:41563]" --> stun1 - homenat -- "[I see 65.4.3.21:29187]" --> stun1 - ``` - - In this example, client and agent are located on different networks and connect to each other over the public Internet. Both client and agent connect to a configured STUN server located on the public Internet to determine the public IP address and port on which they can be reached. They then exchange this information through Coder server, and can then communicate directly with each other through their respective NATs. - - ```mermaid - flowchart LR - subgraph homenet["Home Network"] - direction LR - client["Client workstation\n192.168.1.101:38297"] - homenat["Home Router/NAT\n65.4.3.21:29187"] - end - subgraph corpnet["Corp Network"] - direction LR - agent["Workspace agent\n10.21.43.241:56812"] - corpnat["Corp Router/NAT\n12.34.56.7:41563"] - end - - subgraph internet["Public Internet"] - end - - client -- "[12.34.56.7:41563]" --- homenat - homenat -- "[12.34.56.7:41563]" --- internet - internet -- "[12.34.56.7:41563]" --- corpnat - corpnat -- "[10.21.43.241:56812]" --> agent - ``` - -3. Direct connections with VPN. - - In this example, the client workstation must use a VPN to connect to the corporate network. All traffic from the client will enter through the VPN entry node and exit at the VPN exit node inside the corporate network. Traffic from the client inside the corporate network will appear to be coming from the IP address of the VPN exit node `172.16.1.2`. Traffic from the client to the public internet will appear to have the public IP address of the corporate router `12.34.56.7`. - - The workspace agent is running on a Kubernetes cluster inside the corporate network, which is behind its own layer of NAT. To anyone inside the corporate network but outside the cluster network, its traffic will appear to be coming from `172.16.1.254`. However, traffic from the agent to services on the public Internet will also see traffic originating from the public IP address assigned to the corporate router. - - If the client and agent both use the public STUN server, the addresses discoverd by STUN will both be the public IP address of the corporate router. To correctly route the traffic backwards, the corporate router must correctly map packets sent from the client to the cluster router, and from the agent to the VPN exit node. This behaviour is known as "hairpinning", and does not work in all cases. - - In this configuration, deploying an internal STUN server can aid establishing direct connections between client and agent. When the agent and client query this internal STUN server, they will be able to determine the addresses on the corporate network from which their traffic appears to originate. Using these internal addresses is much more likely to result in a successful direct connection. - - ```mermaid - flowchart TD - subgraph homenet["Home Network"] - client["Client workstation\n192.168.1.101"] - homenat["Home Router/NAT\n65.4.3.21"] - end - - subgraph internet["Public Internet"] - stun1["Public STUN"] - vpn1["VPN entry node"] - end - - subgraph corpnet["Corp Network 172.16.1.0/24"] - corpnat["Corp Router/NAT\n172.16.1.1\n12.34.56.7"] - vpn2["VPN exit node\n172.16.1.2"] - stun2["Private STUN"] - - subgraph cluster["Cluster Network 10.11.12.0/16"] - clusternat["Cluster Router/NAT\n10.11.12.1\n172.16.1.254"] - agent["Workspace agent\n10.11.12.34"] - end - end - - vpn1 === vpn2 - vpn2 --> stun2 - client === homenat - homenat === vpn1 - homenat x-.-x stun1 - agent --- clusternat - clusternat --- corpnat - corpnat --> stun1 - corpnat --> stun2 - ``` - ## coder server Workspaces connect to the coder server via the server's external address, set @@ -261,7 +99,7 @@ Establishing a direct connection can be an involved process because both the client and workspace agent will likely be behind at least one level of NAT, meaning that we need to use STUN to learn the IP address and port under which the client and agent can both contact each other. See -[above](#stun-in-a-natshell) for more information on how this process works. +[STUN and NAT](./stun.md) for more information on how this process works. If a direct connection is not available (e.g. client or server is behind NAT), Coder will use a relayed connection. By default, diff --git a/docs/networking/stun.md b/docs/networking/stun.md new file mode 100644 index 0000000000000..3cf87362d7559 --- /dev/null +++ b/docs/networking/stun.md @@ -0,0 +1,168 @@ +# STUN and NAT + +> [Session Traversal Utilities for NAT (STUN)](https://www.rfc-editor.org/rfc/rfc8489.html) +> is a protocol used to assist applications in establishing peer-to-peer +> communications across Network Address Translations (NATs) or firewalls. +> +> [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) +> is commonly used in private networks to allow multiple devices to share a +> single public IP address. The vast majority of ISPs today use at least one +> level of NAT. + +## Overview + +In order for one application to connect to another across a network, the +connecting application needs to know the IP address and port under which the +target application is reachable. If both applications reside on the same +network, then they can most likely connect directly to each other. In the +context of a Coder workspace agent and client, this is generally not the case, +as both agent and client will most likely be running in different _private_ +networks (e.g. `192.168.1.0/24`). In this case, at least one of the two will +need to know an IP address and port under which they can reach their +counterpart. + +This problem is often referred to as NAT traversal, and Coder uses a standard +protocol named STUN to address this. + +Inside of that network, packets from the agent or client will show up as having +source address `192.168.1.X:12345`. However, outside of this private network, +the source address will show up differently (for example, `12.3.4.56:54321`). In +order for the Coder client and agent to establish a direct connection with each +other, one of them needs to know the `ip:port` pair under which their +counterpart can be reached. Once communication succeeds in one direction, we can +inspect the source address of the received packet to determine the return +address. + + + +At a high level, STUN works like this: + +> The below glosses over a lot of the complexity of traversing NATs. For a more +> in-depth technical explanation, see +> [How NAT traversal works (tailscale.com)](https://tailscale.com/blog/how-nat-traversal-works). + +- **Discovery:** Both the client and agent will send UDP traffic to one or more + configured STUN servers. These STUN servers are generally located on the + public internet, and respond with the public IP address and port from which + the request came. +- **Port Mapping:** The client and agent then exchange this information through + the Coder server. They will then construct packets that should be able to + successfully traverse their counterpart's NATs successfully. +- **NAT Traversal:** The client and agent then send these crafted packets to + their counterpart's public addresses. If all goes well, the NATs on the other + end should route these packets to the correct internal address. +- **Connection:** Once the packets reach the other side, they send a response + back to the source `ip:port` from the packet. Again, the NATs should recognize + these responses as belonging to an ongoing communication, and forward them + accordingly. + +At this point, both the client and agent should be able to send traffic directly +to each other. + +## Examples + +In this example, both the client and agent are located on the network `192.168.21.0/24`. Assuming no firewalls are blocking packets in either direction, both client and agent are able to communicate directly with each other's locally assigned IP address. + +### 1. Direct connections without NAT or STUN + +```mermaid +flowchart LR + subgraph corpnet["Private Network\ne.g. Corp. LAN"] + A[Client Workstation\n192.168.21.47:38297] + C[Workspace Agent\n192.168.21.147:41563] + A <--> C + end +``` + +### 2. Direct connections with one layer of NAT + +In this example, client and agent are located on different networks and connect to each other over the public Internet. Both client and agent connect to a configured STUN server located on the public Internet to determine the public IP address and port on which they can be reached. + +```mermaid +flowchart LR + subgraph homenet["Network A"] + client["Client workstation\n192.168.1.101:38297"] + homenat["NAT\n??.??.??.??:?????"] + end + subgraph internet["Public Internet"] + stun1["STUN server"] + end + subgraph corpnet["Network B"] + agent["Workspace agent\n10.21.43.241:56812"] + corpnat["NAT\n??.??.??.??:?????"] + end + + client --- homenat + agent --- corpnat + corpnat -- "[I see 12.34.56.7:41563]" --> stun1 + homenat -- "[I see 65.4.3.21:29187]" --> stun1 +``` + + +They then exchange this information through Coder server, and can then communicate directly with each other through their respective NATs. + +```mermaid +flowchart LR + subgraph homenet["Home Network"] + direction LR + client["Client workstation\n192.168.1.101:38297"] + homenat["Home Router/NAT\n65.4.3.21:29187"] + end + subgraph corpnet["Corp Network"] + direction LR + agent["Workspace agent\n10.21.43.241:56812"] + corpnat["Corp Router/NAT\n12.34.56.7:41563"] + end + + subgraph internet["Public Internet"] + end + + client -- "[12.34.56.7:41563]" --- homenat + homenat -- "[12.34.56.7:41563]" --- internet + internet -- "[12.34.56.7:41563]" --- corpnat + corpnat -- "[10.21.43.241:56812]" --> agent +``` + +### 3. Direct connections with VPN and NAT hairpinning + +In this example, the client workstation must use a VPN to connect to the corporate network. All traffic from the client will enter through the VPN entry node and exit at the VPN exit node inside the corporate network. Traffic from the client inside the corporate network will appear to be coming from the IP address of the VPN exit node `172.16.1.2`. Traffic from the client to the public internet will appear to have the public IP address of the corporate router `12.34.56.7`. + +The workspace agent is running on a Kubernetes cluster inside the corporate network, which is behind its own layer of NAT. To anyone inside the corporate network but outside the cluster network, its traffic will appear to be coming from `172.16.1.254`. However, traffic from the agent to services on the public Internet will also see traffic originating from the public IP address assigned to the corporate router. + +If the client and agent both use the public STUN server, the addresses discoverd by STUN will both be the public IP address of the corporate router. To correctly route the traffic backwards, the corporate router must correctly map packets sent from the client to the cluster router, and from the agent to the VPN exit node. This behaviour is known as "hairpinning", and does not work in all cases. + +In this configuration, deploying an internal STUN server can aid establishing direct connections between client and agent. When the agent and client query this internal STUN server, they will be able to determine the addresses on the corporate network from which their traffic appears to originate. Using these internal addresses is much more likely to result in a successful direct connection. + +```mermaid +flowchart TD + subgraph homenet["Home Network"] + client["Client workstation\n192.168.1.101"] + homenat["Home Router/NAT\n65.4.3.21"] + end + + subgraph internet["Public Internet"] + stun1["Public STUN"] + vpn1["VPN entry node"] + end + + subgraph corpnet["Corp Network 172.16.1.0/24"] + corpnat["Corp Router/NAT\n172.16.1.1\n12.34.56.7"] + vpn2["VPN exit node\n172.16.1.2"] + stun2["Private STUN"] + + subgraph cluster["Cluster Network 10.11.12.0/16"] + clusternat["Cluster Router/NAT\n10.11.12.1\n172.16.1.254"] + agent["Workspace agent\n10.11.12.34"] + end + end + + vpn1 === vpn2 + vpn2 --> stun2 + client === homenat + homenat === vpn1 + homenat x-.-x stun1 + agent --- clusternat + clusternat --- corpnat + corpnat --> stun1 + corpnat --> stun2 +``` From bd81137e1b6090d45bb4b1f3a816dd6cec5e1696 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Thu, 8 Feb 2024 15:58:26 +0000 Subject: [PATCH 12/18] make fmt --- docs/networking/index.md | 4 ++-- docs/networking/stun.md | 50 +++++++++++++++++++++++++++++----------- 2 files changed, 39 insertions(+), 15 deletions(-) diff --git a/docs/networking/index.md b/docs/networking/index.md index 3b3c8201851c0..c1eb41869b3bf 100644 --- a/docs/networking/index.md +++ b/docs/networking/index.md @@ -98,8 +98,8 @@ move the user and workspace closer together. Establishing a direct connection can be an involved process because both the client and workspace agent will likely be behind at least one level of NAT, meaning that we need to use STUN to learn the IP address and port under which -the client and agent can both contact each other. See -[STUN and NAT](./stun.md) for more information on how this process works. +the client and agent can both contact each other. See [STUN and NAT](./stun.md) +for more information on how this process works. If a direct connection is not available (e.g. client or server is behind NAT), Coder will use a relayed connection. By default, diff --git a/docs/networking/stun.md b/docs/networking/stun.md index 3cf87362d7559..1a34d3f1a2f27 100644 --- a/docs/networking/stun.md +++ b/docs/networking/stun.md @@ -33,8 +33,6 @@ counterpart can be reached. Once communication succeeds in one direction, we can inspect the source address of the received packet to determine the return address. - - At a high level, STUN works like this: > The below glosses over a lot of the complexity of traversing NATs. For a more @@ -61,7 +59,10 @@ to each other. ## Examples -In this example, both the client and agent are located on the network `192.168.21.0/24`. Assuming no firewalls are blocking packets in either direction, both client and agent are able to communicate directly with each other's locally assigned IP address. +In this example, both the client and agent are located on the network +`192.168.21.0/24`. Assuming no firewalls are blocking packets in either +direction, both client and agent are able to communicate directly with each +other's locally assigned IP address. ### 1. Direct connections without NAT or STUN @@ -76,7 +77,10 @@ flowchart LR ### 2. Direct connections with one layer of NAT -In this example, client and agent are located on different networks and connect to each other over the public Internet. Both client and agent connect to a configured STUN server located on the public Internet to determine the public IP address and port on which they can be reached. +In this example, client and agent are located on different networks and connect +to each other over the public Internet. Both client and agent connect to a +configured STUN server located on the public Internet to determine the public IP +address and port on which they can be reached. ```mermaid flowchart LR @@ -98,8 +102,8 @@ flowchart LR homenat -- "[I see 65.4.3.21:29187]" --> stun1 ``` - -They then exchange this information through Coder server, and can then communicate directly with each other through their respective NATs. +They then exchange this information through Coder server, and can then +communicate directly with each other through their respective NATs. ```mermaid flowchart LR @@ -125,13 +129,33 @@ flowchart LR ### 3. Direct connections with VPN and NAT hairpinning -In this example, the client workstation must use a VPN to connect to the corporate network. All traffic from the client will enter through the VPN entry node and exit at the VPN exit node inside the corporate network. Traffic from the client inside the corporate network will appear to be coming from the IP address of the VPN exit node `172.16.1.2`. Traffic from the client to the public internet will appear to have the public IP address of the corporate router `12.34.56.7`. - -The workspace agent is running on a Kubernetes cluster inside the corporate network, which is behind its own layer of NAT. To anyone inside the corporate network but outside the cluster network, its traffic will appear to be coming from `172.16.1.254`. However, traffic from the agent to services on the public Internet will also see traffic originating from the public IP address assigned to the corporate router. - -If the client and agent both use the public STUN server, the addresses discoverd by STUN will both be the public IP address of the corporate router. To correctly route the traffic backwards, the corporate router must correctly map packets sent from the client to the cluster router, and from the agent to the VPN exit node. This behaviour is known as "hairpinning", and does not work in all cases. - -In this configuration, deploying an internal STUN server can aid establishing direct connections between client and agent. When the agent and client query this internal STUN server, they will be able to determine the addresses on the corporate network from which their traffic appears to originate. Using these internal addresses is much more likely to result in a successful direct connection. +In this example, the client workstation must use a VPN to connect to the +corporate network. All traffic from the client will enter through the VPN entry +node and exit at the VPN exit node inside the corporate network. Traffic from +the client inside the corporate network will appear to be coming from the IP +address of the VPN exit node `172.16.1.2`. Traffic from the client to the public +internet will appear to have the public IP address of the corporate router +`12.34.56.7`. + +The workspace agent is running on a Kubernetes cluster inside the corporate +network, which is behind its own layer of NAT. To anyone inside the corporate +network but outside the cluster network, its traffic will appear to be coming +from `172.16.1.254`. However, traffic from the agent to services on the public +Internet will also see traffic originating from the public IP address assigned +to the corporate router. + +If the client and agent both use the public STUN server, the addresses discoverd +by STUN will both be the public IP address of the corporate router. To correctly +route the traffic backwards, the corporate router must correctly map packets +sent from the client to the cluster router, and from the agent to the VPN exit +node. This behaviour is known as "hairpinning", and does not work in all cases. + +In this configuration, deploying an internal STUN server can aid establishing +direct connections between client and agent. When the agent and client query +this internal STUN server, they will be able to determine the addresses on the +corporate network from which their traffic appears to originate. Using these +internal addresses is much more likely to result in a successful direct +connection. ```mermaid flowchart TD From 56929f7496eac937319cb86994ce658834b5d967 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Thu, 8 Feb 2024 16:21:02 +0000 Subject: [PATCH 13/18] fix typo --- docs/networking/stun.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/networking/stun.md b/docs/networking/stun.md index 1a34d3f1a2f27..e07b3dadfbaba 100644 --- a/docs/networking/stun.md +++ b/docs/networking/stun.md @@ -144,11 +144,12 @@ from `172.16.1.254`. However, traffic from the agent to services on the public Internet will also see traffic originating from the public IP address assigned to the corporate router. -If the client and agent both use the public STUN server, the addresses discoverd -by STUN will both be the public IP address of the corporate router. To correctly -route the traffic backwards, the corporate router must correctly map packets -sent from the client to the cluster router, and from the agent to the VPN exit -node. This behaviour is known as "hairpinning", and does not work in all cases. +If the client and agent both use the public STUN server, the addresses +discovered by STUN will both be the public IP address of the corporate router. +To correctly route the traffic backwards, the corporate router must correctly +map packets sent from the client to the cluster router, and from the agent to +the VPN exit node. This behaviour is known as "hairpinning", and does not work +in all cases. In this configuration, deploying an internal STUN server can aid establishing direct connections between client and agent. When the agent and client query From e3a686523a69aeda22750246072e4dfabc3e5ddf Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Thu, 8 Feb 2024 19:18:20 +0000 Subject: [PATCH 14/18] adjust wording --- docs/networking/stun.md | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/docs/networking/stun.md b/docs/networking/stun.md index e07b3dadfbaba..35b3a483b8d3d 100644 --- a/docs/networking/stun.md +++ b/docs/networking/stun.md @@ -142,21 +142,29 @@ network, which is behind its own layer of NAT. To anyone inside the corporate network but outside the cluster network, its traffic will appear to be coming from `172.16.1.254`. However, traffic from the agent to services on the public Internet will also see traffic originating from the public IP address assigned -to the corporate router. +to the corporate router. Additionally, the corporate router will most likely +have a firewall configured to block traffic from the internet to the corporate +network. If the client and agent both use the public STUN server, the addresses discovered by STUN will both be the public IP address of the corporate router. To correctly route the traffic backwards, the corporate router must correctly -map packets sent from the client to the cluster router, and from the agent to -the VPN exit node. This behaviour is known as "hairpinning", and does not work -in all cases. - -In this configuration, deploying an internal STUN server can aid establishing -direct connections between client and agent. When the agent and client query -this internal STUN server, they will be able to determine the addresses on the -corporate network from which their traffic appears to originate. Using these -internal addresses is much more likely to result in a successful direct -connection. +route both: + +- Traffic sent from the client to the external IP of the corporate router back + to the cluster router, and +- Traffic sent from the agent to the external IP of the corporate router to the + VPN exit node. + +This behaviour is known as "hairpinning", and may not be supported in all +network configurations. + +If hairpinning is not supported, deploying an internal STUN server can aid +establishing direct connections between client and agent. When the agent and +client query this internal STUN server, they will be able to determine the +addresses on the corporate network from which their traffic appears to +originate. Using these internal addresses is much more likely to result in a +successful direct connection. ```mermaid flowchart TD From 4b4d24616d82870fce324a53bc0e07127e07f49f Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Mon, 12 Feb 2024 09:06:47 +0000 Subject: [PATCH 15/18] Apply suggestions from code review Co-authored-by: Spike Curtis --- docs/networking/stun.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/networking/stun.md b/docs/networking/stun.md index 35b3a483b8d3d..0e8fb2f1daee2 100644 --- a/docs/networking/stun.md +++ b/docs/networking/stun.md @@ -6,7 +6,7 @@ > > [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) > is commonly used in private networks to allow multiple devices to share a -> single public IP address. The vast majority of ISPs today use at least one +> single public IP address. The vast majority of home and corporate internet connections use at least one > level of NAT. ## Overview @@ -43,7 +43,7 @@ At a high level, STUN works like this: configured STUN servers. These STUN servers are generally located on the public internet, and respond with the public IP address and port from which the request came. -- **Port Mapping:** The client and agent then exchange this information through +- **Coordination:** The client and agent then exchange this information through the Coder server. They will then construct packets that should be able to successfully traverse their counterpart's NATs successfully. - **NAT Traversal:** The client and agent then send these crafted packets to From 9726796262a3ac5688033b70287ba7ac6e8a609f Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Mon, 12 Feb 2024 10:45:29 +0000 Subject: [PATCH 16/18] update manifest --- docs/manifest.json | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/manifest.json b/docs/manifest.json index 58a3e82cdba7b..bb8223dfafbb1 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -292,6 +292,11 @@ "title": "Port Forwarding", "description": "Learn how to forward ports in Coder", "path": "./networking/port-forwarding.md" + }, + { + "title": "STUN and NAT", + "description": "Learn how Coder establishes direct connections", + "path": "./networking/stun.md" } ] }, From 6ac84d26ba846aad794e9f8a35fd9016fd83d8a8 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Mon, 12 Feb 2024 11:32:46 +0000 Subject: [PATCH 17/18] adjust graph for example 2 --- docs/networking/stun.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/networking/stun.md b/docs/networking/stun.md index 0e8fb2f1daee2..793418659ca92 100644 --- a/docs/networking/stun.md +++ b/docs/networking/stun.md @@ -107,24 +107,23 @@ communicate directly with each other through their respective NATs. ```mermaid flowchart LR - subgraph homenet["Home Network"] - direction LR + subgraph homenet["Network A"] client["Client workstation\n192.168.1.101:38297"] - homenat["Home Router/NAT\n65.4.3.21:29187"] + homenat["NAT\n65.4.3.21:29187"] end - subgraph corpnet["Corp Network"] - direction LR + subgraph corpnet["Network B"] agent["Workspace agent\n10.21.43.241:56812"] - corpnat["Corp Router/NAT\n12.34.56.7:41563"] + corpnat["NAT\n12.34.56.7:41563"] end subgraph internet["Public Internet"] end client -- "[12.34.56.7:41563]" --- homenat - homenat -- "[12.34.56.7:41563]" --- internet - internet -- "[12.34.56.7:41563]" --- corpnat - corpnat -- "[10.21.43.241:56812]" --> agent + agent -- "[10.21.43.241:56812]" --- corpnat + corpnat -- "[65.4.3.21:29187]" --> internet + homenat -- "[12.34.56.7:41563]" --> internet + ``` ### 3. Direct connections with VPN and NAT hairpinning From 9cad8e74f9b449a083a32e9abce548c68e414661 Mon Sep 17 00:00:00 2001 From: Cian Johnston Date: Mon, 12 Feb 2024 11:34:18 +0000 Subject: [PATCH 18/18] make fmt --- docs/networking/stun.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/networking/stun.md b/docs/networking/stun.md index 793418659ca92..8686910b03933 100644 --- a/docs/networking/stun.md +++ b/docs/networking/stun.md @@ -6,8 +6,8 @@ > > [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) > is commonly used in private networks to allow multiple devices to share a -> single public IP address. The vast majority of home and corporate internet connections use at least one -> level of NAT. +> single public IP address. The vast majority of home and corporate internet +> connections use at least one level of NAT. ## Overview