Skip to content

Commit 865c8b7

Browse files
bpmctbpmct
authored
docs: additional kubernetes clusters (#7019)
* docs: additional kubernetes clusters * fmt and change wording * fmt
1 parent eb66cc9 commit 865c8b7

File tree

4 files changed

+228
-3
lines changed

4 files changed

+228
-3
lines changed

‎docs/images/platforms/kubernetes/region-picker.png

124 KB
Loading

‎docs/manifest.json

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,14 @@
106106
{
107107
"title": "Kubernetes",
108108
"description": "Set up Coder on Kubernetes",
109-
"path": "./platforms/kubernetes.md"
109+
"path": "./platforms/kubernetes/index.md",
110+
"children": [
111+
{
112+
"title": "Additional clusters",
113+
"description": "Deploy workspaces on additional Kubernetes clusters",
114+
"path": "./platforms/kubernetes/additional-clusters.md"
115+
}
116+
]
110117
},
111118
{
112119
"title": "Other platforms",

‎docs/platforms/kubernetes/additional-clusters.md

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
# Additional clusters
2+
3+
With Coder, you can deploy workspaces in additional Kubernetes clusters using different [authentication methods](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#authentication) in the Terraform provider.
4+
5+
![Region picker in "Create workspace" screen](../../images/platforms/kubernetes/region-picker.png)
6+
7+
## Option 1) Kubernetes contexts and kubeconfig
8+
9+
First, create a kubeconfig file with [multiple contexts](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/).
10+
11+
```sh
12+
$ kubectl config get-contexts
13+
14+
CURRENT NAME CLUSTER
15+
workspaces-europe-west2-c workspaces-europe-west2-c
16+
* workspaces-us-central1-a workspaces-us-central1-a
17+
```
18+
19+
### Kubernetes control plane
20+
21+
If you deployed Coder on Kubernetes, you can attach a kubeconfig as a secret.
22+
23+
This assumes Coder is deployed on the `coder` namespace and your kubeconfig file is in ~/.kube/config.
24+
25+
```sh
26+
kubectl create secret generic kubeconfig-secret -n coder--from-file=~/.kube/config
27+
```
28+
29+
Modify your helm values to mount the secret:
30+
31+
```yaml
32+
coder:
33+
# ...
34+
volumes:
35+
- name: "kubeconfig-mount"
36+
secret:
37+
secretName: "kubeconfig-secret"
38+
volumeMounts:
39+
- name: "kubeconfig-mount"
40+
mountPath: "/mnt/secrets/kube"
41+
readOnly: true
42+
```
43+
44+
[Upgrade Coder](http://localhost:3000/docs/v2/latest/install/kubernetes#upgrading-coder-via-helm) with these new values.
45+
46+
### VM control plane
47+
48+
If you deployed Coder on a VM, copy the kubeconfig file to `/home/coder/.kube/config`.
49+
50+
### Create a Coder template
51+
52+
You can start from our [example template](https://github.com/coder/coder/tree/main/examples/templates/kubernetes). From there, add [template parameters](../../templates/parameters.md) to allow developers to pick their desired cluster.
53+
54+
```hcl
55+
# main.tf
56+
57+
data "coder_parameter" "kube_context" {
58+
name = "kube_context"
59+
display_name = "Cluster"
60+
default = "workspaces-us-central1-a"
61+
mutable = false
62+
option {
63+
name = "US Central"
64+
icon = "/emojis/1f33d.png"
65+
value = "workspaces-us-central1-a"
66+
}
67+
option {
68+
name = "Europe West"
69+
icon = "/emojis/1f482.png"
70+
value = "workspaces-europe-west2-c"
71+
}
72+
}
73+
74+
provider "kubernetes" {
75+
config_path = "~/.kube/config" # or /mnt/secrets/kube/config for Kubernetes
76+
config_context = data.coder_parameter.kube_context.value
77+
}
78+
```
79+
80+
## Option 2) Kubernetes ServiceAccounts
81+
82+
Alternatively, you can authenticate with remote clusters with ServiceAccount tokens. Coder can store these secrets on your behalf with [managed Terraform variables](../../templates/parameters.md#managed-terraform-variables).
83+
84+
Alternatively, these could also be fetched from Kubernetes secrets or even [Hashicorp Vault](https://registry.terraform.io/providers/hashicorp/vault/latest/docs/data-sources/generic_secret).
85+
86+
This guide assumes you have a `coder-workspaces` namespace on your remote cluster. Change the namespace accordingly.
87+
88+
### Create a ServiceAccount
89+
90+
Run this command against your remote cluster to create a ServiceAccount, Role, RoleBinding, and token:
91+
92+
```sh
93+
kubectl apply -n coder-workspaces -f - <<EOF
94+
apiVersion: v1
95+
kind: ServiceAccount
96+
metadata:
97+
name: coder
98+
---
99+
apiVersion: v1
100+
kind: Secret
101+
metadata:
102+
name: coder-service-account-token
103+
annotations:
104+
kubernetes.io/service-account.name: coder
105+
type: kubernetes.io/service-account-token
106+
---
107+
apiVersion: rbac.authorization.k8s.io/v1
108+
kind: Role
109+
metadata:
110+
name: coder
111+
rules:
112+
- apiGroups: ["", "apps", "networking.k8s.io"] # "" indicates the core API group
113+
resources: ["persistentvolumeclaims", "pods", "deployments", "services", "secrets", "pods/exec","pods/log", "events", "networkpolicies", "serviceaccounts"]
114+
verbs: ["create", "get", "list", "watch", "update", "patch", "delete", "deletecollection"]
115+
- apiGroups: ["metrics.k8s.io", "storage.k8s.io"]
116+
resources: ["pods", "storageclasses"]
117+
verbs: ["get", "list", "watch"]
118+
---
119+
apiVersion: rbac.authorization.k8s.io/v1
120+
kind: RoleBinding
121+
metadata:
122+
name: coder
123+
subjects:
124+
- kind: ServiceAccount
125+
name: coder
126+
roleRef:
127+
kind: Role
128+
name: coder
129+
apiGroup: rbac.authorization.k8s.io
130+
EOF
131+
```
132+
133+
The output should be similar to:
134+
135+
```text
136+
serviceaccount/coder created
137+
secret/coder-service-account-token created
138+
role.rbac.authorization.k8s.io/coder created
139+
rolebinding.rbac.authorization.k8s.io/coder created
140+
```
141+
142+
### 2. Modify the Kubernetes template
143+
144+
You can start from our [example template](https://github.com/coder/coder/tree/main/examples/templates/kubernetes).
145+
146+
```hcl
147+
variable "host" {
148+
description = "Cluster host address"
149+
sensitive = true
150+
}
151+
152+
variable "cluster_ca_certificate" {
153+
description = "Cluster CA certificate (base64 encoded)"
154+
sensitive = true
155+
}
156+
157+
variable "token" {
158+
description = "Cluster CA token (base64 encoded)"
159+
sensitive = true
160+
}
161+
162+
variable "namespace" {
163+
description = "Namespace"
164+
}
165+
166+
provider "kubernetes" {
167+
host = var.host
168+
cluster_ca_certificate = base64decode(var.cluster_ca_certificate)
169+
token = base64decode(var.token)
170+
}
171+
```
172+
173+
### Create Coder template with managed variables
174+
175+
Fetch the values from the secret and pass them to Coder. This should work on macOS and Linux.
176+
177+
To get the cluster address:
178+
179+
```sh
180+
$ kubectl cluster-info
181+
Kubernetes control plane is running at https://example.domain:6443
182+
183+
$ export CLUSTER_ADDRESS=https://example.domain:6443
184+
```
185+
186+
To fetch the CA certificate and token:
187+
188+
```sh
189+
export CLUSTER_CA_CERTIFICATE=$(kubectl get secrets coder-service-account-token -n coder-workspaces -o jsonpath="{.data.ca\.crt}")
190+
191+
export CLUSTER_SERVICEACCOUNT_TOKEN=$(kubectl get secrets coder-service-account-token -n coder-workspaces -o jsonpath="{.data.token}")
192+
```
193+
194+
Create the template with these values:
195+
196+
```sh
197+
coder templates create \
198+
--variable host=$CLUSTER_ADDRESS \
199+
--variable cluster_ca_certificate=$CLUSTER_CA_CERTIFICATE \
200+
--variable token=$CLUSTER_SERVICEACCOUNT_TOKEN \
201+
--variable namespace=coder-workspaces
202+
```
203+
204+
If you're on a Windows machine (or if one of the commands fail), try grabbing the values manually:
205+
206+
```sh
207+
# Get cluster API address
208+
kubectl cluster-info
209+
210+
# Get cluster CA and token (base64 encoded)
211+
kubectl get secrets coder-service-account-token -n coder-workspaces -o jsonpath="{.data}"
212+
213+
coder templates create \
214+
--variable host=API_ADDRESS \
215+
--variable cluster_ca_certificate=CLUSTER_CA_CERTIFICATE \
216+
--variable token=CLUSTER_SERVICEACCOUNT_TOKEN \
217+
--variable namespace=coder-workspaces
218+
```

‎docs/platforms/kubernetes.md renamed to ‎docs/platforms/kubernetes/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,13 @@ Refer to our [Helm install docs](../install/kubernetes.md) to deploy Coder on Ku
1414

1515
From the dashboard, import the Kubernetes starter template:
1616

17-
![Kubernetes starter template](../images/platforms/kubernetes/starter-template.png)
17+
![Kubernetes starter template](../../images/platforms/kubernetes/starter-template.png)
1818

1919
In the next screen, set the following template variables:
2020

2121
- use_kubeconfig: `false` (The ServiceAccount will authorize Coder to create pods on your cluster)
2222
- `namespace`: `coder` (or whatever namespace you deployed Coder on)
2323

24-
![Variables for Kubernetes template](../images/platforms/kubernetes/template-variables.png)
24+
![Variables for Kubernetes template](../../images/platforms/kubernetes/template-variables.png)
2525

2626
> If you deployed Coder on another platform besides Kubernetes, you can set `use_kubeconfig: true` for Coder to read the config from your VM, for example.

0 commit comments

Comments
 (0)