Skip to content

Commit 69358cd

Browse files
deansheatherdeansheather
committed
fixup! docs: add workspace process logging doc
1 parent c426130 commit 69358cd

File tree

2 files changed

+170
-147
lines changed

2 files changed

+170
-147
lines changed

‎docs/templates/docker-in-workspaces.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ There are a few ways to run Docker within container-based Coder workspaces.
1313

1414
The [Sysbox](https://github.com/nestybox/sysbox) container runtime allows unprivileged users to run system-level applications, such as Docker, securely from the workspace containers. Sysbox requires a [compatible Linux distribution](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md) to implement these security features. Sysbox can also be used to run systemd inside Coder workspaces. See [Systemd in Docker](#systemd-in-docker).
1515

16-
The Sysbox container runtime is not compatible with our [workspace process logging](./process-logging.md) feature.
16+
The Sysbox container runtime is not compatible with our [workspace process logging](./process-logging.md) feature. Envbox is compatible with process logging, however.
1717

1818
### Use Sysbox in Docker-based templates
1919

‎docs/templates/process-logging.md

Lines changed: 169 additions & 146 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,16 @@ processes executing in the workspace.
66
> **Note:** This feature is only available on Linux in Kubernetes. There are
77
> additional requirements outlined further in this document.
88
>
9-
> This is an Enterprise feature. To learn more about Coder Enterprise, please
10-
> contact sales.
9+
> This is an [Enterprise](https://coder.com/docs/v2/latest/enterprise) feature.
10+
> To learn more about Coder Enterprise, please
11+
> [contact sales](https://coder.com/contact).
1112
1213
Workspace process logging adds a sidecar container to workspace pods that will
1314
log all processes started in the workspace container (e.g., commands executed in
14-
the terminal or processes created in the background by other processes). You
15-
can view the output from the sidecar or send it to a monitoring stack, such as
16-
CloudWatch, for further analysis or long-term storage.
15+
the terminal or processes created in the background by other processes).
16+
Processes launched inside containers or nested containers within the workspace
17+
are also logged. You can view the output from the sidecar or send it to a
18+
monitoring stack, such as CloudWatch, for further analysis or long-term storage.
1719

1820
Please note that these logs are not recorded or captured by the Coder
1921
organization in any way, shape, or form.
@@ -25,10 +27,10 @@ impact) to perform in-kernel logging and filtering of all exec system calls
2527
originating from the workspace container.
2628

2729
The core of this feature is also open source and can be found in the
28-
[exectrace](https://github.com/coder/exectrace) repo on GitHub repo. The
29-
enterprise component (in the `enterprise/` directory of the repo) is responsible
30-
for starting the eBPF program with the correct filtering options for the
31-
specific workspace.
30+
[exectrace](https://github.com/coder/exectrace) GitHub repo. The enterprise
31+
component (in the `enterprise/` directory of the repo) is responsible for
32+
starting the eBPF program with the correct filtering options for the specific
33+
workspace.
3234

3335
## Requirements
3436

@@ -38,18 +40,18 @@ The host machine must be running a Linux kernel >= 5.8 with the kernel config
3840
To check your kernel version, run:
3941

4042
```shell
41-
$ uname -r
43+
uname -r
4244
```
4345

4446
To validate the required kernel config is enabled, run either of the following
4547
commands on your nodes directly (_not_ from a workspace terminal):
4648

4749
```shell
48-
$ cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF
50+
cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF
4951
```
5052

5153
```shell
52-
$ cat "/boot/config-$(uname -r)" | grep CONFIG_DEBUG_INFO_BTF
54+
cat "/boot/config-$(uname -r)" | grep CONFIG_DEBUG_INFO_BTF
5355
```
5456

5557
If these requirements are not met, workspaces will fail to start for security
@@ -75,145 +77,165 @@ would like to add workspace process logging to, follow these steps:
7577

7678
1. Add the following section to your template's `main.tf` file:
7779

78-
```hcl
79-
locals {
80-
# This is the init script for the main workspace container that runs before the
81-
# agent starts to configure workspace process logging.
82-
exectrace_init_script = <<EOT
83-
set -eu
84-
pidns_inum=$(readlink /proc/self/ns/pid | sed 's/[^0-9]//g')
85-
if [ -z "$pidns_inum" ]; then
86-
echo "Could not determine process ID namespace inum"
87-
exit 1
88-
fi
89-
90-
# Before we start the script, does curl exist?
91-
if ! command -v curl >/dev/null 2>&1; then
92-
echo "curl is required to download the Coder binary"
93-
echo "Please install curl to your image and try again"
94-
# 127 is command not found.
95-
exit 127
96-
fi
97-
98-
echo "Sending process ID namespace inum to exectrace sidecar"
99-
rc=0
100-
max_retry=5
101-
counter=0
102-
until [ $counter -ge $max_retry ]; do
103-
set +e
104-
curl \
105-
--fail \
106-
--silent \
107-
--connect-timeout 5 \
108-
-X POST \
109-
-H "Content-Type: text/plain" \
110-
--data "$pidns_inum" \
111-
http://127.0.0.1:56123
112-
rc=$?
113-
set -e
114-
if [ $rc -eq 0 ]; then
115-
break
116-
fi
117-
118-
counter=$((counter+1))
119-
echo "Curl failed with exit code $${rc}, attempt $${counter}/$${max_retry}; Retrying in 3 seconds..."
120-
sleep 3
121-
done
122-
if [ $rc -ne 0 ]; then
123-
echo "Failed to send process ID namespace inum to exectrace sidecar"
124-
exit $rc
125-
fi
126-
127-
EOT
128-
}
129-
```
80+
<!--
81+
If you are updating this section, please also update the example templates
82+
in the exectrace repo.
83+
-->
84+
85+
```hcl
86+
locals {
87+
# This is the init script for the main workspace container that runs before the
88+
# agent starts to configure workspace process logging.
89+
exectrace_init_script = <<EOT
90+
set -eu
91+
pidns_inum=$(readlink /proc/self/ns/pid | sed 's/[^0-9]//g')
92+
if [ -z "$pidns_inum" ]; then
93+
echo "Could not determine process ID namespace inum"
94+
exit 1
95+
fi
96+
97+
# Before we start the script, does curl exist?
98+
if ! command -v curl >/dev/null 2>&1; then
99+
echo "curl is required to download the Coder binary"
100+
echo "Please install curl to your image and try again"
101+
# 127 is command not found.
102+
exit 127
103+
fi
104+
105+
echo "Sending process ID namespace inum to exectrace sidecar"
106+
rc=0
107+
max_retry=5
108+
counter=0
109+
until [ $counter -ge $max_retry ]; do
110+
set +e
111+
curl \
112+
--fail \
113+
--silent \
114+
--connect-timeout 5 \
115+
-X POST \
116+
-H "Content-Type: text/plain" \
117+
--data "$pidns_inum" \
118+
http://127.0.0.1:56123
119+
rc=$?
120+
set -e
121+
if [ $rc -eq 0 ]; then
122+
break
123+
fi
124+
125+
counter=$((counter+1))
126+
echo "Curl failed with exit code $${rc}, attempt $${counter}/$${max_retry}; Retrying in 3 seconds..."
127+
sleep 3
128+
done
129+
if [ $rc -ne 0 ]; then
130+
echo "Failed to send process ID namespace inum to exectrace sidecar"
131+
exit $rc
132+
fi
133+
134+
EOT
135+
}
136+
```
130137

131138
1. Update the `command` of your workspace container like the following:
132139

133-
```hcl
134-
resource "kubernetes_pod" "main" {
135-
...
136-
spec {
137-
...
138-
container {
139-
...
140-
// NOTE: this command is changed compared to the upstream kubernetes
141-
// template
142-
command = [
143-
"sh",
144-
"-c",
145-
"${local.exectrace_init_script}\n\n${coder_agent.main.init_script}",
146-
]
147-
...
148-
}
149-
...
150-
}
151-
...
152-
}
153-
```
154-
155-
> **Note:** If you are using the `envbox` template, you will need to update
156-
> the third argument to be
157-
> `"${local.exectrace_init_script}\n\nexec /envbox docker"` instead.
140+
<!--
141+
If you are updating this section, please also update the example templates
142+
in the exectrace repo.
143+
-->
144+
145+
```hcl
146+
resource "kubernetes_pod" "main" {
147+
...
148+
spec {
149+
...
150+
container {
151+
...
152+
// NOTE: this command is changed compared to the upstream kubernetes
153+
// template
154+
command = [
155+
"sh",
156+
"-c",
157+
"${local.exectrace_init_script}\n\n${coder_agent.main.init_script}",
158+
]
159+
...
160+
}
161+
...
162+
}
163+
...
164+
}
165+
```
166+
167+
> **Note:** If you are using the `envbox` template, you will need to update
168+
> the third argument to be
169+
> `"${local.exectrace_init_script}\n\nexec /envbox docker"` instead.
158170
159171
1. Add the following container to your workspace pod spec.
160172

161-
```hcl
162-
resource "kubernetes_pod" "main" {
163-
...
164-
spec {
165-
...
166-
// NOTE: this container is added compared to the upstream kubernetes
167-
// template
168-
container {
169-
name = "exectrace"
170-
image = "ghcr.io/coder/exectrace:latest"
171-
image_pull_policy = "Always"
172-
command = [
173-
"/opt/exectrace",
174-
"--init-address", "127.0.0.1:56123",
175-
"--label", "workspace_id=${data.coder_workspace.me.id}",
176-
"--label", "workspace_name=${data.coder_workspace.me.name}",
177-
"--label", "user_id=${data.coder_workspace.me.owner_id}",
178-
"--label", "username=${data.coder_workspace.me.owner}",
179-
"--label", "user_email=${data.coder_workspace.me.owner_email}",
180-
]
181-
security_context {
182-
// exectrace must be started as root so it can attach probes into the
183-
// kernel to record process events with high throughput.
184-
run_as_user = "0"
185-
run_as_group = "0"
186-
// exectrace requires a privileged container so it can control mounts
187-
// and perform privileged syscalls against the host kernel to attach
188-
// probes.
189-
privileged = true
190-
}
191-
}
192-
...
193-
}
194-
...
195-
}
196-
```
197-
198-
> **Note:** `exectrace` requires root privileges and a privileged container
199-
> to attach probes to the kernel. This is a requirement of eBPF.
173+
<!--
174+
If you are updating this section, please also update the example templates
175+
in the exectrace repo.
176+
-->
177+
178+
```hcl
179+
resource "kubernetes_pod" "main" {
180+
...
181+
spec {
182+
...
183+
// NOTE: this container is added compared to the upstream kubernetes
184+
// template
185+
container {
186+
name = "exectrace"
187+
image = "ghcr.io/coder/exectrace:latest"
188+
image_pull_policy = "Always"
189+
command = [
190+
"/opt/exectrace",
191+
"--init-address", "127.0.0.1:56123",
192+
"--label", "workspace_id=${data.coder_workspace.me.id}",
193+
"--label", "workspace_name=${data.coder_workspace.me.name}",
194+
"--label", "user_id=${data.coder_workspace.me.owner_id}",
195+
"--label", "username=${data.coder_workspace.me.owner}",
196+
"--label", "user_email=${data.coder_workspace.me.owner_email}",
197+
]
198+
security_context {
199+
// exectrace must be started as root so it can attach probes into the
200+
// kernel to record process events with high throughput.
201+
run_as_user = "0"
202+
run_as_group = "0"
203+
// exectrace requires a privileged container so it can control mounts
204+
// and perform privileged syscalls against the host kernel to attach
205+
// probes.
206+
privileged = true
207+
}
208+
}
209+
...
210+
}
211+
...
212+
}
213+
```
214+
215+
> **Note:** `exectrace` requires root privileges and a privileged container
216+
> to attach probes to the kernel. This is a requirement of eBPF.
200217
201218
1. Add the following environment variable to your workspace pod:
202219

203-
```hcl
204-
resource "kubernetes_pod" "main" {
205-
...
206-
spec {
207-
...
208-
env {
209-
name = "CODER_AGENT_SUBSYSTEM"
210-
value = "exectrace"
211-
}
212-
...
213-
}
214-
...
215-
}
216-
```
220+
<!--
221+
If you are updating this section, please also update the example templates
222+
in the exectrace repo.
223+
-->
224+
225+
```hcl
226+
resource "kubernetes_pod" "main" {
227+
...
228+
spec {
229+
...
230+
env {
231+
name = "CODER_AGENT_SUBSYSTEM"
232+
value = "exectrace"
233+
}
234+
...
235+
}
236+
...
237+
}
238+
```
217239

218240
Once you have made these changes, you can push a new version of your template
219241
and workspace process logging will be enabled for all workspaces once they are
@@ -225,7 +247,7 @@ To view the process logs for a specific workspace you can use `kubectl` to print
225247
the logs:
226248

227249
```bash
228-
$ kubectl logs pod-name --container exectrace
250+
kubectl logs pod-name --container exectrace
229251
```
230252

231253
The raw logs will look something like this:
@@ -259,9 +281,10 @@ The raw logs will look something like this:
259281

260282
### View logs in AWS EKS
261283

262-
If you're using AWS' Elastic Kubernetes Service, you can [configure your
263-
cluster][eks-cloudwatch] to send logs to CloudWatch. This allows you to view the
264-
logs for a specific user or workspace.
284+
If you're using AWS' Elastic Kubernetes Service, you can
285+
[configure your cluster](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html)
286+
to send logs to CloudWatch. This allows you to view the logs for a specific user
287+
or workspace.
265288

266289
To view your logs, go to the CloudWatch dashboard (which is available on the
267290
**Log Insights** tab) and run a query similar to the following:

0 commit comments

Comments
 (0)