Skip to content

Commit c426130

Browse files
deansheatherdeansheather
committed
docs: add workspace process logging doc
1 parent 7b35f3b commit c426130

File tree

2 files changed

+294
-0
lines changed

2 files changed

+294
-0
lines changed

‎docs/templates/docker-in-workspaces.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ There are a few ways to run Docker within container-based Coder workspaces.
1313

1414
The [Sysbox](https://github.com/nestybox/sysbox) container runtime allows unprivileged users to run system-level applications, such as Docker, securely from the workspace containers. Sysbox requires a [compatible Linux distribution](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md) to implement these security features. Sysbox can also be used to run systemd inside Coder workspaces. See [Systemd in Docker](#systemd-in-docker).
1515

16+
The Sysbox container runtime is not compatible with our [workspace process logging](./process-logging.md) feature.
17+
1618
### Use Sysbox in Docker-based templates
1719

1820
After [installing Sysbox](https://github.com/nestybox/sysbox#installation) on the Coder host, modify your template to use the sysbox-runc runtime:

‎docs/templates/process-logging.md

Lines changed: 292 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,292 @@
1+
# Workspace Process Logging
2+
3+
The workspace process logging feature allows you to log all system-level
4+
processes executing in the workspace.
5+
6+
> **Note:** This feature is only available on Linux in Kubernetes. There are
7+
> additional requirements outlined further in this document.
8+
>
9+
> This is an Enterprise feature. To learn more about Coder Enterprise, please
10+
> contact sales.
11+
12+
Workspace process logging adds a sidecar container to workspace pods that will
13+
log all processes started in the workspace container (e.g., commands executed in
14+
the terminal or processes created in the background by other processes). You
15+
can view the output from the sidecar or send it to a monitoring stack, such as
16+
CloudWatch, for further analysis or long-term storage.
17+
18+
Please note that these logs are not recorded or captured by the Coder
19+
organization in any way, shape, or form.
20+
21+
## How this works
22+
23+
Coder uses [eBPF](https://ebpf.io/) (which we chose for its minimal performance
24+
impact) to perform in-kernel logging and filtering of all exec system calls
25+
originating from the workspace container.
26+
27+
The core of this feature is also open source and can be found in the
28+
[exectrace](https://github.com/coder/exectrace) repo on GitHub repo. The
29+
enterprise component (in the `enterprise/` directory of the repo) is responsible
30+
for starting the eBPF program with the correct filtering options for the
31+
specific workspace.
32+
33+
## Requirements
34+
35+
The host machine must be running a Linux kernel >= 5.8 with the kernel config
36+
`CONFIG_DEBUG_INFO_BTF=y` enabled.
37+
38+
To check your kernel version, run:
39+
40+
```shell
41+
$ uname -r
42+
```
43+
44+
To validate the required kernel config is enabled, run either of the following
45+
commands on your nodes directly (_not_ from a workspace terminal):
46+
47+
```shell
48+
$ cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF
49+
```
50+
51+
```shell
52+
$ cat "/boot/config-$(uname -r)" | grep CONFIG_DEBUG_INFO_BTF
53+
```
54+
55+
If these requirements are not met, workspaces will fail to start for security
56+
reasons.
57+
58+
Your template must be a Kubernetes template. Workspace process logging is not
59+
compatible with the `sysbox-runc` runtime due to technical limitations, but it
60+
is compatible with our `envbox` template family.
61+
62+
## Example templates
63+
64+
We provide working example templates for Kubernetes, and Kubernetes with
65+
`envbox` (for [Docker support in workspaces](./docker-in-workspaces.md)). You
66+
can view these templates in the
67+
[exectrace repo](https://github.com/coder/exectrace/tree/main/enterprise/templates).
68+
69+
## Configuring custom templates to use workspace process logging
70+
71+
If you have an existing Kubernetes or Kubernetes with `envbox` template that you
72+
would like to add workspace process logging to, follow these steps:
73+
74+
1. Ensure the image used in your template has `curl` installed.
75+
76+
1. Add the following section to your template's `main.tf` file:
77+
78+
```hcl
79+
locals {
80+
# This is the init script for the main workspace container that runs before the
81+
# agent starts to configure workspace process logging.
82+
exectrace_init_script = <<EOT
83+
set -eu
84+
pidns_inum=$(readlink /proc/self/ns/pid | sed 's/[^0-9]//g')
85+
if [ -z "$pidns_inum" ]; then
86+
echo "Could not determine process ID namespace inum"
87+
exit 1
88+
fi
89+
90+
# Before we start the script, does curl exist?
91+
if ! command -v curl >/dev/null 2>&1; then
92+
echo "curl is required to download the Coder binary"
93+
echo "Please install curl to your image and try again"
94+
# 127 is command not found.
95+
exit 127
96+
fi
97+
98+
echo "Sending process ID namespace inum to exectrace sidecar"
99+
rc=0
100+
max_retry=5
101+
counter=0
102+
until [ $counter -ge $max_retry ]; do
103+
set +e
104+
curl \
105+
--fail \
106+
--silent \
107+
--connect-timeout 5 \
108+
-X POST \
109+
-H "Content-Type: text/plain" \
110+
--data "$pidns_inum" \
111+
http://127.0.0.1:56123
112+
rc=$?
113+
set -e
114+
if [ $rc -eq 0 ]; then
115+
break
116+
fi
117+
118+
counter=$((counter+1))
119+
echo "Curl failed with exit code $${rc}, attempt $${counter}/$${max_retry}; Retrying in 3 seconds..."
120+
sleep 3
121+
done
122+
if [ $rc -ne 0 ]; then
123+
echo "Failed to send process ID namespace inum to exectrace sidecar"
124+
exit $rc
125+
fi
126+
127+
EOT
128+
}
129+
```
130+
131+
1. Update the `command` of your workspace container like the following:
132+
133+
```hcl
134+
resource "kubernetes_pod" "main" {
135+
...
136+
spec {
137+
...
138+
container {
139+
...
140+
// NOTE: this command is changed compared to the upstream kubernetes
141+
// template
142+
command = [
143+
"sh",
144+
"-c",
145+
"${local.exectrace_init_script}\n\n${coder_agent.main.init_script}",
146+
]
147+
...
148+
}
149+
...
150+
}
151+
...
152+
}
153+
```
154+
155+
> **Note:** If you are using the `envbox` template, you will need to update
156+
> the third argument to be
157+
> `"${local.exectrace_init_script}\n\nexec /envbox docker"` instead.
158+
159+
1. Add the following container to your workspace pod spec.
160+
161+
```hcl
162+
resource "kubernetes_pod" "main" {
163+
...
164+
spec {
165+
...
166+
// NOTE: this container is added compared to the upstream kubernetes
167+
// template
168+
container {
169+
name = "exectrace"
170+
image = "ghcr.io/coder/exectrace:latest"
171+
image_pull_policy = "Always"
172+
command = [
173+
"/opt/exectrace",
174+
"--init-address", "127.0.0.1:56123",
175+
"--label", "workspace_id=${data.coder_workspace.me.id}",
176+
"--label", "workspace_name=${data.coder_workspace.me.name}",
177+
"--label", "user_id=${data.coder_workspace.me.owner_id}",
178+
"--label", "username=${data.coder_workspace.me.owner}",
179+
"--label", "user_email=${data.coder_workspace.me.owner_email}",
180+
]
181+
security_context {
182+
// exectrace must be started as root so it can attach probes into the
183+
// kernel to record process events with high throughput.
184+
run_as_user = "0"
185+
run_as_group = "0"
186+
// exectrace requires a privileged container so it can control mounts
187+
// and perform privileged syscalls against the host kernel to attach
188+
// probes.
189+
privileged = true
190+
}
191+
}
192+
...
193+
}
194+
...
195+
}
196+
```
197+
198+
> **Note:** `exectrace` requires root privileges and a privileged container
199+
> to attach probes to the kernel. This is a requirement of eBPF.
200+
201+
1. Add the following environment variable to your workspace pod:
202+
203+
```hcl
204+
resource "kubernetes_pod" "main" {
205+
...
206+
spec {
207+
...
208+
env {
209+
name = "CODER_AGENT_SUBSYSTEM"
210+
value = "exectrace"
211+
}
212+
...
213+
}
214+
...
215+
}
216+
```
217+
218+
Once you have made these changes, you can push a new version of your template
219+
and workspace process logging will be enabled for all workspaces once they are
220+
restarted.
221+
222+
## Viewing workspace process logs
223+
224+
To view the process logs for a specific workspace you can use `kubectl` to print
225+
the logs:
226+
227+
```bash
228+
$ kubectl logs pod-name --container exectrace
229+
```
230+
231+
The raw logs will look something like this:
232+
233+
```json
234+
{
235+
"ts": "2022-02-28T20:29:38.038452202Z",
236+
"level": "INFO",
237+
"msg": "exec",
238+
"fields": {
239+
"labels": {
240+
"user_email": "jessie@coder.com",
241+
"user_id": "5e876e9a-121663f01ebd1522060d5270",
242+
"username": "jessie",
243+
"workspace_id": "621d2e52-a6987ef6c56210058ee2593c",
244+
"workspace_name": "main"
245+
},
246+
"cmdline": "uname -a",
247+
"event": {
248+
"filename": "/usr/bin/uname",
249+
"argv": ["uname", "-a"],
250+
"truncated": false,
251+
"pid": 920684,
252+
"uid": 101000,
253+
"gid": 101000,
254+
"comm": "bash"
255+
}
256+
}
257+
}
258+
```
259+
260+
### View logs in AWS EKS
261+
262+
If you're using AWS' Elastic Kubernetes Service, you can [configure your
263+
cluster][eks-cloudwatch] to send logs to CloudWatch. This allows you to view the
264+
logs for a specific user or workspace.
265+
266+
To view your logs, go to the CloudWatch dashboard (which is available on the
267+
**Log Insights** tab) and run a query similar to the following:
268+
269+
```text
270+
fields @timestamp, log_processed.fields.cmdline
271+
| sort @timestamp asc
272+
| filter kubernetes.container_name="exectrace"
273+
| filter log_processed.fields.labels.username="zac"
274+
| filter log_processed.fields.labels.workspace_name="code"
275+
```
276+
277+
## Usage considerations
278+
279+
- The sidecar attached to each workspace is a privileged container, so you may
280+
need to review your organization's security policies before enabling this
281+
feature. Enabling workspace process logging does _not_ grant extra privileges
282+
to the workspace container itself, however.
283+
- `exectrace` will log processes from nested Docker containers (including deeply
284+
nested containers) correctly, but Coder does not distinguish between processes
285+
started in the workspace and processes started in a child container in the
286+
logs.
287+
- With `envbox` workspaces, this feature will detect and log startup processes
288+
begun in the outer container (including container initialization processes).
289+
- Because this feature logs **all** processes in the workspace, high levels of
290+
usage (e.g., during a `make` run) will result in an abundance of output in the
291+
sidecar container. Depending on how your Kubernetes cluster is configured, you
292+
may incur extra charges from your cloud provider to store the additional logs.

0 commit comments

Comments
 (0)