Skip to content

Commit 88bb9fb

Browse files
matifalimatifali
committed
Add process logging documentation for workspaces
Include instructions for enabling and configuring workspace process logging on Kubernetes to help users monitor system-level processes.
1 parent f709b8e commit 88bb9fb

File tree

2 files changed

+321
-0
lines changed

2 files changed

+321
-0
lines changed

‎docs/admin/templates/extending-templates/process-logging.md

+315
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
# Workspace Process Logging
2+
3+
The workspace process logging feature allows you to log all system-level
4+
processes executing in the workspace.
5+
6+
> **Note:** This feature is only available on Linux in Kubernetes. There are
7+
> additional requirements outlined further in this document.
8+
9+
Workspace process logging adds a sidecar container to workspace pods that will
10+
log all processes started in the workspace container (e.g., commands executed in
11+
the terminal or processes created in the background by other processes).
12+
Processes launched inside containers or nested containers within the workspace
13+
are also logged. You can view the output from the sidecar or send it to a
14+
monitoring stack, such as CloudWatch, for further analysis or long-term storage.
15+
16+
Please note that these logs are not recorded or captured by the Coder
17+
organization in any way, shape, or form.
18+
19+
> This is an [Premium or Enterprise](https://coder.com/pricing) feature. To
20+
> learn more about Coder Enterprise, please
21+
> [contact sales](https://coder.com/contact).
22+
23+
## How this works
24+
25+
Coder uses [eBPF](https://ebpf.io/) (which we chose for its minimal performance
26+
impact) to perform in-kernel logging and filtering of all exec system calls
27+
originating from the workspace container.
28+
29+
The core of this feature is also open source and can be found in the
30+
[exectrace](https://github.com/coder/exectrace) GitHub repo. The enterprise
31+
component (in the `enterprise/` directory of the repo) is responsible for
32+
starting the eBPF program with the correct filtering options for the specific
33+
workspace.
34+
35+
## Requirements
36+
37+
The host machine must be running a Linux kernel >= 5.8 with the kernel config
38+
`CONFIG_DEBUG_INFO_BTF=y` enabled.
39+
40+
To check your kernel version, run:
41+
42+
```shell
43+
uname -r
44+
```
45+
46+
To validate the required kernel config is enabled, run either of the following
47+
commands on your nodes directly (_not_ from a workspace terminal):
48+
49+
```shell
50+
cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF
51+
```
52+
53+
```shell
54+
cat "/boot/config-$(uname -r)" | grep CONFIG_DEBUG_INFO_BTF
55+
```
56+
57+
If these requirements are not met, workspaces will fail to start for security
58+
reasons.
59+
60+
Your template must be a Kubernetes template. Workspace process logging is not
61+
compatible with the `sysbox-runc` runtime due to technical limitations, but it
62+
is compatible with our `envbox` template family.
63+
64+
## Example templates
65+
66+
We provide working example templates for Kubernetes, and Kubernetes with
67+
`envbox` (for [Docker support in workspaces](./docker-in-workspaces.md)). You
68+
can view these templates in the
69+
[exectrace repo](https://github.com/coder/exectrace/tree/main/enterprise/templates).
70+
71+
## Configuring custom templates to use workspace process logging
72+
73+
If you have an existing Kubernetes or Kubernetes with `envbox` template that you
74+
would like to add workspace process logging to, follow these steps:
75+
76+
1. Ensure the image used in your template has `curl` installed.
77+
78+
1. Add the following section to your template's `main.tf` file:
79+
80+
<!--
81+
If you are updating this section, please also update the example templates
82+
in the exectrace repo.
83+
-->
84+
85+
```hcl
86+
locals {
87+
# This is the init script for the main workspace container that runs before the
88+
# agent starts to configure workspace process logging.
89+
exectrace_init_script = <<EOT
90+
set -eu
91+
pidns_inum=$(readlink /proc/self/ns/pid | sed 's/[^0-9]//g')
92+
if [ -z "$pidns_inum" ]; then
93+
echo "Could not determine process ID namespace inum"
94+
exit 1
95+
fi
96+
97+
# Before we start the script, does curl exist?
98+
if ! command -v curl >/dev/null 2>&1; then
99+
echo "curl is required to download the Coder binary"
100+
echo "Please install curl to your image and try again"
101+
# 127 is command not found.
102+
exit 127
103+
fi
104+
105+
echo "Sending process ID namespace inum to exectrace sidecar"
106+
rc=0
107+
max_retry=5
108+
counter=0
109+
until [ $counter -ge $max_retry ]; do
110+
set +e
111+
curl \
112+
--fail \
113+
--silent \
114+
--connect-timeout 5 \
115+
-X POST \
116+
-H "Content-Type: text/plain" \
117+
--data "$pidns_inum" \
118+
http://127.0.0.1:56123
119+
rc=$?
120+
set -e
121+
if [ $rc -eq 0 ]; then
122+
break
123+
fi
124+
125+
counter=$((counter+1))
126+
echo "Curl failed with exit code $${rc}, attempt $${counter}/$${max_retry}; Retrying in 3 seconds..."
127+
sleep 3
128+
done
129+
if [ $rc -ne 0 ]; then
130+
echo "Failed to send process ID namespace inum to exectrace sidecar"
131+
exit $rc
132+
fi
133+
134+
EOT
135+
}
136+
```
137+
138+
1. Update the `command` of your workspace container like the following:
139+
140+
<!--
141+
If you are updating this section, please also update the example templates
142+
in the exectrace repo.
143+
-->
144+
145+
```hcl
146+
resource "kubernetes_pod" "main" {
147+
...
148+
spec {
149+
...
150+
container {
151+
...
152+
// NOTE: this command is changed compared to the upstream kubernetes
153+
// template
154+
command = [
155+
"sh",
156+
"-c",
157+
"${local.exectrace_init_script}\n\n${coder_agent.main.init_script}",
158+
]
159+
...
160+
}
161+
...
162+
}
163+
...
164+
}
165+
```
166+
167+
> **Note:** If you are using the `envbox` template, you will need to update
168+
> the third argument to be
169+
> `"${local.exectrace_init_script}\n\nexec /envbox docker"` instead.
170+
171+
1. Add the following container to your workspace pod spec.
172+
173+
<!--
174+
If you are updating this section, please also update the example templates
175+
in the exectrace repo.
176+
-->
177+
178+
```hcl
179+
resource "kubernetes_pod" "main" {
180+
...
181+
spec {
182+
...
183+
// NOTE: this container is added compared to the upstream kubernetes
184+
// template
185+
container {
186+
name = "exectrace"
187+
image = "ghcr.io/coder/exectrace:latest"
188+
image_pull_policy = "Always"
189+
command = [
190+
"/opt/exectrace",
191+
"--init-address", "127.0.0.1:56123",
192+
"--label", "workspace_id=${data.coder_workspace.me.id}",
193+
"--label", "workspace_name=${data.coder_workspace.me.name}",
194+
"--label", "user_id=${data.coder_workspace_owner.me.id}",
195+
"--label", "username=${data.coder_workspace_owner.me.name}",
196+
"--label", "user_email=${data.coder_workspace_owner.me.email}",
197+
]
198+
security_context {
199+
// exectrace must be started as root so it can attach probes into the
200+
// kernel to record process events with high throughput.
201+
run_as_user = "0"
202+
run_as_group = "0"
203+
// exectrace requires a privileged container so it can control mounts
204+
// and perform privileged syscalls against the host kernel to attach
205+
// probes.
206+
privileged = true
207+
}
208+
}
209+
...
210+
}
211+
...
212+
}
213+
```
214+
215+
> **Note:** `exectrace` requires root privileges and a privileged container
216+
> to attach probes to the kernel. This is a requirement of eBPF.
217+
218+
1. Add the following environment variable to your workspace pod:
219+
220+
<!--
221+
If you are updating this section, please also update the example templates
222+
in the exectrace repo.
223+
-->
224+
225+
```hcl
226+
resource "kubernetes_pod" "main" {
227+
...
228+
spec {
229+
...
230+
env {
231+
name = "CODER_AGENT_SUBSYSTEM"
232+
value = "exectrace"
233+
}
234+
...
235+
}
236+
...
237+
}
238+
```
239+
240+
Once you have made these changes, you can push a new version of your template
241+
and workspace process logging will be enabled for all workspaces once they are
242+
restarted.
243+
244+
## Viewing workspace process logs
245+
246+
To view the process logs for a specific workspace you can use `kubectl` to print
247+
the logs:
248+
249+
```bash
250+
kubectl logs pod-name --container exectrace
251+
```
252+
253+
The raw logs will look something like this:
254+
255+
```json
256+
{
257+
"ts": "2022-02-28T20:29:38.038452202Z",
258+
"level": "INFO",
259+
"msg": "exec",
260+
"fields": {
261+
"labels": {
262+
"user_email": "jessie@coder.com",
263+
"user_id": "5e876e9a-121663f01ebd1522060d5270",
264+
"username": "jessie",
265+
"workspace_id": "621d2e52-a6987ef6c56210058ee2593c",
266+
"workspace_name": "main"
267+
},
268+
"cmdline": "uname -a",
269+
"event": {
270+
"filename": "/usr/bin/uname",
271+
"argv": ["uname", "-a"],
272+
"truncated": false,
273+
"pid": 920684,
274+
"uid": 101000,
275+
"gid": 101000,
276+
"comm": "bash"
277+
}
278+
}
279+
}
280+
```
281+
282+
### View logs in AWS EKS
283+
284+
If you're using AWS' Elastic Kubernetes Service, you can
285+
[configure your cluster](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html)
286+
to send logs to CloudWatch. This allows you to view the logs for a specific user
287+
or workspace.
288+
289+
To view your logs, go to the CloudWatch dashboard (which is available on the
290+
**Log Insights** tab) and run a query similar to the following:
291+
292+
```text
293+
fields @timestamp, log_processed.fields.cmdline
294+
| sort @timestamp asc
295+
| filter kubernetes.container_name="exectrace"
296+
| filter log_processed.fields.labels.username="zac"
297+
| filter log_processed.fields.labels.workspace_name="code"
298+
```
299+
300+
## Usage considerations
301+
302+
- The sidecar attached to each workspace is a privileged container, so you may
303+
need to review your organization's security policies before enabling this
304+
feature. Enabling workspace process logging does _not_ grant extra privileges
305+
to the workspace container itself, however.
306+
- `exectrace` will log processes from nested Docker containers (including deeply
307+
nested containers) correctly, but Coder does not distinguish between processes
308+
started in the workspace and processes started in a child container in the
309+
logs.
310+
- With `envbox` workspaces, this feature will detect and log startup processes
311+
begun in the outer container (including container initialization processes).
312+
- Because this feature logs **all** processes in the workspace, high levels of
313+
usage (e.g., during a `make` run) will result in an abundance of output in the
314+
sidecar container. Depending on how your Kubernetes cluster is configured, you
315+
may incur extra charges from your cloud provider to store the additional logs.

‎docs/manifest.json

+6
Original file line numberDiff line numberDiff line change
@@ -351,6 +351,12 @@
351351
"title": "Provider Authentication",
352352
"description": "Authenticate with provider APIs to provision workspaces",
353353
"path": "./admin/templates/extending-templates/provider-authentication.md"
354+
},
355+
{
356+
"title": "Process Logging",
357+
"description": "Log workspace processes",
358+
"path": "./admin/templates/extending-templates/process-logging.md",
359+
"state": ["enterprise", "premium"]
354360
}
355361
]
356362
},

0 commit comments

Comments
 (0)