coder · deansheather · Aug 18, 2023 · Aug 9, 2023 · Aug 18, 2023 · bpmct
diff --git a/docs/templates/docker-in-workspaces.md b/docs/templates/docker-in-workspaces.md
@@ -13,6 +13,8 @@ There are a few ways to run Docker within container-based Coder workspaces.
 
 The [Sysbox](https://github.com/nestybox/sysbox) container runtime allows unprivileged users to run system-level applications, such as Docker, securely from the workspace containers. Sysbox requires a [compatible Linux distribution](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md) to implement these security features. Sysbox can also be used to run systemd inside Coder workspaces. See [Systemd in Docker](#systemd-in-docker).
 
+The Sysbox container runtime is not compatible with our [workspace process logging](./process-logging.md) feature. Envbox is compatible with process logging, however.
+
 ### Use Sysbox in Docker-based templates
 
 After [installing Sysbox](https://github.com/nestybox/sysbox#installation) on the Coder host, modify your template to use the sysbox-runc runtime:

diff --git a/docs/templates/process-logging.md b/docs/templates/process-logging.md
@@ -0,0 +1,315 @@
+# Workspace Process Logging
+
+The workspace process logging feature allows you to log all system-level
+processes executing in the workspace.
+
+> **Note:** This feature is only available on Linux in Kubernetes. There are
+> additional requirements outlined further in this document.
+>
+> This is an [Enterprise](https://coder.com/docs/v2/latest/enterprise) feature.
+> To learn more about Coder Enterprise, please
+> [contact sales](https://coder.com/contact).
+
+Workspace process logging adds a sidecar container to workspace pods that will
+log all processes started in the workspace container (e.g., commands executed in
+the terminal or processes created in the background by other processes).
+Processes launched inside containers or nested containers within the workspace
+are also logged. You can view the output from the sidecar or send it to a
+monitoring stack, such as CloudWatch, for further analysis or long-term storage.
+
+Please note that these logs are not recorded or captured by the Coder
+organization in any way, shape, or form.
+
+## How this works
+
+Coder uses [eBPF](https://ebpf.io/) (which we chose for its minimal performance
+impact) to perform in-kernel logging and filtering of all exec system calls
+originating from the workspace container.
+
+The core of this feature is also open source and can be found in the
+[exectrace](https://github.com/coder/exectrace) GitHub repo. The enterprise
+component (in the `enterprise/` directory of the repo) is responsible for
+starting the eBPF program with the correct filtering options for the specific
+workspace.
+
+## Requirements
+
+The host machine must be running a Linux kernel >= 5.8 with the kernel config
+`CONFIG_DEBUG_INFO_BTF=y` enabled.
+
+To check your kernel version, run:
+
+```shell
+uname -r
+```
+
+To validate the required kernel config is enabled, run either of the following
+commands on your nodes directly (_not_ from a workspace terminal):
+
+```shell
+cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF
+```
+
+```shell
+cat "/boot/config-$(uname -r)" | grep CONFIG_DEBUG_INFO_BTF
+```
+
+If these requirements are not met, workspaces will fail to start for security
+reasons.
+
+Your template must be a Kubernetes template. Workspace process logging is not
+compatible with the `sysbox-runc` runtime due to technical limitations, but it
+is compatible with our `envbox` template family.
+
+## Example templates
+
+We provide working example templates for Kubernetes, and Kubernetes with
+`envbox` (for [Docker support in workspaces](./docker-in-workspaces.md)). You
+can view these templates in the
+[exectrace repo](https://github.com/coder/exectrace/tree/main/enterprise/templates).
+
+## Configuring custom templates to use workspace process logging
+
+If you have an existing Kubernetes or Kubernetes with `envbox` template that you
+would like to add workspace process logging to, follow these steps:
+
+1. Ensure the image used in your template has `curl` installed.
+
+1. Add the following section to your template's `main.tf` file:
+
+   <!--
+     If you are updating this section, please also update the example templates
+     in the exectrace repo.
+   -->
+
+   ```hcl
+   locals {
+     # This is the init script for the main workspace container that runs before the
+     # agent starts to configure workspace process logging.
+     exectrace_init_script = <<EOT
+       set -eu
+       pidns_inum=$(readlink /proc/self/ns/pid | sed 's/[^0-9]//g')
+       if [ -z "$pidns_inum" ]; then
+         echo "Could not determine process ID namespace inum"
+         exit 1
+       fi
+
+       # Before we start the script, does curl exist?
+       if ! command -v curl >/dev/null 2>&1; then
+         echo "curl is required to download the Coder binary"
+         echo "Please install curl to your image and try again"
+         # 127 is command not found.
+         exit 127
+       fi
+
+       echo "Sending process ID namespace inum to exectrace sidecar"
+       rc=0
+       max_retry=5
+       counter=0
+       until [ $counter -ge $max_retry ]; do
+         set +e
+         curl \
+           --fail \
+           --silent \
+           --connect-timeout 5 \
+           -X POST \
+           -H "Content-Type: text/plain" \
+           --data "$pidns_inum" \
+           http://127.0.0.1:56123
+         rc=$?
+         set -e
+         if [ $rc -eq 0 ]; then
+           break
+         fi
+
+         counter=$((counter+1))
+         echo "Curl failed with exit code $${rc}, attempt $${counter}/$${max_retry}; Retrying in 3 seconds..."
+         sleep 3
+       done
+       if [ $rc -ne 0 ]; then
+         echo "Failed to send process ID namespace inum to exectrace sidecar"
+         exit $rc
+       fi
+
+     EOT
+   }
+   ```
+
+1. Update the `command` of your workspace container like the following:
+
+   <!--
+     If you are updating this section, please also update the example templates
+     in the exectrace repo.
+   -->
+
+   ```hcl
+   resource "kubernetes_pod" "main" {
+     ...
+     spec {
+       ...
+       container {
+         ...
+         // NOTE: this command is changed compared to the upstream kubernetes
+         // template
+         command = [
+           "sh",
+           "-c",
+           "${local.exectrace_init_script}\n\n${coder_agent.main.init_script}",
+         ]
+         ...
+       }
+       ...
+     }
+     ...
+   }
+   ```
+
+   > **Note:** If you are using the `envbox` template, you will need to update
+   > the third argument to be
+   > `"${local.exectrace_init_script}\n\nexec /envbox docker"` instead.
+
+1. Add the following container to your workspace pod spec.
+
+   <!--
+     If you are updating this section, please also update the example templates
+     in the exectrace repo.
+   -->
+
+   ```hcl
+   resource "kubernetes_pod" "main" {
+     ...
+     spec {
+       ...
+       // NOTE: this container is added compared to the upstream kubernetes
+       // template
+       container {
+         name              = "exectrace"
+         image             = "ghcr.io/coder/exectrace:latest"
+         image_pull_policy = "Always"
+         command = [
+           "/opt/exectrace",
+           "--init-address", "127.0.0.1:56123",
+           "--label", "workspace_id=${data.coder_workspace.me.id}",
+           "--label", "workspace_name=${data.coder_workspace.me.name}",
+           "--label", "user_id=${data.coder_workspace.me.owner_id}",
+           "--label", "username=${data.coder_workspace.me.owner}",
+           "--label", "user_email=${data.coder_workspace.me.owner_email}",
+         ]
+         security_context {
+           // exectrace must be started as root so it can attach probes into the
+           // kernel to record process events with high throughput.
+           run_as_user  = "0"
+           run_as_group = "0"
+           // exectrace requires a privileged container so it can control mounts
+           // and perform privileged syscalls against the host kernel to attach
+           // probes.
+           privileged = true
+         }
+       }
+       ...
+     }
+     ...
+   }
+   ```
+
+   > **Note:** `exectrace` requires root privileges and a privileged container
+   > to attach probes to the kernel. This is a requirement of eBPF.
+
+1. Add the following environment variable to your workspace pod:
+
+   <!--
+     If you are updating this section, please also update the example templates
+     in the exectrace repo.
+   -->
+
+   ```hcl
+   resource "kubernetes_pod" "main" {
+     ...
+     spec {
+       ...
+       env {
+         name = "CODER_AGENT_SUBSYSTEM"
+         value = "exectrace"
+       }
+       ...
+     }
+     ...
+   }
+   ```
+
+Once you have made these changes, you can push a new version of your template
+and workspace process logging will be enabled for all workspaces once they are
+restarted.
+
+## Viewing workspace process logs
+
+To view the process logs for a specific workspace you can use `kubectl` to print
+the logs:
+
+```bash
+kubectl logs pod-name --container exectrace
+```
+
+The raw logs will look something like this:
+
+```json
+{
+  "ts": "2022-02-28T20:29:38.038452202Z",
+  "level": "INFO",
+  "msg": "exec",
+  "fields": {
+    "labels": {
+      "user_email": "jessie@coder.com",
+      "user_id": "5e876e9a-121663f01ebd1522060d5270",
+      "username": "jessie",
+      "workspace_id": "621d2e52-a6987ef6c56210058ee2593c",
+      "workspace_name": "main"
+    },
+    "cmdline": "uname -a",
+    "event": {
+      "filename": "/usr/bin/uname",
+      "argv": ["uname", "-a"],
+      "truncated": false,
+      "pid": 920684,
+      "uid": 101000,
+      "gid": 101000,
+      "comm": "bash"
+    }
+  }
+}
+```
+
+### View logs in AWS EKS
+
+If you're using AWS' Elastic Kubernetes Service, you can
+[configure your cluster](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html)
+to send logs to CloudWatch. This allows you to view the logs for a specific user
+or workspace.
+
+To view your logs, go to the CloudWatch dashboard (which is available on the
+**Log Insights** tab) and run a query similar to the following:
+
+```text
+fields @timestamp, log_processed.fields.cmdline
+| sort @timestamp asc
+| filter kubernetes.container_name="exectrace"
+| filter log_processed.fields.labels.username="zac"
+| filter log_processed.fields.labels.workspace_name="code"
+```
+
+## Usage considerations
+
+- The sidecar attached to each workspace is a privileged container, so you may
+  need to review your organization's security policies before enabling this
+  feature. Enabling workspace process logging does _not_ grant extra privileges
+  to the workspace container itself, however.
+- `exectrace` will log processes from nested Docker containers (including deeply
+  nested containers) correctly, but Coder does not distinguish between processes
+  started in the workspace and processes started in a child container in the
+  logs.
+- With `envbox` workspaces, this feature will detect and log startup processes
+  begun in the outer container (including container initialization processes).
+- Because this feature logs **all** processes in the workspace, high levels of
+  usage (e.g., during a `make` run) will result in an abundance of output in the
+  sidecar container. Depending on how your Kubernetes cluster is configured, you
+  may incur extra charges from your cloud provider to store the additional logs.