Skip to content

add docs: "docker in docker" and "systemd in docker" #4051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 19, 2022
Merged

add docs: "docker in docker" and "systemd in docker" #4051

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5bf07af
docs: docker in docker
bpmct Sep 13, 2022
c523571
add k8s
bpmct Sep 14, 2022
c4bdc5f
wrap up
bpmct Sep 14, 2022
f3797ac
add privileged sidecar container method for docker
bpmct Sep 19, 2022
a7032b4
consolidate systemd in docker docs
bpmct Sep 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/images/icons/system.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 17 additions & 1 deletion docs/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
{
"versions": ["main", "v0.8.1", "v0.7.12"],
"versions": [
"main",
"v0.8.1",
"v0.7.12"
],
"routes": [
{
"title": "About",
Expand Down Expand Up @@ -84,6 +88,18 @@
"description": "Learn how to expose resource data to users",
"path": "./templates/resource-metadata.md",
"icon_path": "./images/icons/table-rows.svg"
},
{
"title": "Docker in Docker",
"description": "Use docker inside containerized templates",
"path": "./templates/docker-in-docker.md",
"icon_path": "./images/icons/docker.svg"
},
{
"title": "Systemd in Docker",
"description": "Use systemd inside containerized templates",
"path": "./templates/systemd-in-docker.md",
"icon_path": "./images/icons/system.svg"
}
]
},
Expand Down
170 changes: 170 additions & 0 deletions docs/templates/docker-in-docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
There are a few ways to run Docker within container-based Coder workspaces.

## Sysbox runtime (recommended)

The [Sysbox](https://github.com/nestybox/sysbox) container runtime allows unprivileged users to run system-level applications, such as Docker, securely from the workspace containers. Sysbox requires a [compatible Linux distribution](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md) to implement these security features.

> Sysbox can also be used to run systemd inside Coder workspaces. See [Systemd in Docker](./systemd-in-docker.md).

### Use Sysbox in Docker-based templates:

After [installing Sysbox](https://github.com/nestybox/sysbox#installation) on the Coder host, modify your template to use the sysbox-runc runtime:

```hcl
resource "docker_container" "workspace" {
# ...
name = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
image = "codercom/enterprise-base:ubuntu"
env = ["CODER_AGENT_TOKEN=${coder_agent.main.token}"]
command = ["sh", "-c", coder_agent.main.init_script]
# Use the Sysbox container runtime (required)
runtime = "sysbox-runc"
}

resource "coder_agent" "main" {
arch = data.coder_provisioner.me.arch
os = "linux"
startup_script = <<EOF
#!/bin/sh

# Start Docker
sudo dockerd &

# ...
EOF
}
```

### Use Sysbox in Kubernetes-based templates:

After [installing Sysbox on Kubernetes](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md), modify your template to use the sysbox-runc RuntimeClass.

> Currently, the official [Kubernetes Terraform Provider](https://registry.terraform.io/providers/hashicorp/kubernetes/latest) does not support specifying a custom RuntimeClass. [mingfang/k8s](https://registry.terraform.io/providers/mingfang/k8s), a third-party provider, can be used instead.

```hcl
resource "coder_agent" "main" {
os = "linux"
arch = "amd64"
dir = "/home/coder"
startup_script = <<EOF
#!/bin/sh

# Start Docker
sudo dockerd &

# ...
EOF
}

resource "k8s_core_v1_pod" "dev" {
count = data.coder_workspace.me.start_count
metadata {
name = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
namespace = var.workspaces_namespace
annotations = {
"io.kubernetes.cri-o.userns-mode" = "auto:size=65536"
}
}

# Use the Sysbox container runtime (required)
runtime_class_name = "sysbox-runc

spec {
security_context {
run_asuser = 1000
fsgroup = 1000
}
containers {
name = "dev"
env {
name = "CODER_AGENT_TOKEN"
value = coder_agent.main.token
}
image = "codercom/enterprise-base:ubuntu"
command = ["sh", "-c", coder_agent.main.init_script]
}
}
}
```

> Sysbox CE (Community Edition) supports a maximum of 16 pods (workspaces) per node on Kubernetes. See the [Sysbox documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#limitations) for more details.

## Privileged sidecar container (Kubernetes)

While less secure, you can attach a [privileged container](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) to your templates. This may come in handy if your nodes cannot run Sysbox.

```hcl
resource "coder_agent" "main" {
os = "linux"
arch = "amd64"
}

resource "kubernetes_pod" "main" {
count = data.coder_workspace.me.start_count
metadata {
name = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
namespace = var.namespace
}
spec {
# Run a privileged dind (Docker in Docker) container
container {
name = "docker-sidecar"
image = "docker:dind"
security_context {
privileged = true
}
command = ["dockerd", "-H", "tcp://127.0.0.1:2375"]
}
container {
name = "dev"
image = "codercom/enterprise-base:ubuntu"
command = ["sh", "-c", coder_agent.main.init_script]
security_context {
run_as_user = "1000"
}
env {
name = "CODER_AGENT_TOKEN"
value = coder_agent.main.token
}
# Use the Docker daemon in the "docker-sidecar" container
env {
name = "DOCKER_HOST"
value = "localhost:2375"
}
}
}
}

```

## Shared Docker socket (Docker)

While less secure, Docker-based templates can share the host's Docker socket.

````hcl
resource "coder_agent" "main" {
arch = data.coder_provisioner.me.arch
os = "linux"
startup_script = <<EOF
#!/bin/sh

# Give the internal "coder" user permission
# to use the Docker socket
sudo chmod 666 /var/run/socker.sock

EOF
}

resource "docker_container" "workspace" {
count = data.coder_workspace.me.start_count
image = "codercom/enterprise-base:ubuntu"
name = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
command = ["sh", "-c", coder_agent.main.init_script]
env = ["CODER_AGENT_TOKEN=${coder_agent.main.token}"]
volumes {
container_path = "/var/run/docker.sock"
host_path = "/var/run/docker.sock"
}
}
```hcl
````
115 changes: 115 additions & 0 deletions docs/templates/systemd-in-docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
The [Sysbox](https://github.com/nestybox/sysbox) container runtime allows unprivileged users to run system-level applications, such as Systemd, securely from the workspace containers. Sysbox requires a [compatible Linux distribution](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md) to implement these security features.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unfortunate how much of this doc is duplicated with the docker-in-docker.md. Perhaps we can make the meat of it a suffix to the docker-in-docker.md doc?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense - I added the systemd stuff to the end of docker-in-docker.md and removed systemd-in-docker.md


## Docker

After [installing Sysbox](https://github.com/nestybox/sysbox#installation) on the Coder host, modify your template to use the sysbox-runc runtime and start systemd:

```hcl
resource "docker_container" "workspace" {
image = "codercom/enterprise-base:ubuntu"
name = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"

# Use Sysbox container runtime (required)
runtime = "sysbox-runc"
# Run as root in order to start systemd (required)
user = "0:0"

# Start systemd and the Coder agent
command = ["sh", "-c", <<EOF
# Start the Coder agent as the "coder" user
# once systemd has started up
sudo -u coder --preserve-env=CODER_AGENT_TOKEN /bin/bash -- <<-' EOT' &
while [[ ! $(systemctl is-system-running) =~ ^(running|degraded) ]]
do
echo "Waiting for system to start... $(systemctl is-system-running)"
sleep 2
done
${coder_agent.main.init_script}
EOT

exec /sbin/init
EOF
,
]
env = ["CODER_AGENT_TOKEN=${coder_agent.main.token}"]
}

resource "coder_agent" "main" {
arch = data.coder_provisioner.me.arch
os = "linux"
}
```

## Kubernetes

After [installing Sysbox on Kubernetes](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md), modify your template to use the sysbox-runc RuntimeClass.

> Currently, the official [Kubernetes Terraform Provider](https://registry.terraform.io/providers/hashicorp/kubernetes/latest) does not support specifying a custom RuntimeClass. [mingfang/k8s](https://registry.terraform.io/providers/mingfang/k8s), a third-party provider, can be used instead.

```hcl
terraform {
required_providers {
coder = {
source = "coder/coder"
}
k8s = {
source = "mingfang/k8s"
}
}
}


resource "coder_agent" "main" {
os = "linux"
arch = "amd64"
dir = "/home/coder"
}

resource "k8s_core_v1_pod" "dev" {
count = data.coder_workspace.me.start_count
metadata {
name = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
namespace = var.workspaces_namespace
annotations = {
"io.kubernetes.cri-o.userns-mode" = "auto:size=65536"
}
}


spec {

# Use Sysbox container runtime (required)
runtime_class_name = "sysbox-runc"

# Run as root in order to start systemd (required)
security_context {
run_asuser = 0
fsgroup = 0
}

containers {
name = "dev"
env {
name = "CODER_AGENT_TOKEN"
value = coder_agent.main.token
}
image = "codercom/enterprise-base:ubuntu"
command = ["sh", "-c", <<EOF
# Start the Coder agent as the "coder" user
# once systemd has started up
sudo -u coder --preserve-env=CODER_AGENT_TOKEN /bin/bash -- <<-' EOT' &
while [[ ! $(systemctl is-system-running) =~ ^(running|degraded) ]]
do
echo "Waiting for system to start... $(systemctl is-system-running)"
sleep 2
done
${coder_agent.main.init_script}
EOT

exec /sbin/init
EOF
]
}
}
}
```