Skip to content

Commit f1763f2

Browse files
sreyasreya
authored
chore: add envbox documentation (#7198)
1 parent 87b7537 commit f1763f2

File tree

3 files changed

+397
-5
lines changed

3 files changed

+397
-5
lines changed

‎docs/templates/docker-in-workspaces.md

+63-5
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,12 @@
22

33
There are a few ways to run Docker within container-based Coder workspaces.
44

5-
| Method | Description | Limitations |
6-
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
7-
| [Sysbox container runtime](#sysbox-container-runtime) | Install the sysbox runtime on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
8-
| [Rootless Podman](#rootless-podman) | Run podman inside Coder workspaces. Does not require a custom runtime or privileged containers. Works with GKE, EKS, AKS, RKE, OpenShift | Requires smarter-device-manager for FUSE mounts. [See all](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman) |
9-
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privileged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |
5+
| Method | Description | Limitations |
6+
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
7+
| [Sysbox container runtime](#sysbox-container-runtime) | Install the sysbox runtime on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
8+
| [Envbox](#envbox) | A container image with all the packages necessary to run an inner sysbox container. Removes the need to setup sysbox-runc on your nodes. Works with GKE, EKS, AKS. | Requires running the outer container as privileged (the inner container that acts as the workspace is locked down). Requires compatible [nodes](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md#sysbox-distro-compatibility). |
9+
| [Rootless Podman](#rootless-podman) | Run podman inside Coder workspaces. Does not require a custom runtime or privileged containers. Works with GKE, EKS, AKS, RKE, OpenShift | Requires smarter-device-manager for FUSE mounts. [See all](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman) |
10+
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privileged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |
1011

1112
## Sysbox container runtime
1213

@@ -110,6 +111,63 @@ resource "kubernetes_pod" "dev" {
110111

111112
> Sysbox CE (Community Edition) supports a maximum of 16 pods (workspaces) per node on Kubernetes. See the [Sysbox documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#limitations) for more details.
112113
114+
## Envbox
115+
116+
[Envbox](https://github.com/coder/envbox) is an image developed and maintained by Coder that bundles the sysbox runtime. It works
117+
by starting an outer container that manages the various sysbox daemons and spawns an unprivileged
118+
inner container that acts as the user's workspace. The inner container is able to run system-level
119+
software similar to a regular virtual machine (e.g. `systemd`, `dockerd`, etc). Envbox offers the
120+
following benefits over running sysbox directly on the nodes:
121+
122+
- No custom runtime installation or management on your Kubernetes nodes.
123+
- No limit to the number of pods that run envbox.
124+
125+
Some drawbacks include:
126+
127+
- The outer container must be run as privileged
128+
- Note: the inner container is _not_ privileged. For more information on the security of sysbox
129+
containers see sysbox's [official documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md).
130+
- Initial workspace startup is slower than running `sysbox-runc` directly on the nodes. This is due
131+
to `envbox` having to pull the image to its own Docker cache on its initial startup. Once the image
132+
is cached in `envbox`, startup performance is similar.
133+
134+
Envbox requires the same kernel requirements as running sysbox directly on the nodes. Refer
135+
to sysbox's [compatibility matrix](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md#sysbox-distro-compatibility) to ensure your nodes are compliant.
136+
137+
To get started with `envbox` check out the [starter template](../../examples/templates/envbox) or visit the [repo](https://github.com/coder/envbox).
138+
139+
### Authenticating with a Private Registry
140+
141+
Authenticating with a private container registry can be done by referencing the credentials
142+
via the `CODER_IMAGE_PULL_SECRET` environment variable. It is encouraged to populate this
143+
[environment variable](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-container-environment-variables-using-secret-data) by using a Kubernetes [secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials).
144+
145+
Refer to your container registry documentation to understand how to best create this secret.
146+
147+
The following shows a minimal example using a the JSON API key from a GCP service account to pull
148+
a private image:
149+
150+
```bash
151+
# Create the secret
152+
$ kubectl create secret docker-registry <name> \
153+
--docker-server=us.gcr.io \
154+
--docker-username=_json_key \
155+
--docker-password="$(cat ./json-key-file.yaml)" \
156+
--docker-email=<service-account-email>
157+
```
158+
159+
```hcl
160+
env {
161+
name = "CODER_IMAGE_PULL_SECRET"
162+
value_from {
163+
secret_key_ref {
164+
name = "<name>"
165+
key = ".dockerconfigjson"
166+
}
167+
}
168+
}
169+
```
170+
113171
## Rootless podman
114172

115173
[Podman](https://docs.podman.io/en/latest/) is Docker alternative that is compatible with OCI containers specification. which can run rootless inside Kubernetes pods. No custom RuntimeClass is required.

‎examples/templates/envbox/README.md

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# envbox
2+
3+
## Introduction
4+
5+
`envbox` is an image that enables creating non-privileged containers capable of running system-level software (e.g. `dockerd`, `systemd`, etc) in Kubernetes.
6+
7+
It mainly acts as a wrapper for the excellent [sysbox runtime](https://github.com/nestybox/sysbox/) developed by [Nestybox](https://www.nestybox.com/). For more details on the security of `sysbox` containers see sysbox's [official documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md).
8+
9+
## Envbox Configuration
10+
11+
The following environment variables can be used to configure various aspects of the inner and outer container.
12+
13+
| env | usage | required |
14+
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
15+
| `CODER_INNER_IMAGE` | The image to use for the inner container. | True |
16+
| `CODER_INNER_USERNAME` | The username to use for the inner container. | True |
17+
| `CODER_AGENT_TOKEN` | The [Coder Agent](https://coder.com/docs/v2/latest/about/architecture#agents) token to pass to the inner container. | True |
18+
| `CODER_INNER_ENVS` | The environment variables to pass to the inner container. A wildcard can be used to match a prefix. Ex: `CODER_INNER_ENVS=KUBERNETES_*,MY_ENV,MY_OTHER_ENV` | false |
19+
| `CODER_INNER_HOSTNAME` | The hostname to use for the inner container. | false |
20+
| `CODER_IMAGE_PULL_SECRET` | The docker credentials to use when pulling the inner container. The recommended way to do this is to create an [Image Pull Secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) and then reference the secret using an [environment variable](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-container-environment-variables-using-secret-data). | false |
21+
| `CODER_DOCKER_BRIDGE_CIDR` | The bridge CIDR to start the Docker daemon with. | false |
22+
| `CODER_MOUNTS` | A list of mounts to mount into the inner container. Mounts default to `rw`. Ex: `CODER_MOUNTS=/home/coder:/home/coder,/var/run/mysecret:/var/run/mysecret:ro` | false |
23+
| `CODER_USR_LIB_DIR` | The mountpoint of the host `/usr/lib` directory. Only required when using GPUs. | false |
24+
| `CODER_ADD_TUN` | If `CODER_ADD_TUN=true` add a TUN device to the inner container. | false |
25+
| `CODER_ADD_FUSE` | If `CODER_ADD_FUSE=true` add a FUSE device to the inner container. | false |
26+
| `CODER_ADD_GPU` | If `CODER_ADD_GPU=true` add detected GPUs and related files to the inner container. Requires setting `CODER_USR_LIB_DIR` and mounting in the hosts `/usr/lib/` directory. | false |
27+
| `CODER_CPUS` | Dictates the number of CPUs to allocate the inner container. It is recommended to set this using the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-container-fields-as-values-for-environment-variables). | false |
28+
| `CODER_MEMORY` | Dictates the max memory (in bytes) to allocate the inner container. It is recommended to set this using the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-container-fields-as-values-for-environment-variables). | false |
29+
30+
## Contributions
31+
32+
Contributions are welcome and can be made against the [envbox repo](https://github.com/coder/envbox).

0 commit comments

Comments
 (0)