Skip to content

chore: add envbox documentation #7198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 26, 2023
Merged

chore: add envbox documentation #7198

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3987853
chore: add envbox documentation
sreya Apr 19, 2023
ae8e742
grammar
sreya Apr 19, 2023
e2f47e1
fmt
sreya Apr 19, 2023
54eb04f
add example of providing image pull secrets
sreya Apr 20, 2023
a25750f
comments
sreya Apr 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 63 additions & 5 deletions docs/templates/docker-in-workspaces.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@

There are a few ways to run Docker within container-based Coder workspaces.

| Method | Description | Limitations |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Sysbox container runtime](#sysbox-container-runtime) | Install the sysbox runtime on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
| [Rootless Podman](#rootless-podman) | Run podman inside Coder workspaces. Does not require a custom runtime or privileged containers. Works with GKE, EKS, AKS, RKE, OpenShift | Requires smarter-device-manager for FUSE mounts. [See all](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman) |
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privileged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |
| Method | Description | Limitations |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Sysbox container runtime](#sysbox-container-runtime) | Install the sysbox runtime on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
| [Envbox](#envbox) | A container image with all the packages necessary to run an inner sysbox container. Removes the need to setup sysbox-runc on your nodes. Works with GKE, EKS, AKS. | Requires running the outer container as privileged (the inner container that acts as the workspace is locked down). Requires compatible [nodes](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md#sysbox-distro-compatibility). |
| [Rootless Podman](#rootless-podman) | Run podman inside Coder workspaces. Does not require a custom runtime or privileged containers. Works with GKE, EKS, AKS, RKE, OpenShift | Requires smarter-device-manager for FUSE mounts. [See all](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman) |
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privileged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |

## Sysbox container runtime

Expand Down Expand Up @@ -110,6 +111,63 @@ resource "kubernetes_pod" "dev" {

> Sysbox CE (Community Edition) supports a maximum of 16 pods (workspaces) per node on Kubernetes. See the [Sysbox documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#limitations) for more details.

## Envbox

[Envbox](https://github.com/coder/envbox) is an image developed and maintained by Coder that bundles the sysbox runtime. It works
by starting an outer container that manages the various sysbox daemons and spawns an unprivileged
inner container that acts as the user's workspace. The inner container is able to run system-level
software similar to a regular virtual machine (e.g. `systemd`, `dockerd`, etc). Envbox offers the
following benefits over running sysbox directly on the nodes:

- No custom runtime installation or management on your Kubernetes nodes.
- No limit to the number of pods that run envbox.

Some drawbacks include:

- The outer container must be run as privileged
- Note: the inner container is _not_ privileged. For more information on the security of sysbox
containers see sysbox's [official documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md).
- Initial workspace startup is slower than running `sysbox-runc` directly on the nodes. This is due
to `envbox` having to pull the image to its own Docker cache on its initial startup. Once the image
is cached in `envbox`, startup performance is similar.

Envbox requires the same kernel requirements as running sysbox directly on the nodes. Refer
to sysbox's [compatibility matrix](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md#sysbox-distro-compatibility) to ensure your nodes are compliant.

To get started with `envbox` check out the [starter template](../../examples/templates/envbox) or visit the [repo](https://github.com/coder/envbox).

### Authenticating with a Private Registry

Authenticating with a private container registry can be done by referencing the credentials
via the `CODER_IMAGE_PULL_SECRET` environment variable. It is encouraged to populate this
[environment variable](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-container-environment-variables-using-secret-data) by using a Kubernetes [secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials).

Refer to your container registry documentation to understand how to best create this secret.

The following shows a minimal example using a the JSON API key from a GCP service account to pull
a private image:

```bash
# Create the secret
$ kubectl create secret docker-registry <name> \
--docker-server=us.gcr.io \
--docker-username=_json_key \
--docker-password="$(cat ./json-key-file.yaml)" \
--docker-email=<service-account-email>
```

```hcl
env {
name = "CODER_IMAGE_PULL_SECRET"
value_from {
secret_key_ref {
name = "<name>"
key = ".dockerconfigjson"
}
}
}
```

## Rootless podman

[Podman](https://docs.podman.io/en/latest/) is Docker alternative that is compatible with OCI containers specification. which can run rootless inside Kubernetes pods. No custom RuntimeClass is required.
Expand Down
32 changes: 32 additions & 0 deletions examples/templates/envbox/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# envbox

## Introduction

`envbox` is an image that enables creating non-privileged containers capable of running system-level software (e.g. `dockerd`, `systemd`, etc) in Kubernetes.

It mainly acts as a wrapper for the excellent [sysbox runtime](https://github.com/nestybox/sysbox/) developed by [Nestybox](https://www.nestybox.com/). For more details on the security of `sysbox` containers see sysbox's [official documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md).

## Envbox Configuration

The following environment variables can be used to configure various aspects of the inner and outer container.

| env | usage | required |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `CODER_INNER_IMAGE` | The image to use for the inner container. | True |
| `CODER_INNER_USERNAME` | The username to use for the inner container. | True |
| `CODER_AGENT_TOKEN` | The [Coder Agent](https://coder.com/docs/v2/latest/about/architecture#agents) token to pass to the inner container. | True |
| `CODER_INNER_ENVS` | The environment variables to pass to the inner container. A wildcard can be used to match a prefix. Ex: `CODER_INNER_ENVS=KUBERNETES_*,MY_ENV,MY_OTHER_ENV` | false |
| `CODER_INNER_HOSTNAME` | The hostname to use for the inner container. | false |
| `CODER_IMAGE_PULL_SECRET` | The docker credentials to use when pulling the inner container. The recommended way to do this is to create an [Image Pull Secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) and then reference the secret using an [environment variable](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-container-environment-variables-using-secret-data). | false |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry. I totally missed this part 🤦🏼

If you want to remove this from the docker in docker docs in favor of this, that's fine.

| `CODER_DOCKER_BRIDGE_CIDR` | The bridge CIDR to start the Docker daemon with. | false |
| `CODER_MOUNTS` | A list of mounts to mount into the inner container. Mounts default to `rw`. Ex: `CODER_MOUNTS=/home/coder:/home/coder,/var/run/mysecret:/var/run/mysecret:ro` | false |
| `CODER_USR_LIB_DIR` | The mountpoint of the host `/usr/lib` directory. Only required when using GPUs. | false |
| `CODER_ADD_TUN` | If `CODER_ADD_TUN=true` add a TUN device to the inner container. | false |
| `CODER_ADD_FUSE` | If `CODER_ADD_FUSE=true` add a FUSE device to the inner container. | false |
| `CODER_ADD_GPU` | If `CODER_ADD_GPU=true` add detected GPUs and related files to the inner container. Requires setting `CODER_USR_LIB_DIR` and mounting in the hosts `/usr/lib/` directory. | false |
| `CODER_CPUS` | Dictates the number of CPUs to allocate the inner container. It is recommended to set this using the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-container-fields-as-values-for-environment-variables). | false |
| `CODER_MEMORY` | Dictates the max memory (in bytes) to allocate the inner container. It is recommended to set this using the Kubernetes [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-container-fields-as-values-for-environment-variables). | false |

## Contributions

Contributions are welcome and can be made against the [envbox repo](https://github.com/coder/envbox).
Loading