Skip to content

docs: rootless podman support #6026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 6, 2023
Merged

docs: rootless podman support #6026

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6833898
rootless podman WIP
bpmct Feb 3, 2023
b908ae0
docs: rootless podman support
bpmct Feb 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 74 additions & 4 deletions docs/templates/docker-in-docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@

There are a few ways to run Docker within container-based Coder workspaces.

| Method | Description | Limitations |
| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Sysbox container runtime](#sysbox-container-runtime) | Install sysbox on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privilged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |
| Method | Description | Limitations |
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Sysbox container runtime](#sysbox-container-runtime) | Install the sysbox runtime on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
| [Rootless Podman](https://github.com/bpmct/coder-templates/tree/main/rootless-podman) | Run podman inside Coder workspaces. Does not require a custom runtime or privileged containers. Works with GKE, EKS, AKS, RKE, OpenShift | Requires smarter-device-manager for FUSE mounts. [See all](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman) |
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privileged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |

## Sysbox container runtime

Expand Down Expand Up @@ -109,6 +110,75 @@ resource "kubernetes_pod" "dev" {

> Sysbox CE (Community Edition) supports a maximum of 16 pods (workspaces) per node on Kubernetes. See the [Sysbox documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#limitations) for more details.

## Rootless podman

[Podman](https://docs.podman.io/en/latest/) is Docker alternative that is compatible with OCI containers specification. which can run rootless inside Kubernetes pods. No custom RuntimeClass is required.

Prior to completing the steps below, please review the following Podman documentation:

- [Basic setup and use of Podman in a rootless environment](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md)

- [Shortcomings of Rootless Podman](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman)

1. Enable [smart-device-manager](https://gitlab.com/arm-research/smarter/smarter-device-manager#enabling-access) to securely expose a FUSE devices to pods.

```sh
cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fuse-device-plugin-daemonset
namespace: kube-system
spec:
selector:
matchLabels:
name: fuse-device-plugin-ds
template:
metadata:
labels:
name: fuse-device-plugin-ds
spec:
hostNetwork: true
containers:
- image: soolaugust/fuse-device-plugin:v1.0
name: fuse-device-plugin-ctr
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
imagePullSecrets:
- name: registry-secret
EOF
```

2. Be sure to label your nodes to enable smarter-device-manager:

```sh
kubectl get nodes
kubectl label nodes --all smarter-device-manager=enabled
```

> ⚠️ **Warning**: If you are using a managed Kubernetes distribution (e.g. AKS, EKS, GKE), be sure to set node labels via your cloud provider. Otherwise, your nodes may drop the labels and break podman functionality.

3. For systems running SELinux (typically Fedora-, CentOS-, and Red Hat-based systems), you may need to disable SELinux or set it to permissive mode.

4. Import our [kubernetes-podman](https://github.com/coder/coder/tree/main/examples/templates/kubernetes-podman) example template, or make your own.

```sh
echo "kubernetes-podman" | coder templates init
cd ./kubernetes-podman
coder templates create
```

> For more information around the requirements of rootless podman pods, see: [How to run Podman inside of Kubernetes](https://www.redhat.com/sysadmin/podman-inside-kubernetes)

## Privileged sidecar container

A [privileged container](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) can be added to your templates to add docker support. This may come in handy if your nodes cannot run Sysbox.
Expand Down
117 changes: 117 additions & 0 deletions examples/templates/kubernetes-with-podman/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
---
name: Develop in Kubernetes
description: Get started with Kubernetes development.
tags: [cloud, kubernetes]
icon: /icon/k8s.png
---

# Getting started

This template creates [rootless podman](./images) pods with either an Ubuntu or Fedora base image.

> **Warning**: This template requires additional configuration on the Kubernetes cluster, such as installing `smarter-device-manager` for FUSE mounts. See our [Docker-in-Docker documentation](https://coder.com/docs/v2/latest/templates/docker-in-docker#rootless-podman) for instructions.

Base images are pushed to [Docker Hub](https://hub.docker.com//codercom)

## RBAC

The Coder provisioner requires permission to administer pods to use this template. The template
creates workspaces in a single Kubernetes namespace, using the `workspaces_namespace` parameter set
while creating the template.

Create a role as follows and bind it to the user or service account that runs the coder host.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: coder
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["*"]
```

## Authentication

This template can authenticate using in-cluster authentication, or using a kubeconfig local to the
Coder host. For additional authentication options, consult the [Kubernetes provider
documentation](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs).

### kubeconfig on Coder host

If the Coder host has a local `~/.kube/config`, you can use this to authenticate
with Coder. Make sure this is done with same user that's running the `coder` service.

To use this authentication, set the parameter `use_kubeconfig` to true.

### In-cluster authentication

If the Coder host runs in a Pod on the same Kubernetes cluster as you are creating workspaces in,
you can use in-cluster authentication.

To use this authentication, set the parameter `use_kubeconfig` to false.

The Terraform provisioner will automatically use the service account associated with the pod to
authenticate to Kubernetes. Be sure to bind a [role with appropriate permission](#rbac) to the
service account. For example, assuming the Coder host runs in the same namespace as you intend
to create workspaces:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: coder

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: coder
subjects:
- kind: ServiceAccount
name: coder
roleRef:
kind: Role
name: coder
apiGroup: rbac.authorization.k8s.io
```

Then start the Coder host with `serviceAccountName: coder` in the pod spec.

## Namespace

The target namespace in which the pod will be deployed is defined via the `coder_workspace`
variable. The namespace must exist prior to creating workspaces.

## Persistence

The `/home/coder` directory in this example is persisted via the attached PersistentVolumeClaim.
Any data saved outside of this directory will be wiped when the workspace stops.

Since most binary installations and environment configurations live outside of
the `/home` directory, we suggest including these in the `startup_script` argument
of the `coder_agent` resource block, which will run each time the workspace starts up.

For example, when installing the `aws` CLI, the install script will place the
`aws` binary in `/usr/local/bin/aws`. To ensure the `aws` CLI is persisted across
workspace starts/stops, include the following code in the `coder_agent` resource
block of your workspace template:

```terraform
resource "coder_agent" "main" {
startup_script = <<-EOT
set -e
# install AWS CLI
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
EOT
}
```

## code-server

`code-server` is installed via the `startup_script` argument in the `coder_agent`
resource block. The `coder_app` resource is defined to access `code-server` through
the dashboard UI over `localhost:13337`.
35 changes: 35 additions & 0 deletions examples/templates/kubernetes-with-podman/images/Dockerfile.fedora
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
FROM registry.fedoraproject.org/fedora:latest

LABEL org.opencontainers.image.description="Base Fedora image for rootless podman in Coder. See https://coder.com/docs/v2/latest/templates/docker-in-docker#rootless-podman"

RUN dnf -y update && \
rpm --setcaps shadow-utils 2>/dev/null && \
dnf -y install podman fuse-overlayfs openssh-clients \
--exclude container-selinux && \
dnf clean all && \
rm -rf /var/cache /var/log/dnf* /var/log/yum.*

RUN useradd podman; \
echo -e "podman:1:999\npodman:1001:64535" > /etc/subuid; \
echo -e "podman:1:999\npodman:1001:64535" > /etc/subgid;

ADD containers.conf /etc/containers/containers.conf
ADD storage.conf /etc/containers/storage.conf
RUN chmod 644 /etc/containers/containers.conf && \
chmod 644 /etc/containers/storage.conf

RUN mkdir -p /var/lib/shared/overlay-images \
/var/lib/shared/overlay-layers \
/var/lib/shared/vfs-images \
/var/lib/shared/vfs-layers && \
touch /var/lib/shared/overlay-images/images.lock && \
touch /var/lib/shared/overlay-layers/layers.lock && \
touch /var/lib/shared/vfs-images/images.lock && \
touch /var/lib/shared/vfs-layers/layers.lock

# Alias "docker" to "podman"
RUN ln -s /usr/bin/podman /usr/bin/docker

USER podman

ENV _CONTAINERS_USERNS_CONFIGURED=""
59 changes: 59 additions & 0 deletions examples/templates/kubernetes-with-podman/images/Dockerfile.ubuntu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
FROM ubuntu:22.04

LABEL org.opencontainers.image.description="Base Ubuntu image for rootless podman in Coder. See https://coder.com/docs/v2/latest/templates/docker-in-docker#rootless-podman"

USER root

# Install dependencies
RUN apt-get update && apt-get install -y sudo gnupg2 curl vim fuse-overlayfs libvshadow-utils openssh-client

# Install podman
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_22.04/Release.key \
| gpg --dearmor \
| tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null
RUN echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg]\
https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_22.04/ /" \
| tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null
RUN apt-get update && apt-get -y install podman

RUN setcap cap_setuid+ep /usr/bin/newuidmap
RUN setcap cap_setgid+ep /usr/bin/newgidmap
RUN chmod 0755 /usr/bin/newuidmap
RUN chmod 0755 /usr/bin/newgidmap

RUN useradd podman
RUN echo "podman:100000:65536" > /etc/subuid
RUN echo "podman:100000:65536" > /etc/subgid
RUN echo "podman ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers

ADD containers.conf /etc/containers/containers.conf
ADD storage.conf /etc/containers/storage.conf
RUN chmod 644 /etc/containers/containers.conf && \
chmod 644 /etc/containers/storage.conf

RUN mkdir -p /home/podman/.local/share/containers && \
chown podman:podman -R /home/podman && \
chmod 644 /etc/containers/containers.conf

RUN mkdir -p /var/lib/shared/overlay-images \
/var/lib/shared/overlay-layers \
/var/lib/shared/vfs-images \
/var/lib/shared/vfs-layers && \
touch /var/lib/shared/overlay-images/images.lock && \
touch /var/lib/shared/overlay-layers/layers.lock && \
touch /var/lib/shared/vfs-images/images.lock && \
touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

# Alias "docker" to "podman"
RUN ln -s /usr/bin/podman /usr/bin/docker

RUN chsh -s /bin/bash podman


USER podman

ENV SHELL=/bin/bash
16 changes: 16 additions & 0 deletions examples/templates/kubernetes-with-podman/images/containers.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[containers]
netns="host"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
cgroups="disabled"
log_driver = "k8s-file"
volumes = [
"/proc:/proc",
]
default_sysctls = []
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
runtime="crun"
Loading