Skip to content

Commit 968d7e4

Browse files
bpmctbpmct
authored
docs: rootless podman support (#6026)
* rootless podman WIP * docs: rootless podman support
1 parent e70b3f2 commit 968d7e4

File tree

7 files changed

+651
-4
lines changed

7 files changed

+651
-4
lines changed

‎docs/templates/docker-in-docker.md

+74-4
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,11 @@
22

33
There are a few ways to run Docker within container-based Coder workspaces.
44

5-
| Method | Description | Limitations |
6-
| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
7-
| [Sysbox container runtime](#sysbox-container-runtime) | Install sysbox on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
8-
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privilged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |
5+
| Method | Description | Limitations |
6+
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
7+
| [Sysbox container runtime](#sysbox-container-runtime) | Install the sysbox runtime on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. | Requires [compatible nodes](https://github.com/nestybox/sysbox#host-requirements). Max of 16 sysbox pods per node. [See all](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md) |
8+
| [Rootless Podman](https://github.com/bpmct/coder-templates/tree/main/rootless-podman) | Run podman inside Coder workspaces. Does not require a custom runtime or privileged containers. Works with GKE, EKS, AKS, RKE, OpenShift | Requires smarter-device-manager for FUSE mounts. [See all](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman) |
9+
| [Privileged docker sidecar](#privileged-sidecar-container) | Run docker as a privileged sidecar container. | Requires a privileged container. Workspaces can break out to root on the host machine. |
910

1011
## Sysbox container runtime
1112

@@ -109,6 +110,75 @@ resource "kubernetes_pod" "dev" {
109110

110111
> Sysbox CE (Community Edition) supports a maximum of 16 pods (workspaces) per node on Kubernetes. See the [Sysbox documentation](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#limitations) for more details.
111112
113+
## Rootless podman
114+
115+
[Podman](https://docs.podman.io/en/latest/) is Docker alternative that is compatible with OCI containers specification. which can run rootless inside Kubernetes pods. No custom RuntimeClass is required.
116+
117+
Prior to completing the steps below, please review the following Podman documentation:
118+
119+
- [Basic setup and use of Podman in a rootless environment](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md)
120+
121+
- [Shortcomings of Rootless Podman](https://github.com/containers/podman/blob/main/rootless.md#shortcomings-of-rootless-podman)
122+
123+
1. Enable [smart-device-manager](https://gitlab.com/arm-research/smarter/smarter-device-manager#enabling-access) to securely expose a FUSE devices to pods.
124+
125+
```sh
126+
cat <<EOF | kubectl create -f -
127+
apiVersion: apps/v1
128+
kind: DaemonSet
129+
metadata:
130+
name: fuse-device-plugin-daemonset
131+
namespace: kube-system
132+
spec:
133+
selector:
134+
matchLabels:
135+
name: fuse-device-plugin-ds
136+
template:
137+
metadata:
138+
labels:
139+
name: fuse-device-plugin-ds
140+
spec:
141+
hostNetwork: true
142+
containers:
143+
- image: soolaugust/fuse-device-plugin:v1.0
144+
name: fuse-device-plugin-ctr
145+
securityContext:
146+
allowPrivilegeEscalation: false
147+
capabilities:
148+
drop: ["ALL"]
149+
volumeMounts:
150+
- name: device-plugin
151+
mountPath: /var/lib/kubelet/device-plugins
152+
volumes:
153+
- name: device-plugin
154+
hostPath:
155+
path: /var/lib/kubelet/device-plugins
156+
imagePullSecrets:
157+
- name: registry-secret
158+
EOF
159+
```
160+
161+
2. Be sure to label your nodes to enable smarter-device-manager:
162+
163+
```sh
164+
kubectl get nodes
165+
kubectl label nodes --all smarter-device-manager=enabled
166+
```
167+
168+
> ⚠️ **Warning**: If you are using a managed Kubernetes distribution (e.g. AKS, EKS, GKE), be sure to set node labels via your cloud provider. Otherwise, your nodes may drop the labels and break podman functionality.
169+
170+
3. For systems running SELinux (typically Fedora-, CentOS-, and Red Hat-based systems), you may need to disable SELinux or set it to permissive mode.
171+
172+
4. Import our [kubernetes-podman](https://github.com/coder/coder/tree/main/examples/templates/kubernetes-podman) example template, or make your own.
173+
174+
```sh
175+
echo "kubernetes-podman" | coder templates init
176+
cd ./kubernetes-podman
177+
coder templates create
178+
```
179+
180+
> For more information around the requirements of rootless podman pods, see: [How to run Podman inside of Kubernetes](https://www.redhat.com/sysadmin/podman-inside-kubernetes)
181+
112182
## Privileged sidecar container
113183
114184
A [privileged container](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) can be added to your templates to add docker support. This may come in handy if your nodes cannot run Sysbox.

‎examples/templates/kubernetes-with-podman/README.md

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
name: Develop in Kubernetes
3+
description: Get started with Kubernetes development.
4+
tags: [cloud, kubernetes]
5+
icon: /icon/k8s.png
6+
---
7+
8+
# Getting started
9+
10+
This template creates [rootless podman](./images) pods with either an Ubuntu or Fedora base image.
11+
12+
> **Warning**: This template requires additional configuration on the Kubernetes cluster, such as installing `smarter-device-manager` for FUSE mounts. See our [Docker-in-Docker documentation](https://coder.com/docs/v2/latest/templates/docker-in-docker#rootless-podman) for instructions.
13+
14+
Base images are pushed to [Docker Hub](https://hub.docker.com//codercom)
15+
16+
## RBAC
17+
18+
The Coder provisioner requires permission to administer pods to use this template. The template
19+
creates workspaces in a single Kubernetes namespace, using the `workspaces_namespace` parameter set
20+
while creating the template.
21+
22+
Create a role as follows and bind it to the user or service account that runs the coder host.
23+
24+
```yaml
25+
apiVersion: rbac.authorization.k8s.io/v1
26+
kind: Role
27+
metadata:
28+
name: coder
29+
rules:
30+
- apiGroups: [""]
31+
resources: ["pods"]
32+
verbs: ["*"]
33+
```
34+
35+
## Authentication
36+
37+
This template can authenticate using in-cluster authentication, or using a kubeconfig local to the
38+
Coder host. For additional authentication options, consult the [Kubernetes provider
39+
documentation](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs).
40+
41+
### kubeconfig on Coder host
42+
43+
If the Coder host has a local `~/.kube/config`, you can use this to authenticate
44+
with Coder. Make sure this is done with same user that's running the `coder` service.
45+
46+
To use this authentication, set the parameter `use_kubeconfig` to true.
47+
48+
### In-cluster authentication
49+
50+
If the Coder host runs in a Pod on the same Kubernetes cluster as you are creating workspaces in,
51+
you can use in-cluster authentication.
52+
53+
To use this authentication, set the parameter `use_kubeconfig` to false.
54+
55+
The Terraform provisioner will automatically use the service account associated with the pod to
56+
authenticate to Kubernetes. Be sure to bind a [role with appropriate permission](#rbac) to the
57+
service account. For example, assuming the Coder host runs in the same namespace as you intend
58+
to create workspaces:
59+
60+
```yaml
61+
apiVersion: v1
62+
kind: ServiceAccount
63+
metadata:
64+
name: coder
65+
66+
---
67+
apiVersion: rbac.authorization.k8s.io/v1
68+
kind: RoleBinding
69+
metadata:
70+
name: coder
71+
subjects:
72+
- kind: ServiceAccount
73+
name: coder
74+
roleRef:
75+
kind: Role
76+
name: coder
77+
apiGroup: rbac.authorization.k8s.io
78+
```
79+
80+
Then start the Coder host with `serviceAccountName: coder` in the pod spec.
81+
82+
## Namespace
83+
84+
The target namespace in which the pod will be deployed is defined via the `coder_workspace`
85+
variable. The namespace must exist prior to creating workspaces.
86+
87+
## Persistence
88+
89+
The `/home/coder` directory in this example is persisted via the attached PersistentVolumeClaim.
90+
Any data saved outside of this directory will be wiped when the workspace stops.
91+
92+
Since most binary installations and environment configurations live outside of
93+
the `/home` directory, we suggest including these in the `startup_script` argument
94+
of the `coder_agent` resource block, which will run each time the workspace starts up.
95+
96+
For example, when installing the `aws` CLI, the install script will place the
97+
`aws` binary in `/usr/local/bin/aws`. To ensure the `aws` CLI is persisted across
98+
workspace starts/stops, include the following code in the `coder_agent` resource
99+
block of your workspace template:
100+
101+
```terraform
102+
resource "coder_agent" "main" {
103+
startup_script = <<-EOT
104+
set -e
105+
# install AWS CLI
106+
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
107+
unzip awscliv2.zip
108+
sudo ./aws/install
109+
EOT
110+
}
111+
```
112+
113+
## code-server
114+
115+
`code-server` is installed via the `startup_script` argument in the `coder_agent`
116+
resource block. The `coder_app` resource is defined to access `code-server` through
117+
the dashboard UI over `localhost:13337`.

‎examples/templates/kubernetes-with-podman/images/Dockerfile.fedora

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
FROM registry.fedoraproject.org/fedora:latest
2+
3+
LABEL org.opencontainers.image.description="Base Fedora image for rootless podman in Coder. See https://coder.com/docs/v2/latest/templates/docker-in-docker#rootless-podman"
4+
5+
RUN dnf -y update && \
6+
rpm --setcaps shadow-utils 2>/dev/null && \
7+
dnf -y install podman fuse-overlayfs openssh-clients \
8+
--exclude container-selinux && \
9+
dnf clean all && \
10+
rm -rf /var/cache /var/log/dnf* /var/log/yum.*
11+
12+
RUN useradd podman; \
13+
echo -e "podman:1:999\npodman:1001:64535" > /etc/subuid; \
14+
echo -e "podman:1:999\npodman:1001:64535" > /etc/subgid;
15+
16+
ADD containers.conf /etc/containers/containers.conf
17+
ADD storage.conf /etc/containers/storage.conf
18+
RUN chmod 644 /etc/containers/containers.conf && \
19+
chmod 644 /etc/containers/storage.conf
20+
21+
RUN mkdir -p /var/lib/shared/overlay-images \
22+
/var/lib/shared/overlay-layers \
23+
/var/lib/shared/vfs-images \
24+
/var/lib/shared/vfs-layers && \
25+
touch /var/lib/shared/overlay-images/images.lock && \
26+
touch /var/lib/shared/overlay-layers/layers.lock && \
27+
touch /var/lib/shared/vfs-images/images.lock && \
28+
touch /var/lib/shared/vfs-layers/layers.lock
29+
30+
# Alias "docker" to "podman"
31+
RUN ln -s /usr/bin/podman /usr/bin/docker
32+
33+
USER podman
34+
35+
ENV _CONTAINERS_USERNS_CONFIGURED=""

‎examples/templates/kubernetes-with-podman/images/Dockerfile.ubuntu

+59
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
FROM ubuntu:22.04
2+
3+
LABEL org.opencontainers.image.description="Base Ubuntu image for rootless podman in Coder. See https://coder.com/docs/v2/latest/templates/docker-in-docker#rootless-podman"
4+
5+
USER root
6+
7+
# Install dependencies
8+
RUN apt-get update && apt-get install -y sudo gnupg2 curl vim fuse-overlayfs libvshadow-utils openssh-client
9+
10+
# Install podman
11+
RUN mkdir -p /etc/apt/keyrings
12+
RUN curl -fsSL https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_22.04/Release.key \
13+
| gpg --dearmor \
14+
| tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null
15+
RUN echo \
16+
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg]\
17+
https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_22.04/ /" \
18+
| tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null
19+
RUN apt-get update && apt-get -y install podman
20+
21+
RUN setcap cap_setuid+ep /usr/bin/newuidmap
22+
RUN setcap cap_setgid+ep /usr/bin/newgidmap
23+
RUN chmod 0755 /usr/bin/newuidmap
24+
RUN chmod 0755 /usr/bin/newgidmap
25+
26+
RUN useradd podman
27+
RUN echo "podman:100000:65536" > /etc/subuid
28+
RUN echo "podman:100000:65536" > /etc/subgid
29+
RUN echo "podman ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers
30+
31+
ADD containers.conf /etc/containers/containers.conf
32+
ADD storage.conf /etc/containers/storage.conf
33+
RUN chmod 644 /etc/containers/containers.conf && \
34+
chmod 644 /etc/containers/storage.conf
35+
36+
RUN mkdir -p /home/podman/.local/share/containers && \
37+
chown podman:podman -R /home/podman && \
38+
chmod 644 /etc/containers/containers.conf
39+
40+
RUN mkdir -p /var/lib/shared/overlay-images \
41+
/var/lib/shared/overlay-layers \
42+
/var/lib/shared/vfs-images \
43+
/var/lib/shared/vfs-layers && \
44+
touch /var/lib/shared/overlay-images/images.lock && \
45+
touch /var/lib/shared/overlay-layers/layers.lock && \
46+
touch /var/lib/shared/vfs-images/images.lock && \
47+
touch /var/lib/shared/vfs-layers/layers.lock
48+
49+
ENV _CONTAINERS_USERNS_CONFIGURED=""
50+
51+
# Alias "docker" to "podman"
52+
RUN ln -s /usr/bin/podman /usr/bin/docker
53+
54+
RUN chsh -s /bin/bash podman
55+
56+
57+
USER podman
58+
59+
ENV SHELL=/bin/bash

‎examples/templates/kubernetes-with-podman/images/containers.conf

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
[containers]
2+
netns="host"
3+
userns="host"
4+
ipcns="host"
5+
utsns="host"
6+
cgroupns="host"
7+
cgroups="disabled"
8+
log_driver = "k8s-file"
9+
volumes = [
10+
"/proc:/proc",
11+
]
12+
default_sysctls = []
13+
[engine]
14+
cgroup_manager = "cgroupfs"
15+
events_logger="file"
16+
runtime="crun"

0 commit comments

Comments
 (0)