Description
Overview
Add documentation for userspace filesystems in CVMs. The background and implementation details are provided for context and do not need to be included in the user-facing documentation.
Background
In 1.28, we introduced a feature that allows the creation of network interfaces (TUN devices) inside CVM workspaces. This device is used by VPN software or other overlay networks to allow connectivity between devices over a tunneled connect (connection-in-connection, such as IP-over-TCP or TCP-over-TCP).
In 1.29, we used a similar mechanism to introduce a feature that allows the creation of custom filesystems inside CVM workspaces. Filesystem in Userspace (FUSE) is a way to create arbitrary filesystems, and is useful for mounting remote volumes. There are a number of implementations of FUSE filesystems:
- rclone: mount Google Drive, Box, Dropbox, etc as a filesystem
- sshfs: mount a path from a remote machine using SSH and SFTP
- gcsfuse: mount a bucket from Google Cloud Storage
- s3fuse: mount a bucket from AWS S3
Implementation details
To implement both of these features, we pass through the /dev/net/tun and /dev/fuse devices from the privileged outer container (envbox) to the unprivileged inner container (the workspace itself).
In the case of non-cached CVMs, we create a new TUN or FUSE device using mknod and we configure docker to pass these devices to the workspace. In the case of cached CVMs, we generate a runc configuration that creates these devices. Functionally, both are equivalent, and the difference owes only to implementation differences between regular and cached CVMs (the former uses docker to containerd to runc, and the latter invokes runc directly).
As a limitation of this approach, the feature is only available in CVMs, because creating the device requires some capabilities (such as CAP_MKNOD) that are not available in an unprivileged context. In essence, sysbox and envbox act as a kind of hypervisor, where the hypervisor itself (the outer container/privileged container) needs to perform an action on behalf of the guest (the inner/unprivileged container). In this case, creating the device requires privileges, but once created, it can be added into the unprivileged container and used normally.
Proposed changes
-
Move the existing TUN device documentation to a more general "devices" page under the "CVM" section, since this applies only to CVMs
Alternatively, we can make this a more general page that also includes how to get TUN/FUSE devices for unprivileged workspaces, as a less convenient example way to achieve the same is to use smarter-device-manager as needed for Rootless Podman support.
-
Add a section in the same document explaining that admins can enable FUSE devices in CVMs using a similar checkbox (currently visible/usable in master)
Testing notes
The pull request that implements this feature is https://github.com/coder/m/pull/11815
In order to test this feature, I suggest using unionfs-fuse, which is available as a Ubuntu package (install it with apt install unionfs-fuse). What this does is allow you to create a virtual filesystem which is a view of a combination (or union) of multiple other filesystems, effectively merging them together. In my user directory, I've created two directories (tmp1 and tmp2) and mounted those into a union tmp directory:
$ unionfs-fuse tmp1:tmp2 tmp
$ findmnt
TARGET SOURCE FSTYPE OPTIONS
/ overlay overla rw,rela
|-/sys sysfs sysfs rw,nosu
| |-/sys/firmware tmpfs tmpfs ro,rela
| |-/sys/fs/cgroup tmpfs tmpfs ro,nosu
| | |-/sys/fs/cgroup/systemd systemd cgroup rw,nosu
| | |-/sys/fs/cgroup/devices cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/blkio cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/memory cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/cpuset cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/net_cls,net_prio
| | | cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/pids cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/perf_event cgroup cgroup rw,nosu
| | |-/sys/fs/cgroup/freezer cgroup cgroup rw,nosu
| | `-/sys/fs/cgroup/rdma cgroup cgroup rw,nosu
| |-/sys/kernel/config tmpfs tmpfs rw,nosu
| |-/sys/kernel/debug tmpfs tmpfs rw,nosu
| |-/sys/kernel/tracing tmpfs tmpfs rw,nosu
| |-/sys/devices/virtual/dmi/id/product_uuid
| | sysboxfs[/sys/devices/virtual/dmi/id/product_uuid]
| | fuse rw,nosu
| `-/sys/module/nf_conntrack/parameters/hashsize
| sysboxfs[/sys/module/nf_conntrack/parameters/hashsize]
| fuse rw,nosu
|-/proc proc proc rw,nosu
| |-/proc/bus proc[/bus] proc ro,nosu
| |-/proc/fs proc[/fs] proc ro,nosu
| |-/proc/irq proc[/irq] proc ro,nosu
| |-/proc/sysrq-trigger proc[/sysrq-trigger] proc ro,nosu
| |-/proc/acpi tmpfs tmpfs ro,rela
| |-/proc/keys tmpfs[/null] tmpfs rw,nosu
| |-/proc/timer_list tmpfs[/null] tmpfs rw,nosu
| |-/proc/sched_debug tmpfs[/null] tmpfs rw,nosu
| |-/proc/scsi tmpfs tmpfs ro,rela
| |-/proc/swaps sysboxfs[/proc/swaps] fuse rw,nosu
| |-/proc/sys sysboxfs[/proc/sys] fuse rw,nosu
| `-/proc/uptime sysboxfs[/proc/uptime] fuse rw,nosu
|-/dev tmpfs tmpfs rw,nosu
| |-/dev/mqueue mqueue mqueue rw,nosu
| |-/dev/pts devpts devpts rw,nosu
| |-/dev/shm shm tmpfs rw,nosu
| |-/dev/null tmpfs[/null] tmpfs rw,nosu
| |-/dev/kmsg tmpfs[/null] tmpfs rw,nosu
| |-/dev/random tmpfs[/random] tmpfs rw,nosu
| |-/dev/full tmpfs[/full] tmpfs rw,nosu
| |-/dev/tty tmpfs[/tty] tmpfs rw,nosu
| |-/dev/zero tmpfs[/zero] tmpfs rw,nosu
| |-/dev/urandom tmpfs[/urandom] tmpfs rw,nosu
| |-/dev/net/tun tmpfs[/net/tun] tmpfs rw,nosu
| `-/dev/fuse tmpfs[/fuse] tmpfs rw,nosu
|-/run tmpfs tmpfs rw,nosu
| `-/run/lock tmpfs tmpfs rw,nosu
|-/home/coder /dev/sdc[/home] ext4 rw,rela
| `-/home/coder/tmp unionfs-fuse fuse.u rw,nosu
|-/etc/resolv.conf /dev/sdc[/envbox/docker/100000.100000/containers/16b0389c87104d343e7cfc51a0794fbfee0420cbd5990f54269b8f352175b26c/resolv.conf]
| ext4 rw,rela
|-/etc/hostname /dev/sdc[/envbox/docker/100000.100000/containers/16b0389c87104d343e7cfc51a0794fbfee0420cbd5990f54269b8f352175b26c/hostname]
| ext4 rw,rela
|-/etc/hosts /dev/sdc[/envbox/docker/100000.100000/containers/16b0389c87104d343e7cfc51a0794fbfee0420cbd5990f54269b8f352175b26c/hosts]
| ext4 rw,rela
|-/var/lib/containers /dev/sdc[/cache/containers] ext4 rw,rela
|-/var/lib/docker /dev/sdc[/cache/docker] ext4 rw,rela
|-/var/lib/kubelet /dev/root[/var/lib/kubelet/pods/dbaf76c7-4081-4d84-a86d-d728a6777c4c/volumes/kubernetes.io~empty-dir/sysbox/kubelet/16b0389c87104d343e7cfc51a0794fbfee0420cbd5990f54269b8f352175b26c]
| ext4 rw,rela
|-/var/lib/rancher/k3s /dev/root[/var/lib/kubelet/pods/dbaf76c7-4081-4d84-a86d-d728a6777c4c/volumes/kubernetes.io~empty-dir/sysbox/rancher-k3s/16b0389c87104d343e7cfc51a0794fbfee0420cbd5990f54269b8f352175b26c]
| ext4 rw,rela
|-/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
| /dev/root[/var/lib/kubelet/pods/dbaf76c7-4081-4d84-a86d-d728a6777c4c/volumes/kubernetes.io~empty-dir/sysbox/containerd/16b0389c87104d343e7cfc51a0794fbfee0420cbd5990f54269b8f352175b26c]
| ext4 rw,rela
|-/usr/src/linux-headers-5.4.0-1054-gke
| /dev/root[/usr/src/linux-headers-5.4.0-1054-gke] ext4 ro,rela
|-/usr/src/linux-gke-headers-5.4.0-1054
| /dev/root[/usr/src/linux-gke-headers-5.4.0-1054] ext4 ro,rela
`-/usr/lib/modules/5.4.0-1054-gke /dev/root[/usr/lib/modules/5.4.0-1054-gke] ext4 ro,rela
I verified that I can modify files in tmp1 and tmp2 and that they appear in tmp:
$ ls -al tmp tmp1 tmp2
tmp:
total 8
drwxr-xr-x 1 coder coder 4096 Feb 3 20:53 .
drwxr-xr-x 8 coder coder 4096 Feb 3 20:52 ..
-rw-r--r-- 1 coder coder 0 Feb 3 20:53 test
-rw-r--r-- 1 coder coder 0 Feb 3 20:53 test2
tmp1:
total 8
drwxr-xr-x 2 coder coder 4096 Feb 3 20:53 .
drwxr-xr-x 8 coder coder 4096 Feb 3 20:52 ..
-rw-r--r-- 1 coder coder 0 Feb 3 20:53 test2
tmp2:
total 8
drwxr-xr-x 2 coder coder 4096 Feb 3 20:53 .
drwxr-xr-x 8 coder coder 4096 Feb 3 20:52 ..
-rw-r--r-- 1 coder coder 0 Feb 3 20:53 test