I've been dogfooding it for the last few days without any major issues, so putting this under VCS.

This template uses the latest version of coder/envbuilder, and makes heavy use of a Docker cache to avoid having to actually execute all of the layers.
With this, a newly created workspace is available in approximately 7 minutes from the envbuilder image starting.

Some notes and future work:

1. Some directives in our dogfood Dockerfile currently are not (yet!) supported by our caching (in particular, ENV and ARG commands appear to bust the cache),
2. Extracting the layers to the filesystem currently takes a significant amount of time. There may be some changes we can do to reduce the time spent here.
3. If we check the remote cache (e.g. via checking docker logs output) and just run that pre-built cached image, we would have a big speed-up!
4. On most newer distros, /var/run is a symlink to /run. Kaniko appears to ignore symbolic links in images, meaning that in the built image there is no /var/run folder. For now I'm symlinking this in the agent startup script before starting the Docker daemon.
5. Because the image gets built in-place, the docker group is added to the running system. However, the Coder agent executed by the non-root user doesn't end up in this group. As a workaround, I've made /var/run/docker.sock world read-writable. The alternative is hacking the agent startup script to include newgrp or adding the ability for envbuilder to do the equivalent of a newgrp invocation.