Problem

We write a whole bunch of data to CODER_CACHE_DIRECTORY but we never seem to clean it out.
This can end up consuming the entire partition, potentially leading to service interruption / outage.

Solution

Periodically clean CODER_CACHE_DIRECTORY.

• The simplest approach here is probably to remove files older than a configurable threshold.
• An alternative approach could be to remove files, starting with the oldest first, until the size of the cache directory is below a configurable threshold.
• Care will need to be taken that we do not perform this task while there are in-progress provisioner jobs, as this could cause them to fail.
  • We may need to provide a mechanism to temporarily "lock" local provisioners and pause them from accepting new jobs. We could potentially accomplish this by simply locking the jobs table.