Problem

From testing in #8213, provisionerd uses the vast majority of CPU resources. Large number of concurrent provisioner jobs can easily saturate the CPU causing knock-on effects in Coder HTTP API (latency etc.).

Our recommended way of doing this is with external provisioners.

However, there is currently no simple path to deploying external provisioners in Helm. Creating an external provisioner currently requires a CODER_TOKEN, which presents us with a chicken-and-egg problem in the current Helm chart.

You can work around this currently by manually applying a deployment manifest with a manually generated coder token, but it would be a nice experience for a Kubernetes operator if this were supported natively in the Helm chart.

Proposed Solution

Per @deansheather's comment in #8183 (comment):

• Add configuration knobs in our current Helm chart to enable creation of an external provisionerd enddpoint, something like:
  • provisionerd.external.enable (create a separate provisionerd deployment)
  • provisionerd.external.replicas (replica count of the provisionerd deployment)
  • provisionerd.external.concurrency (max number of jobs per provisionerd replica)
  • provisionerd.external.resources.{requests,limits} (requests, limits per provisionerd replica)
• Add the ability for external provisioners to authenticate with coder using a pre-shared secret.
• Randomly generate a shared secret between the coder and provisionerd deployments, and pass this into both deployments.