From last week's All Hands, Ammar highlighted the need for shorter workspace startup times to unlock some new use-cases such as coder run.

• First, I want to enumerate the phases of workspace creation, and visualize/capture how the time was spent at each stage (if we can get down to the provider / resource level, that would be ideal).
  • Even if we optimize as much as we can, if a template uses some super slow cloud API it'll negatively affect startup time - so at least we should give users/operator insight into where the time is being spent
  • We already have distributed traces in place in our provisioners, so we may just be able to use that data.
• I want to investigate the use of tf-exec, instead of spawning new terraform processes in provisioner/terraform/executor.go. Given the uncertain state of Terraform, we can look at using OpenTofu's fork - however there is some question about whether this library will be maintained.
• Lastly I want to look at ways we can tune terraform to run faster
  • plugin cache: on my machine, I was able to shave 10s off a very basic Docker template by using the plugin cache; we already make use of this automatically but it only works on Linux and it appears to not be operational on dogfood at first glance, or least not used in the apply stage
    
  • parallelism: terraform has a -parallelism flag, which is set to 10. I imagine that terraform will mostly be blocked on network requests, which means we can bump up the parallelism quite a lot. terraform is not a very CPU-intensive program, and therefore having 10s or even 100s of goroutines handling API requests should not impede performance but in fact improve it
    
    Provisioners use barely any CPU (CPU time in this graph is cumulative across all CPU modes) (source)