Same with this one @ammario With this sprint so packed to get to Beta and that we have a sprint before the switch happens, I am going to align this with the first sprint of community. That does not mean it can't be picked up before that though. Let me know if you still want to inject this into Beta. CC @tjcran

Hey team! Please add your planning poker estimate with ZenHub @jsjoeio @bpmct

I think this needs some discovery work before it can be estimated. The bug could be a quick 20m fix or a multi-day long architectural revision. It's impossible to tell from the error which is closer.

Removed myself from this one, it's a bit outside my wheelhouse

I'm also not too sure, I took a wild guess but don't have a ton of context on this. I would probably groom this with the Backend team.

I ran into this while troubleshooting an issue with Docker on armv7. On my end, the issue was an architecture mismatch between the system (arm) and the enterprise-image (amd64).

When running the image in Docker directly, the error is clear:

benpotter@elijo:~ $ docker run codercom/enterprise-base:ubuntu
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
standard_init_linux.go:228: exec user process caused: exec format error

However, in Coder(/Terraform?), this error is not displayed:

docker_container.workspace[0]: Creation errored after 2s
Error: container exited immediately

I did a tiny bit of digging, but didn't try directly provisioning in TF to see if more errors/details are shared. I agree this is a confusing debugging experience

Just a recommendation (feature): add docker logs to Coder Terraform logs so that you can debug easily than ever.

Please add your planning poker estimate with ZenHub @kylecarbs

@johnstcn and @mafredri timebox this investigation to see if you can find root cause and get an estimate for points and then we will evaluate if this is still a switch blocker

Hey team! Please add your planning poker estimate with ZenHub @johnstcn @mafredri

Paired with @mafredri -- spent a couple hours and was able to partially reproduce the issue, although there have been several improvements since then.

Also noting I am able to run through the docker-local flow on my M1 mac on latest master (using Colima instead of Docker Desktop). So I'm not sure if this can still be called a switch-blocker.

Steps to reproduce

• Input invalid Terraform to cause the import job to fail (for example, reference a missing variable or provide an invalid parameter).
• Terraform v1.1.9 has some fixes for CLI output (we currently embed 1.1.7).
• We didn't have arm64 scripts for coder agent at the time; I believe this might have been the root cause of the missing variable in the terraform input.
• job has already been marked as failed doesn't appear to happen any more.
• We do now have the Terraform output in the CLI so that at least provides more detail

$ coder templates create
> Create and upload "/tmp/docker-local"? (yes/no)  
✔ Queued [28ms]
✔ Setting up [3ms]
✔ Adding README.md... [0ms]
✔ Parse parameters [2ms]
⧗  Detecting persistent resources 
  Terraform 1.1.7
  Error: Unsupported attribute
  This object has no argument, nested block, or exported attribute named "init_scripttt". Did you mean "init_script"?
✘ Detecting persistent resources [4694ms]
template import provision for start: recv import provision: plan terraform: exit status 1

Fixes

• Improve CLI error message in provisionerd when running import job fails (estimate: 1 point)
• Log provisioner output in coderd server log (we already output it to stdout) (estimate: 2 points) Provisionerd: log provisioner output to provisionerd log #1733
• Improve logging in provisionerd as well, provide more useful information to user (estimate: 1 point) (fix: add stdout to errBuf, make synchronous kylecarbs/terraform-exec#1)
• Update built-in Terraform version; there have been a few CLI fixes since 1.1.7 -- chore: update terraform version to 1.1.9 #1542
• Improve debug ability of coder agent script -- Difficult to debug failures of agent init script in Docker provider #1544

Notes

• Unable to reproduce issue on rPi 3 with latest version (arm64)
• Unable to reproduce with auto-installed terraform on version 8661f92 -- coder server --dev fails to start with the error install terraform: unexpected Content-Type: "application/vnd+hashicorp.releases-api.v0+json"

CC @tjcran @misskniss

@johnstcn chiming in to see if I can help move this forward. Should we turn your other fixes(first 3) into issues and close this?

@jsjoeio I don't think that much overhead is necessary! I was thinking of just attaching PRs to this ticket for each of those.

One thing that helps a bit with troubleshooting this kind of problem: the default Docker logging driver deletes logs as soon as a container is deleted. But if you use the journald driver instead, you can view logs even after container deletion, using journalctl -u docker.service.

Ended up making a separate issue for provisioner job output, as we need to ensure adminstrators can set its verbosity independently #1733

Closing this issue out, as:

• Issue is no longer reproducible on current version
• Root cause of issue is known
• Future work broken out in other issues / PRs (Provisionerd: log provisioner output to provisionerd log #1733, fix: add stdout to errBuf, make synchronous kylecarbs/terraform-exec#1, chore: update terraform version to 1.1.9 #1542, Difficult to debug failures of agent init script in Docker provider #1544)