Terraform plugins use https://github.com/hashicorp/go-plugin to communicate with Terraform. Plugins themselves are gRPC services that listen on a Unix domain socket in a temp directory (or on localhost on Windows). Terraform is the "client" of the gRPC service, but manages the lifecycle of the "service".

1. Terraform starts the provider binary as a subcommand
2. The provider starts a Unix domain socket in a temp directory
3. The provider writes the socket path to stdout (along with things like protocol version)
4. Terraform reads the socket path over the pipe to the provider process
5. Terraform connects to the gRPC server over the domain, and uses the provider via it's gRPC API

At the end of the day, Terraform is supposed to call a special API to tell the provider to shut down, and/or forcibly kill the child process. However, it seems that sometimes that's not happening, and the provider process can linger, just listening on its domain socket.

Some ideas:

If we get "text file busy" error:

1. we could try to find (via ps or the /proc system) the offending process and kill it
2. we could attempt to connect to its unix domain socket and send the Shutdown command --- might be complicated by authentication protocols (go-plugin supports an "auto mTLS" setting that ensures only the original client can connect).
3. we could just have that particular provisioner daemon exit. If it's external, then the cluster manager (e.g. K8s) can restart it. If its in-process with Coderd, then we could have some threshold of killed provisioner daemons that triggers coderd to also exit and be restarted

We could fix Terraform and/or OpenTofu such that they don't reinstall the provider binary if it already exists (possibly including hashing contents). This would sidestep the issue, since we don't write to the file. However, the underlying issue of leaking provider processes would remain.

I'll also check whether Terraform is setting Pdeathsignal in https://pkg.go.dev/syscall#SysProcAttr

UPDATE: Terraform doesn't set any special SysProcAttr, so I think that just means the child gets SIGHUP by default if the parent dies without sending it a signal.

I don't think I have the full story yet, but I've just confirmed that if you specify a provider_installation -> filesystem_mirror in your .terraformrc, then when "installing" providers, Terraform just creates symlinks to your mirror. That means that when running multiple provisionerds on a single host, they're not actually being isolated at the filesystem level, even though we take some care to give them unique cache directories.

If you use the unpacked layout, Terraform will attempt to create a symbolic link to the mirror directory when installing the provider, rather than creating a deep copy of the directory. The packed layout prevents this because Terraform must extract the zip file during installation.

https://developer.hashicorp.com/terraform/cli/config/config-file#filesystem_mirror

Pretty sure at least one customer who is seeing this issue is using a filesystem_mirror, we should confirm with others.

One of the customers seeing this is using the zipped layout for their filesystem mirror. That explains why the provider gets opened for writing (unzipping the provider package), but doesn't explain why the provider is still executing, since the symlink stuff I mentioned only applies to unpacked layout.

I've confirmed that the output of lsof on Busybox which the customer is using lists the real path of the executable, and it shows that the cache directory is not symlinked to the filesystem mirror. So symlinking to the mirror doesn't explain how they saw text file busy.

I'm following up with the other customers who have seen the issue, so see what kind of layout they are using.

I'm back to thinking that our provider must be still running from a previous build on the same provisionerd. Not sure how this could happen.

Another customer has said they aren't using a filesystem_mirror, so that discredits the symlinking theory even more.

I also get this error:
Initializing the backend...
Initializing provider plugins...

• Finding latest version of hashicorp/kubernetes...
• Finding latest version of coder/coder...
• Installing hashicorp/kubernetes v2.32.0...
• Installed hashicorp/kubernetes v2.32.0 (signed by HashiCorp)
• Installing coder/coder v1.0.3...
  Error: Failed to install provider
  Error while installing coder/coder v1.0.3: open
  /home/coder/.cache/coder/provisioner-0/tf/registry.terraform.io/coder/coder/1.0.3/linux_amd64/terraform-provider-coder_v1.0.3:
  text file busy

this is the tf file:
main.tf.txt

and here are the coderd provisioner logs (busy error):
coderd logs.txt

Well, I guess that means coder/terraform-provider-coder#290 didn't help, since that was released in v1.0.3

@datapedd what was the workspace ID of the build that failed with "text file busy"?

Is there any chance you could exec into the coderd pod and see if /home/coder/.cache/coder/provisioner-0/tf/registry.terraform.io/coder/coder/1.0.3/linux_amd64/terraform-provider-coder_v1.0.3 is still running, and if so, get a core dump?

In coderd pod there is the provisioner running (ps aux). Cant create a dump as read only it say. After I killed the process for the provisioner it worked again.

Maybe related to some pod failing during the bash script to get released by terraform?

@datapedd what was the exact time the build that failed started? You should be able to see this by clicking the builds icon on the left:

Then, click the failed build.