Template create tar stdin option hangs and exits #7043

onyi-ada · 2023-04-07T02:38:09Z

In creating a template using the using '-' to read tar from stdin with the -d/--directory option, the output hangs on
⧗ Queued then exits without creating the template or displaying any error output.

The command used is what is outlined here Support terraform resources outside the given template directory

version v0.21.3

The text was updated successfully, but these errors were encountered:

matifali · 2023-04-08T07:36:18Z

Can you set CODER_VERBOSE=true and try again. Please share any logs.

onyi-ada · 2023-04-10T14:21:06Z

CODER_VERBOSE=true

With that, I get an output that indicates that the terraform isn't there, even though it is. Running without the tar option and simply pointing -d to the directory does create a template.

$ CODER_VERBOSE=true tar -cvh dev_templates/aws-linux | coder templates create -d - test-dev-template
dev_templates/aws-linux/
dev_templates/aws-linux/main.tf
dev_templates/aws-linux/README.md
✔ Queued [955ms]
✔ Running [87ms]
✔ Setting up [0ms]
✔ Parsing template parameters [37ms]
⧗  Detecting persistent resources
  terraform environment variable: TF_LOG=INFO
  terraform environment variable: TF_LOG_PATH=/var/log/terraform.log
  Terraform 1.3.4
  Error: No configuration files
  Plan requires configuration to be present. Planning without a configuration would mark everything for destruction, which is normally not what is desired. If you would like to destroy everything, run plan with the -destroy option. Otherwise, create a Terraform configuration file (.tf file) and try again.
✔ Detecting persistent resources [77ms]
✘ Cleaning Up [121ms]
template import provision for start: recv import provision: plan terraform: terraform plan: exit status 1

matifali · 2023-04-10T14:41:06Z

main.tf file should exist in the root of the tar you are creating.

onyi-ada · 2023-04-10T16:59:52Z

main.tf file should exist in the root of the tar you are creating.

The main.tf file is already the root. I re-initialized terraform, this is the output

$ CODER_VERBOSE=true tar -cvh dev_templates/aws-linux | coder templates create -d - test-dev-template
dev_templates/aws-linux/
dev_templates/aws-linux/.terraform/
dev_templates/aws-linux/.terraform/providers/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/coder/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/coder/coder/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/coder/coder/0.6.23/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/terraform-provider-coder_v0.6.23
◳ Uploading directory...dev_templates/aws-linux/.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/README.md
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/LICENSE
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/hashicorp/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/hashicorp/aws/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/linux_amd64/
dev_templates/aws-linux/.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/linux_amd64/terraform-provider-aws_v4.62.0_x5
Failed to read file from request.

matifali · 2023-04-10T17:55:07Z

@ammario can you take a look what is happenings here.

ammario · 2023-04-10T19:01:52Z

The problem here is that tar is keeping the dev_templates/aws-linux prefix on all files, so there isn't a main.tf in the root of the archive. I suggest using the -C flag like this:

tar -cvh -C dev_templates/aws-linux . | coder templates create -d - test-dev-template

onyi-ada · 2023-04-10T19:32:25Z

The problem here is that tar is keeping the dev_templates/aws-linux prefix on all files, so there isn't a main.tf in the root of the archive. I suggest using the -C flag like this:
tar -cvh -C dev_templates/aws-linux . | coder templates create -d - test-dev-template

That produces the same result

$ tar -cvh -C dev_templates/aws-linux . | coder templates create -d - test-dev-template
./
./.terraform/
./.terraform/providers/
./.terraform/providers/registry.terraform.io/
./.terraform/providers/registry.terraform.io/coder/
./.terraform/providers/registry.terraform.io/coder/coder/
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/terraform-provider-coder_v0.6.23
◳ Uploading directory..../.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/README.md
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/LICENSE
./.terraform/providers/registry.terraform.io/hashicorp/
./.terraform/providers/registry.terraform.io/hashicorp/aws/
./.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/
./.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/linux_amd64/
./.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/linux_amd64/terraform-provider-aws_v4.62.0_x5
Failed to read file from request.

ammario · 2023-04-10T20:03:53Z

Hmm, well the error has changed slightly. Now it's in the correct directory but I think the template is too big due to the .terraform folder. (#7071)

Try adding a --exclude='^\.' to your tar invocation. E.g.:

tar -cvh --exclude='^\.' -C dev_templates/aws-linux . | coder templates create -d - test-dev-template

onyi-ada · 2023-04-10T20:34:08Z

Hmm, well the error has changed slightly. Now it's in the correct directory but I think the template is too big due to the .terraform folder. (#7071)

Try adding a --exclude='^\.' to your tar invocation. E.g.:
tar -cvh --exclude='^\.' -C dev_templates/aws-linux . | coder templates create -d - test-dev-template

I see the same error Failed to read file from request.

ammario · 2023-04-10T21:22:11Z

Can you send the full error again? Is it still tarring the '.terraform' directory? Can you try tarring into a file with the same exact command and tell me the size?

onyi-ada · 2023-04-10T21:29:07Z

Yes, this is the full error, same as above

$ tar -cvh --exclude='^\.' -C dev_templates/aws-linux . | CODER_VERBOSE=true coder templates create -d - test-dev-template
./
./.terraform/
./.terraform/providers/
./.terraform/providers/registry.terraform.io/
./.terraform/providers/registry.terraform.io/coder/
./.terraform/providers/registry.terraform.io/coder/coder/
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/terraform-provider-coder_v0.6.23
◱ Uploading directory..../.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/README.md
./.terraform/providers/registry.terraform.io/coder/coder/0.6.23/linux_amd64/LICENSE
./.terraform/providers/registry.terraform.io/hashicorp/
./.terraform/providers/registry.terraform.io/hashicorp/aws/
./.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/
./.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/linux_amd64/
./.terraform/providers/registry.terraform.io/hashicorp/aws/4.62.0/linux_amd64/terraform-provider-aws_v4.62.0_x5
Failed to read file from request.

I've tarred the directory, it's about 80M
80M Apr 10 21:26 aws_linux.tar.gz

ammario · 2023-04-10T21:31:46Z

Ah ok, your goal should be to play around with the tar flags to reduce the size of the archive to less than a megabyte. The .terraform directory is probably the main problem.

I'm going to be out of pocket for a while, feel free to ping me on Discord, but may not be responsive here.

onyi-ada · 2023-04-11T00:33:03Z

Ah ok, your goal should be to play around with the tar flags to reduce the size of the archive to less than a megabyte. The .terraform directory is probably the main problem.

I'm going to be out of pocket for a while, feel free to ping me on Discord, but may not be responsive here.

I've reduced the size to less than a megabyte. It hangs on queued, then exits

3.0K Apr 11 00:27 aws_linux.tar.gz

$ tar --exclude='.[^/]*' -cvh -C dev_templates/aws-linux . | CODER_VERBOSE=true coder templates create -d - test-dev-template
./
./main.tf
./README.md
⧗  Queued

$

ammario · 2023-04-11T00:37:57Z

Ok, would you mind uploading the template archive here (with any secrets stripped) so that I can try to reproduce?

onyi-ada · 2023-04-11T00:49:32Z

aws_linux.tar.gz

Ok, would you mind uploading the template archive here (with any secrets stripped) so that I can try to reproduce?

onyi-ada · 2023-04-17T23:32:56Z

@ammario Were you able to reproduce the issue?

ammario · 2023-04-18T01:11:34Z

I was able to run:

tar -C dev_templates/aws-linux  -cvh . | coder templates create -d - debug

with the provided archive with no issue. Is it possible you have a load balancer between you and the coder server that is causing the request to fail?

onyi-ada · 2023-04-18T16:58:24Z

I was able to run:
tar -C dev_templates/aws-linux  -cvh . | coder templates create -d - debug
with the provided archive with no issue. Is it possible you have a load balancer between you and the coder server that is causing the request to fail?

We do have a load balancer.
We are able to create templates without the tar option though, so I'm wondering what issue using the load balancer would be causing?

ammario · 2023-04-18T18:54:19Z

@onyi-ada

Try again using the --debug-http flag introduced here in your templates create invocation. You can install that version of the binary with:

git clone https://github.com/coder/coder
cd coder
go install enterprise/cmd/coder
~/go/bin/coder

Note that the output will contain your session tokens so be careful about sharing. Hopefully it will show us something useful.

onyi-ada · 2023-04-21T20:48:46Z

@ammario It appears the latest version of Coder allows us to get further. I'll close the issue for now and report back if we are still running into an issue with this feature. Thanks for your support

ammario mentioned this issue Apr 18, 2023

feat(cli): add --debug-http flag #7192

Merged

onyi-ada closed this as completed Apr 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Template create tar stdin option hangs and exits #7043

Template create tar stdin option hangs and exits #7043

onyi-ada commented Apr 7, 2023 •

edited

Loading

matifali commented Apr 8, 2023

onyi-ada commented Apr 10, 2023 •

edited

Loading

matifali commented Apr 10, 2023

onyi-ada commented Apr 10, 2023

matifali commented Apr 10, 2023

ammario commented Apr 10, 2023

onyi-ada commented Apr 10, 2023

ammario commented Apr 10, 2023

onyi-ada commented Apr 10, 2023 •

edited

Loading

ammario commented Apr 10, 2023

onyi-ada commented Apr 10, 2023 •

edited

Loading

ammario commented Apr 10, 2023

onyi-ada commented Apr 11, 2023

ammario commented Apr 11, 2023

onyi-ada commented Apr 11, 2023

onyi-ada commented Apr 17, 2023

ammario commented Apr 18, 2023

onyi-ada commented Apr 18, 2023

ammario commented Apr 18, 2023 •

edited

Loading

onyi-ada commented Apr 21, 2023 •

edited

Loading

Template create tar stdin option hangs and exits #7043

Template create tar stdin option hangs and exits #7043

Comments

onyi-ada commented Apr 7, 2023 • edited Loading

matifali commented Apr 8, 2023

onyi-ada commented Apr 10, 2023 • edited Loading

matifali commented Apr 10, 2023

onyi-ada commented Apr 10, 2023

matifali commented Apr 10, 2023

ammario commented Apr 10, 2023

onyi-ada commented Apr 10, 2023

ammario commented Apr 10, 2023

onyi-ada commented Apr 10, 2023 • edited Loading

ammario commented Apr 10, 2023

onyi-ada commented Apr 10, 2023 • edited Loading

ammario commented Apr 10, 2023

onyi-ada commented Apr 11, 2023

ammario commented Apr 11, 2023

onyi-ada commented Apr 11, 2023

onyi-ada commented Apr 17, 2023

ammario commented Apr 18, 2023

onyi-ada commented Apr 18, 2023

ammario commented Apr 18, 2023 • edited Loading

onyi-ada commented Apr 21, 2023 • edited Loading

onyi-ada commented Apr 7, 2023 •

edited

Loading

onyi-ada commented Apr 10, 2023 •

edited

Loading

onyi-ada commented Apr 10, 2023 •

edited

Loading

onyi-ada commented Apr 10, 2023 •

edited

Loading

ammario commented Apr 18, 2023 •

edited

Loading

onyi-ada commented Apr 21, 2023 •

edited

Loading