Skip to content

chore(provisioner/terraform): make testdata generate.sh parallel #16326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 29, 2025
Merged

chore(provisioner/terraform): make testdata generate.sh parallel #16326

merged 1 commit into from
Jan 29, 2025

Conversation

mafredri
Copy link
Member

I got tired of waiting for this script to complete and being filled with scary output, so I made it parallel and hide output on the happy path:

Before:

# Lots and lots of output, not sure what's going on...
./provisioner/terraform/testdata/generate.sh  45.85s user 11.82s system 141% cpu 40.680 total

After:

❯ time ./provisioner/terraform/testdata/generate.sh
== Generating test data for: chaining-resources
== Skipping: cleanup-stale-plugins
== Generating test data for: calling-module
== Generating test data for: display-apps
== Generating test data for: conflicting-resources
== Generating test data for: display-apps-disabled
== Skipping: kubernetes-metadata
== Generating test data for: git-auth-providers
== Generating test data for: instance-id
== Generating test data for: external-auth-providers
== Generating test data for: mapped-apps
== Generating test data for: multiple-agents-multiple-apps
== Generating test data for: multiple-agents-multiple-envs
== Generating test data for: multiple-agents-multiple-scripts
== Generating test data for: multiple-apps
== Generating test data for: multiple-agents
== Generating test data for: resource-metadata-duplicate
== Generating test data for: resource-metadata
== Generating test data for: rich-parameters
== Generating test data for: rich-parameters-order
== Skipping: timings-aggregation
== Generating test data for: rich-parameters-validation
== Done generating test data for: display-apps-disabled
== Done generating test data for: external-auth-providers
== Done generating test data for: resource-metadata-duplicate
== Done generating test data for: mapped-apps
== Done generating test data for: display-apps
== Done generating test data for: instance-id
== Done generating test data for: multiple-agents-multiple-scripts
== Done generating test data for: git-auth-providers
== Done generating test data for: conflicting-resources
== Done generating test data for: multiple-apps
== Done generating test data for: rich-parameters-order
== Done generating test data for: resource-metadata
== Done generating test data for: chaining-resources
== Done generating test data for: multiple-agents
== Done generating test data for: multiple-agents-multiple-apps
== Done generating test data for: multiple-agents-multiple-envs
== Done generating test data for: calling-module
== Done generating test data for: rich-parameters-validation
== Done generating test data for: rich-parameters
./provisioner/terraform/testdata/generate.sh  35.37s user 10.56s system 2087% cpu 2.200 total

Unhappy path:

❯ time ./provisioner/terraform/testdata/generate.sh
== Generating test data for: calling-module
== Generating test data for: conflicting-resources
== Skipping: cleanup-stale-plugins
== Generating test data for: chaining-resources
== Generating test data for: display-apps
== Generating test data for: display-apps-disabled
== Generating test data for: external-auth-providers
== Generating test data for: mapped-apps
== Skipping: kubernetes-metadata
== Generating test data for: git-auth-providers
== Generating test data for: instance-id
== Generating test data for: multiple-agents-multiple-apps
== Generating test data for: multiple-agents-multiple-envs
== Generating test data for: multiple-agents-multiple-scripts
== Generating test data for: multiple-apps
== Generating test data for: multiple-agents
== Generating test data for: resource-metadata-assignment
== Generating test data for: resource-metadata
== Generating test data for: resource-metadata-duplicate
== Generating test data for: rich-parameters-order
== Skipping: timings-aggregation
== Generating test data for: rich-parameters-validation
== Generating test data for: rich-parameters
=== BEGIN: resource-metadata-assignment
Terraform initialized in an empty directory!

The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
╷
│ Error: No configuration files
│
│ Plan requires configuration to be present. Planning without a configuration
│ would mark everything for destruction, which is normally not what is
│ desired. If you would like to destroy everything, run plan with the
│ -destroy option. Otherwise, create a Terraform configuration file (.tf
│ file) and try again.
╵
=== END: resource-metadata-assignment
== Error generating test data for: resource-metadata-assignment
== Done generating test data for: display-apps-disabled
== Done generating test data for: multiple-agents-multiple-apps
== Done generating test data for: instance-id
== Done generating test data for: external-auth-providers
== Done generating test data for: multiple-apps
== Done generating test data for: rich-parameters-order
== Done generating test data for: resource-metadata
== Done generating test data for: display-apps
== Done generating test data for: conflicting-resources
== Done generating test data for: git-auth-providers
== Done generating test data for: multiple-agents
== Done generating test data for: multiple-agents-multiple-scripts
== Done generating test data for: multiple-agents-multiple-envs
== Done generating test data for: mapped-apps
== Done generating test data for: resource-metadata-duplicate
== Done generating test data for: chaining-resources
== Done generating test data for: calling-module
== Done generating test data for: rich-parameters-validation
== Done generating test data for: rich-parameters
ERROR: Failed to generate test data for 1 modules
./provisioner/terraform/testdata/generate.sh  35.37s user 10.54s system 2091% cpu 2.195 total

@mafredri mafredri requested review from johnstcn and SasSwart January 29, 2025 13:50
johnstcn
johnstcn reviewed Jan 29, 2025
View reviewed changes
Copy link
Member

@johnstcn johnstcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

provisioner/terraform/testdata/generate.sh
Comment on lines +10 to +18
terraform init -upgrade &&
terraform plan -out terraform.tfplan &&
terraform show -json ./terraform.tfplan | jq >"$name".tfplan.json &&
terraform graph -type=plan >"$name".tfplan.dot &&
rm terraform.tfplan &&
terraform apply -auto-approve &&
terraform show -json ./terraform.tfstate | jq >"$name".tfstate.json &&
rm terraform.tfstate &&
terraform graph -type=plan >"$name".tfstate.dot
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the &&'s necessary if we have -e set?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't be, but I was having trouble getting this working without them. I might've had some other issue before that caused it though so I can retry.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, printing "end" is dependent on the chaining, which makes it easier to discern what output belongs where.

@mafredri mafredri merged commit 760a70d into main Jan 29, 2025
33 checks passed
@mafredri mafredri deleted the mafredri/chore-make-provisioner-tf-generator-parallel branch January 29, 2025 15:24
@github-actions github-actions bot locked and limited conversation to collaborators Jan 29, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@johnstcn johnstcn johnstcn approved these changes

@SasSwart SasSwart Awaiting requested review from SasSwart

Assignees

@mafredri mafredri

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mafredri @johnstcn