Skip to content

chore: persist template import terraform plan in postgres #17012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Mar 24, 2025

Conversation

aslilac
Copy link
Member

@aslilac aslilac commented Mar 19, 2025

Adds plumbing to persist terraform plan data from template import provisioner builds.


After some deliberation, I went with storing what I have called the cached_plan in a separate table because these values are incredibly large, the template_versions table already has a lot of columns, and a lot of queries use template_version.* to get those many columns. Accidentally including the cached_plan column in those queries would be very undesirable, and this value mostly exists for future use on the backend. I don't think this is the sort of thing we would ever actually want to send to a client, except maybe a page for debugging the current state.

This also means that we can track it's updated_at time separately quite easily, since we're planning on letting it be "refreshed".

Closes #17050

@aslilac aslilac changed the title chore: persist import terraform graph in postgres chore: persist template import terraform plan in postgres Mar 20, 2025
@aslilac aslilac marked this pull request as ready for review March 20, 2025 21:34
@Emyrk
Copy link
Member

Emyrk commented Mar 21, 2025

The purpose of this is not widely known, so just dropping a note.

The output of terraform plan is going to be used to support dynamic parameters. Specifically, we are keeping the raw outputs (rather than preparsing) as an input to the new parameter engine.

The phasing in of this new feature is that it will live alongside the existing flow for some time. So this new table will be write-only for production instances until we get a bit further with the parameter engine experiment.

Copy link
Member

@Emyrk Emyrk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all looking good. I am wondering if in the future we will need to implement a way to garbage collect up this data.

We might be able to do some trimming in the future, but keeping the raw plans gives us the most flexibility in the future.

EDIT: We store the original TF too, so versions are already bloated

@aslilac aslilac requested a review from Emyrk March 21, 2025 18:57
@aslilac aslilac merged commit 5b3eda6 into main Mar 24, 2025
35 checks passed
@aslilac aslilac deleted the lilac/import-graph branch March 24, 2025 16:01
@github-actions github-actions bot locked and limited conversation to collaborators Mar 24, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Save terraform plan output json to database for dynamic param use
2 participants