Skip to content

feat(provisioner/terraform/tfparse): add support for built-in Terraform functions #16183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 20, 2025
Merged

feat(provisioner/terraform/tfparse): add support for built-in Terraform functions #16183

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2eedbfc
feat(tfparse): add support for built-in Terraform functions
johnstcn Jan 17, 2025
7a2b804
fixup! feat(tfparse): add support for built-in Terraform functions
johnstcn Jan 17, 2025
dd52290
rename unsafeFileFunctions -> stubFileFunctions
johnstcn Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions coderd/templateversions_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,11 +489,11 @@ func TestPostTemplateVersionsByOrganization(t *testing.T) {
"foo": "bar",
"a": var.a,
"b": data.coder_parameter.b.value,
"test": try(null_resource.test.name, "whatever"),
"test": pathexpand("~/file.txt"),
}
}`,
},
expectError: `Function calls not allowed; Functions may not be called here.`,
expectError: `function "pathexpand" may not be used here`,
},
// We will allow coder_workspace_tags to set the scope on a template version import job
// BUT the user ID will be ultimately determined by the API key in the scope.
Expand Down
7 changes: 7 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,11 @@ require (
sigs.k8s.io/yaml v1.4.0 // indirect
)

require (
github.com/aquasecurity/trivy-iac v0.8.0
github.com/zclconf/go-cty-yaml v1.0.3
)

require (
github.com/DataDog/datadog-agent/pkg/proto v0.58.0 // indirect
github.com/DataDog/datadog-agent/pkg/trace v0.58.0 // indirect
Expand All @@ -445,6 +450,8 @@ require (
github.com/DataDog/go-runtime-metrics-internal v0.0.0-20241106155157-194426bbbd59 // indirect
github.com/DataDog/go-sqllexer v0.0.14 // indirect
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/attributes v0.20.0 // indirect
github.com/apparentlymart/go-cidr v1.1.0 // indirect
github.com/bmatcuk/doublestar/v4 v4.6.1 // indirect
github.com/cihub/seelog v0.0.0-20170130134532-f561c5e57575 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/lufia/plan9stats v0.0.0-20220913051719-115f729f3c8c // indirect
Expand Down
8 changes: 8 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,13 @@ github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7X
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be/go.mod h1:ySMOLuWl6zY27l47sB3qLNK6tF2fkHG55UZxx8oIVo4=
github.com/apparentlymart/go-cidr v1.1.0 h1:2mAhrMoF+nhXqxTzSZMUzDHkLjmIHC+Zzn4tdgBZjnU=
github.com/apparentlymart/go-cidr v1.1.0/go.mod h1:EBcsNrHc3zQeuaeCeCtQruQm+n9/YjEn/vI25Lg7Gwc=
github.com/apparentlymart/go-textseg/v12 v12.0.0/go.mod h1:S/4uRK2UtaQttw1GenVJEynmyUenKwP++x/+DdGV/Ec=
github.com/apparentlymart/go-textseg/v15 v15.0.0 h1:uYvfpb3DyLSCGWnctWKGj857c6ew1u1fNQOlOtuGxQY=
github.com/apparentlymart/go-textseg/v15 v15.0.0/go.mod h1:K8XmNZdhEBkdlyDdvbmmsvpAG721bKi0joRfFdHIWJ4=
github.com/aquasecurity/trivy-iac v0.8.0 h1:NKFhk/BTwQ0jIh4t74V8+6UIGUvPlaxO9HPlSMQi3fo=
github.com/aquasecurity/trivy-iac v0.8.0/go.mod h1:ARiMeNqcaVWOXJmp8hmtMnNm/Jd836IOmDBUW5r4KEk=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 h1:jfIu9sQUG6Ig+0+Ap1h4unLjW6YQJpKZVmUzxsD4E/Q=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0/go.mod h1:t2tdKJDJF9BV14lnkjHmOQgcvEKgtqs5a1N3LNdJhGE=
github.com/armon/circbuf v0.0.0-20190214190532-5111143e8da2 h1:7Ip0wMmLHLRJdrloDxZfhMm0xrLXZS8+COSu2bXmEQs=
Expand Down Expand Up @@ -167,6 +171,8 @@ github.com/bep/tmc v0.5.1/go.mod h1:tGYHN8fS85aJPhDLgXETVKp+PR382OvFi2+q2GkGsq0=
github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
github.com/bgentry/speakeasy v0.2.0 h1:tgObeVOf8WAvtuAX6DhJ4xks4CFNwPDZiqzGqIHE51E=
github.com/bgentry/speakeasy v0.2.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
github.com/bmatcuk/doublestar/v4 v4.6.1 h1:FH9SifrbvJhnlQpztAx++wlkk70QBf0iBWDwNy7PA4I=
github.com/bmatcuk/doublestar/v4 v4.6.1/go.mod h1:xBQ8jztBU6kakFMg+8WGxn0c6z1fTSPVIjEY1Wr7jzc=
github.com/bool64/shared v0.1.5 h1:fp3eUhBsrSjNCQPcSdQqZxxh9bBwrYiZ+zOKFkM0/2E=
github.com/bool64/shared v0.1.5/go.mod h1:081yz68YC9jeFB3+Bbmno2RFWvGKv1lPKkMP6MHJlPs=
github.com/bramvdbogaerde/go-scp v1.5.0 h1:a9BinAjTfQh273eh7vd3qUgmBC+bx+3TRDtkZWmIpzM=
Expand Down Expand Up @@ -965,6 +971,8 @@ github.com/zclconf/go-cty v1.16.0 h1:xPKEhst+BW5D0wxebMZkxgapvOE/dw7bFTlgSc9nD6w
github.com/zclconf/go-cty v1.16.0/go.mod h1:VvMs5i0vgZdhYawQNq5kePSpLAoz8u1xvZgrPIxfnZE=
github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940 h1:4r45xpDWB6ZMSMNJFMOjqrGHynW3DIBuR2H9j0ug+Mo=
github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940/go.mod h1:CmBdvvj3nqzfzJ6nTCIwDTPZ56aVGvDrmztiO5g3qrM=
github.com/zclconf/go-cty-yaml v1.0.3 h1:og/eOQ7lvA/WWhHGFETVWNduJM7Rjsv2RRpx1sdFMLc=
github.com/zclconf/go-cty-yaml v1.0.3/go.mod h1:9YLUH4g7lOhVWqUbctnVlZ5KLpg7JAprQNgxSZ1Gyxs=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
github.com/zeebo/errs v1.3.0 h1:hmiaKqgYZzcVgRL1Vkc1Mn2914BbzB0IBxs+ebeutGs=
Expand Down
162 changes: 162 additions & 0 deletions provisioner/terraform/tfparse/funcs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
package tfparse

import (
"github.com/aquasecurity/trivy-iac/pkg/scanners/terraform/parser/funcs"
"github.com/hashicorp/hcl/v2/ext/tryfunc"
ctyyaml "github.com/zclconf/go-cty-yaml"
"github.com/zclconf/go-cty/cty"
"github.com/zclconf/go-cty/cty/function"
"github.com/zclconf/go-cty/cty/function/stdlib"
"golang.org/x/xerrors"
)

// Functions returns a set of functions that are safe to use in the context of
// evaluating Terraform expressions without any ability to reference local files.
// Functions that refer to file operations are replaced with stubs that return a
// descriptive error to the user.
func Functions() map[string]function.Function {
return allFunctions
}

var (
// Adapted from github.com/aquasecurity/trivy-iac@v0.8.0/pkg/scanners/terraform/parser/functions.go
// We cannot support all available functions here, as the result of reading a file will be different
// depending on the execution environment.
safeFunctions = map[string]function.Function{
"abs": stdlib.AbsoluteFunc,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-merge thought: how do we align Terraform versions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I could tell it doesn't look like these functions change all that often.

However, f Terraform adds a new function in future, we'll need to add support for it ourselves. Then again, so will aquasec/trivy-iac. This is an unfortunate consequence of Terraform having made the bulk of their internal functions un-importable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, I think we're fine here and we can just update the lib.
My only concern is if they change the semantics of the functions between versions, but I doubt they'd do that.

"basename": funcs.BasenameFunc,
"base64decode": funcs.Base64DecodeFunc,
"base64encode": funcs.Base64EncodeFunc,
"base64gzip": funcs.Base64GzipFunc,
"base64sha256": funcs.Base64Sha256Func,
"base64sha512": funcs.Base64Sha512Func,
"bcrypt": funcs.BcryptFunc,
"can": tryfunc.CanFunc,
"ceil": stdlib.CeilFunc,
"chomp": stdlib.ChompFunc,
"cidrhost": funcs.CidrHostFunc,
"cidrnetmask": funcs.CidrNetmaskFunc,
"cidrsubnet": funcs.CidrSubnetFunc,
"cidrsubnets": funcs.CidrSubnetsFunc,
"coalesce": funcs.CoalesceFunc,
"coalescelist": stdlib.CoalesceListFunc,
"compact": stdlib.CompactFunc,
"concat": stdlib.ConcatFunc,
"contains": stdlib.ContainsFunc,
"csvdecode": stdlib.CSVDecodeFunc,
"dirname": funcs.DirnameFunc,
"distinct": stdlib.DistinctFunc,
"element": stdlib.ElementFunc,
"chunklist": stdlib.ChunklistFunc,
"flatten": stdlib.FlattenFunc,
"floor": stdlib.FloorFunc,
"format": stdlib.FormatFunc,
"formatdate": stdlib.FormatDateFunc,
"formatlist": stdlib.FormatListFunc,
"indent": stdlib.IndentFunc,
"index": funcs.IndexFunc, // stdlib.IndexFunc is not compatible
"join": stdlib.JoinFunc,
"jsondecode": stdlib.JSONDecodeFunc,
"jsonencode": stdlib.JSONEncodeFunc,
"keys": stdlib.KeysFunc,
"length": funcs.LengthFunc,
"list": funcs.ListFunc,
"log": stdlib.LogFunc,
"lookup": funcs.LookupFunc,
"lower": stdlib.LowerFunc,
"map": funcs.MapFunc,
"matchkeys": funcs.MatchkeysFunc,
"max": stdlib.MaxFunc,
"md5": funcs.Md5Func,
"merge": stdlib.MergeFunc,
"min": stdlib.MinFunc,
"parseint": stdlib.ParseIntFunc,
"pow": stdlib.PowFunc,
"range": stdlib.RangeFunc,
"regex": stdlib.RegexFunc,
"regexall": stdlib.RegexAllFunc,
"replace": funcs.ReplaceFunc,
"reverse": stdlib.ReverseListFunc,
"rsadecrypt": funcs.RsaDecryptFunc,
"setintersection": stdlib.SetIntersectionFunc,
"setproduct": stdlib.SetProductFunc,
"setsubtract": stdlib.SetSubtractFunc,
"setunion": stdlib.SetUnionFunc,
"sha1": funcs.Sha1Func,
"sha256": funcs.Sha256Func,
"sha512": funcs.Sha512Func,
"signum": stdlib.SignumFunc,
"slice": stdlib.SliceFunc,
"sort": stdlib.SortFunc,
"split": stdlib.SplitFunc,
"strrev": stdlib.ReverseFunc,
"substr": stdlib.SubstrFunc,
"timestamp": funcs.TimestampFunc,
"timeadd": stdlib.TimeAddFunc,
"title": stdlib.TitleFunc,
"tostring": funcs.MakeToFunc(cty.String),
"tonumber": funcs.MakeToFunc(cty.Number),
"tobool": funcs.MakeToFunc(cty.Bool),
"toset": funcs.MakeToFunc(cty.Set(cty.DynamicPseudoType)),
"tolist": funcs.MakeToFunc(cty.List(cty.DynamicPseudoType)),
"tomap": funcs.MakeToFunc(cty.Map(cty.DynamicPseudoType)),
"transpose": funcs.TransposeFunc,
"trim": stdlib.TrimFunc,
"trimprefix": stdlib.TrimPrefixFunc,
"trimspace": stdlib.TrimSpaceFunc,
"trimsuffix": stdlib.TrimSuffixFunc,
"try": tryfunc.TryFunc,
"upper": stdlib.UpperFunc,
"urlencode": funcs.URLEncodeFunc,
"uuid": funcs.UUIDFunc,
"uuidv5": funcs.UUIDV5Func,
"values": stdlib.ValuesFunc,
"yamldecode": ctyyaml.YAMLDecodeFunc,
"yamlencode": ctyyaml.YAMLEncodeFunc,
"zipmap": stdlib.ZipmapFunc,
}

// the below functions are not safe for usage in the context of tfparse, as their return
// values may change depending on the underlying filesystem.
stubFileFunctions = map[string]function.Function{
"abspath": makeStubFunction("abspath", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"file": makeStubFunction("file", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"fileexists": makeStubFunction("fileexists", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"fileset": makeStubFunction("fileset", cty.String, function.Parameter{Name: "path", Type: cty.String}, function.Parameter{Name: "pattern", Type: cty.String}),
"filebase64": makeStubFunction("filebase64", cty.String, function.Parameter{Name: "path", Type: cty.String}, function.Parameter{Name: "pattern", Type: cty.String}),
"filebase64sha256": makeStubFunction("filebase64sha256", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"filebase64sha512": makeStubFunction("filebase64sha512", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"filemd5": makeStubFunction("filemd5", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"filesha1": makeStubFunction("filesha1", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"filesha256": makeStubFunction("filesha256", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"filesha512": makeStubFunction("filesha512", cty.String, function.Parameter{Name: "path", Type: cty.String}),
"pathexpand": makeStubFunction("pathexpand", cty.String, function.Parameter{Name: "path", Type: cty.String}),
}

allFunctions = mergeMaps(safeFunctions, stubFileFunctions)
)

// mergeMaps returns a new map which is the result of merging each key and value
// of all maps in ms, in order. Successive maps may override values of previous
// maps.
func mergeMaps[K, V comparable](ms ...map[K]V) map[K]V {
merged := make(map[K]V)
for _, m := range ms {
for k, v := range m {
merged[k] = v
}
}
return merged
}

// makeStubFunction returns a function.Function with the required return type and parameters
// that will always return an unknown type and an error.
func makeStubFunction(name string, returnType cty.Type, params ...function.Parameter) function.Function {
var spec function.Spec
spec.Params = params
spec.Type = function.StaticReturnType(returnType)
spec.Impl = func(_ []cty.Value, _ cty.Type) (cty.Value, error) {
return cty.UnknownVal(returnType), xerrors.Errorf("function %q may not be used here", name)
}
return function.New(&spec)
}
2 changes: 1 addition & 1 deletion provisioner/terraform/tfparse/tfparse.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -477,7 +477,7 @@ func BuildEvalContext(vars map[string]string, params map[string]string) *hcl.Eva
// The default function map for Terraform is not exposed, so we would essentially
// have to re-implement or copy the entire map or a subset thereof.
// ref: https://github.com/hashicorp/terraform/blob/e044e569c5bc81f82e9a4d7891f37c6fbb0a8a10/internal/lang/functions.go#L54
Functions: nil,
Functions: Functions(),
}
if len(varDefaultsM) != 0 {
evalCtx.Variables["var"] = cty.MapVal(varDefaultsM)
Expand Down
44 changes: 42 additions & 2 deletions provisioner/terraform/tfparse/tfparse_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,13 +416,52 @@ func Test_WorkspaceTagDefaultsFromFile(t *testing.T) {
expectError: `There is no variable named "foo_bar"`,
},
{
name: "main.tf with functions in workspace tags",
name: "main.tf with allowed functions in workspace tags",
files: map[string]string{
"main.tf": `
provider "foo" {}
resource "foo_bar" "baz" {
name = "foobar"
}
locals {
some_path = pathexpand("file.txt")
}
variable "region" {
type = string
default = "us"
}
data "coder_parameter" "unrelated" {
name = "unrelated"
type = "list(string)"
default = jsonencode(["a", "b"])
}
data "coder_parameter" "az" {
name = "az"
type = "string"
default = "a"
}
data "coder_workspace_tags" "tags" {
tags = {
"platform" = "kubernetes",
"cluster" = "${"devel"}${"opers"}"
"region" = try(split(".", var.region)[1], "placeholder")
"az" = try(split(".", data.coder_parameter.az.value)[1], "placeholder")
}
}`,
},
expectTags: map[string]string{"platform": "kubernetes", "cluster": "developers", "region": "placeholder", "az": "placeholder"},
},
{
name: "main.tf with disallowed functions in workspace tags",
files: map[string]string{
"main.tf": `
provider "foo" {}
resource "foo_bar" "baz" {
name = "foobar"
}
locals {
some_path = pathexpand("file.txt")
}
variable "region" {
type = string
default = "region.us"
Expand All @@ -443,11 +482,12 @@ func Test_WorkspaceTagDefaultsFromFile(t *testing.T) {
"cluster" = "${"devel"}${"opers"}"
"region" = try(split(".", var.region)[1], "placeholder")
"az" = try(split(".", data.coder_parameter.az.value)[1], "placeholder")
"some_path" = pathexpand("~/file.txt")
}
}`,
},
expectTags: nil,
expectError: `Function calls not allowed; Functions may not be called here.`,
expectError: `function "pathexpand" may not be used here`,
},
{
name: "supported types",
Expand Down
Loading