Skip to content

Dependencies for cloudinit_post_nodeadm content result in Invalid count argument error #3334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
1 task done
daniel-palmer-gu opened this issue Apr 3, 2025 · 5 comments
Labels

Comments

@daniel-palmer-gu
Copy link

daniel-palmer-gu commented Apr 3, 2025

Description

It would seem that cloudinit_post_nodeadm content can not depend on resources that terraform needs to create. Doing so results in a Invalid count argument error.

Below is an example that adds dummy data to the content. It doesn't make sense but proves the point. If there were a real template that used that data, the result should be the same.

  • ✋ I have searched the open/closed issues and my issue is not listed.

Versions

  • Module version [Required]: 20.35.0

  • Terraform version: Terraform v1.5.7

  • Provider version(s):

Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.94.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.6
+ provider registry.terraform.io/hashicorp/null v3.2.3
+ provider registry.terraform.io/hashicorp/random v3.7.1
+ provider registry.terraform.io/hashicorp/time v0.13.0
+ provider registry.terraform.io/hashicorp/tls v4.0.6

Reproduction Code [Required]

module "eks_al2023" {
  source  = "terraform-aws-modules/eks/aws"
  version = "= 20.35.0"

  cluster_name    = "testing-al2023"
  cluster_version = "1.31"

  # EKS Addons
  cluster_addons = {
    coredns                = {}
    eks-pod-identity-agent = {}
    kube-proxy             = {}
    vpc-cni                = {}
  }

  # vpc_id     = module.vpc.vpc_id
  # subnet_ids = module.vpc.private_subnets
  vpc_id = "vpc-myvpc"
  subnet_ids = [
    "subnet-mysubnet1",
    "subnet-mysubnet2",
    "subnet-mysubnet3"
  ]

  eks_managed_node_groups = {
    example = {
      # Starting on 1.30, AL2023 is the default AMI type for EKS managed node groups
      instance_types = ["m6i.large"]

      min_size = 1
      max_size = 1
      # This value is ignored after the initial creation
      # https://github.com/bryantbiggs/eks-desired-size-hack
      desired_size = 1

      # This is not required - demonstrates how to pass additional configuration to nodeadm
      # Ref https://awslabs.github.io/amazon-eks-ami/nodeadm/doc/api/
      cloudinit_pre_nodeadm = [
        {
          content_type = "application/node.eks.aws"
          content      = <<-EOT
            ---
            apiVersion: node.eks.aws/v1alpha1
            kind: NodeConfig
            spec:
              kubelet:
                config:
                  shutdownGracePeriod: 30s
                  featureGates:
                    DisableKubeletCloudCredentialProviders: true
          EOT
        }
      ]
    }
  }

  eks_managed_node_group_defaults = {
    enable_bootstrap_user_data = true
    ami_type               = "AL2023_x86_64_STANDARD"
    cloudinit_post_nodeadm = [{
      content_type = "text/x-shellscript; charset=\"us-ascii\""
      content = random_password.my_password.result
    }]
  }
}

resource "random_password" "my_password" {
  length = 32
}

Steps to reproduce the behavior:

  1. Create a fresh main.tf file with the contents above
  2. Update vpc_id and subnet_ids with ones you have access to
  3. terraform init
  4. terraform apply (or plan)

Expected behavior

Plan succeeds and can apply the code

Actual behavior

Plan fails with the error below

Terminal Output Screenshot(s)

Image

Text version of error:

│ Error: Invalid count argument
│ 
│   on .terraform/modules/eks_al2023/modules/_user_data/main.tf line 131, in data "cloudinit_config" "al2023_eks_managed_node_group":
│  131:   count = var.create && local.user_data_type == "al2023" && length(local.nodeadm_cloudinit) > 0 ? 1 : 0
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target
│ argument to first apply only the resources that the count depends on.

Additional context

Removing the dependency on the random_password and replacing it's contents with "mypassword" allows the plan to succeed.

@daniel-palmer-gu
Copy link
Author

daniel-palmer-gu commented Apr 3, 2025

Additional additional context:

Due to the nature of the error, this really only matters on the initial deploy of the resources. If the random_password has been created in a previous apply, this error does not present.

I recently started moving over to AL2023, and noticed this issue. When using post_bootstrap_user_data for non-AL2023 AMIs, this error doesn't occur likely due to the difference in the way the user data is prepared.

@daniel-palmer-gu daniel-palmer-gu changed the title Dependencies on cloudinit_post_nodeadm content result in Invalid count argument error Dependencies for cloudinit_post_nodeadm content result in Invalid count argument error Apr 3, 2025
@bryantbiggs
Copy link
Member

this isn't quite valid - cloudinit_post_nodeadm (AL2023) and post_bootstrap_user_data (AL2) are not used when using EKS optimized AMIs within EKS managed node groups

@daniel-palmer-gu
Copy link
Author

I'm not sure I am tracking. The output above would indicate that it is trying to use it - no?

Can you provide more information on what is not valid about the example?

@bryantbiggs
Copy link
Member

https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-user-data

you can only specify user data that is added *ahead of what EKS MNG adds.

in the case of AL2, the user data supplied is added *before the bootstrap.sh script line is injected by EKS MNG

in the case of AL2023, the user data supplied is merged *before the nodeadm cloud init block added by EKS MNG

therefore, what you posted above and what I've modified below, they result in the same outcome in terms of resources/configuration created:

module "eks_al2023" {
  source  = "terraform-aws-modules/eks/aws"
  version = "= 20.35.0"

  cluster_name    = "testing-al2023"
  cluster_version = "1.31"

  # EKS Addons
  cluster_addons = {
    coredns                = {}
    eks-pod-identity-agent = {}
    kube-proxy             = {}
    vpc-cni                = {}
  }

  # vpc_id     = module.vpc.vpc_id
  # subnet_ids = module.vpc.private_subnets
  vpc_id = "vpc-myvpc"
  subnet_ids = [
    "subnet-mysubnet1",
    "subnet-mysubnet2",
    "subnet-mysubnet3"
  ]

  eks_managed_node_groups = {
    example = {
      # Starting on 1.30, AL2023 is the default AMI type for EKS managed node groups
      instance_types = ["m6i.large"]

      min_size = 1
      max_size = 1
      # This value is ignored after the initial creation
      # https://github.com/bryantbiggs/eks-desired-size-hack
      desired_size = 1

      # This is not required - demonstrates how to pass additional configuration to nodeadm
      # Ref https://awslabs.github.io/amazon-eks-ami/nodeadm/doc/api/
      cloudinit_pre_nodeadm = [
        {
          content_type = "application/node.eks.aws"
          content      = <<-EOT
            ---
            apiVersion: node.eks.aws/v1alpha1
            kind: NodeConfig
            spec:
              kubelet:
                config:
                  shutdownGracePeriod: 30s
                  featureGates:
                    DisableKubeletCloudCredentialProviders: true
          EOT
        }
      ]
    }
  }
}

@daniel-palmer-gu
Copy link
Author

daniel-palmer-gu commented Apr 4, 2025

Two things I am noticing:

  1. In my actual code base, I use the post section to configure some containerd stuff. Theres no reason I couldn't move this code to cloudinit_pre_nodeadm. However, I still get the same error. (see example below)
  2. Are you sure that is accurate for the AL2023 case? It looks like user data can add in content after by using cloudinit_post_nodeadm. When I run this and stand up the instance after the password already exists, it shows up as part-003 in the cloud init folder on the instance.
module "eks_al2023" {
  source  = "terraform-aws-modules/eks/aws"
  version = "= 20.35.0"

  cluster_name    = "testing-al2023"
  cluster_version = "1.31"

  # EKS Addons
  cluster_addons = {
    coredns                = {}
    eks-pod-identity-agent = {}
    kube-proxy             = {}
    vpc-cni                = {}
  }

  # vpc_id     = module.vpc.vpc_id
  # subnet_ids = module.vpc.private_subnets
  vpc_id = "vpc-myvpc"
  subnet_ids = [
    "subnet-mysubnet1",
    "subnet-mysubnet2",
    "subnet-mysubnet3"
  ]

  eks_managed_node_groups = {
    example = {
      # Starting on 1.30, AL2023 is the default AMI type for EKS managed node groups
      instance_types = ["m6i.large"]

      min_size = 1
      max_size = 1
      # This value is ignored after the initial creation
      # https://github.com/bryantbiggs/eks-desired-size-hack
      desired_size = 1
    }
  }

  eks_managed_node_group_defaults = {
    enable_bootstrap_user_data = true
    ami_type               = "AL2023_x86_64_STANDARD"
    cloudinit_pre_nodeadm = [{
      content_type = "text/x-shellscript; charset=\"us-ascii\""
      content = random_password.my_password
    }]
  }
}

resource "random_password" "my_password" {
  length           = 32
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants