Skip to content

When used in tandem with aws-load-balancer-controller, passing in a value is quoted when it shouldn't always be. #13691

@thehandsomezebra

Description

@thehandsomezebra

What happened:

We are using deploying the ingress-nginx helm chart version 4.12.1 with aws-load-balancer-controller 2.12 via NLB.

We are using the stringList annotation as --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="arn:aws:acm:us-east-1:12345:certificate/cert1\,arn:aws:acm:us-east-1:12345:certificate/cert2"

Nothing -> NLB

  • install fresh (no NLB), attempt to create with two certs, it added zero, 400 error. FAIL

Existing NLB

  • upgrade started with zero certs, attempted to add one. PASS
  • upgrade started with cert1, attempted to add secondary. FAIL
  • manually modified to add secondary via AWS Console, then re-run upgrade with cert1 & cert2, still produces the 400 error, but did not modify anything in a bad way. SOFT PASS but still ERROR
 helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace --version 4.12.1 
 --set controller.replicaCount=3 
 --set controller.nodeSelector.node=static 
 --set controller.service.targetPorts.http=http 
 --set controller.service.targetPorts.https=http 
 --set controller.config.allow-snippet-annotations=true 
 --set controller.config.annotations-risk-level=Critical 
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"=http 
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="arn:aws:acm:us-east-1:12345:certificate/cert1\,arn:aws:acm:us-east-1:12345:certificate/cert2" 
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"=https 
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-connection-idle-timeout"=3600 
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing 
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb

What you expected to happen:
Expected load balancer created with 1 default ssl cert (first one listed will be default) & 1 additional cert.

NGINX Ingress controller version

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.12.1
  Build:         51c2b819690bbf1709b844dbf321a9acf6eda5a7
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.32.5-eks-5d4a308

Environment:
EKS running Bottlerocket OS 1.43.0 (aws-k8s-1.32) 6.1.141 containerd://1.7.27+bottlerocket

➜  ~ helm version
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}
  • How was the ingress-nginx-controller installed:
    Installed via helm, see code block mentioned above

  • Current State of the controller:

Warning  SyncLoadBalancerFailed  
service-controller  Error syncing load balancer: failed to ensure load balancer: error updating load balancer listener: "ValidationError: Certificate ARN 'arn:aws:acm:us-east-1:12345:certificate/cert1,arn:aws:acm:us-east-1:12345:certificate/cert2 is not valid\n\tstatus code: 400, request id: xxxxx
  • Current state of ingress object, if applicable:
    N/A

  • Others:

    • Any other related information like ;
      • copy/paste of the snippet (if applicable)
      • kubectl describe ... of any custom configmap(s) created and in use
      • Any other related information that may help

How to reproduce this issue:
Using the ingress-nginx Helm chart, attempt to create with annotations for the controller.service.annotations.* and you will see that all values are coming in as quoted.
Quotes should be handled via the values.yaml per value and not pre-quoted.

Anything else we need to know:

We can see that in the documentation for the aws-load-balancer-controller that it does support the two certs, comma separated:
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.13/guide/service/annotations/#tls

The Helm template for rendering Service annotations globally applies the | quote function. This causes an issue with annotations that rely on comma-separated values, such as service.beta.kubernetes.io/aws-load-balancer-ssl-cert, because the entire list is wrapped in a single set of quotes. The downstream Go parser is then unable to split the string into individual items (ParseStringSliceAnnotation https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/pkg/service/model_build_listener.go#L181 ), leading to an AWS API ValidationError.

Here's a few issues that I believe might be related

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions