-
Notifications
You must be signed in to change notification settings - Fork 8.4k
Description
What happened:
We are using deploying the ingress-nginx helm chart version 4.12.1 with aws-load-balancer-controller 2.12 via NLB.
We are using the stringList annotation as --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="arn:aws:acm:us-east-1:12345:certificate/cert1\,arn:aws:acm:us-east-1:12345:certificate/cert2"
Nothing -> NLB
- install fresh (no NLB), attempt to create with two certs, it added zero, 400 error.
FAIL
Existing NLB
- upgrade started with zero certs, attempted to add one.
PASS
- upgrade started with cert1, attempted to add secondary.
FAIL
- manually modified to add secondary via AWS Console, then re-run upgrade with cert1 & cert2, still produces the 400 error, but did not modify anything in a bad way.
SOFT PASS but still ERROR
helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace --version 4.12.1
--set controller.replicaCount=3
--set controller.nodeSelector.node=static
--set controller.service.targetPorts.http=http
--set controller.service.targetPorts.https=http
--set controller.config.allow-snippet-annotations=true
--set controller.config.annotations-risk-level=Critical
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"=http
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="arn:aws:acm:us-east-1:12345:certificate/cert1\,arn:aws:acm:us-east-1:12345:certificate/cert2"
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"=https
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-connection-idle-timeout"=3600
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb
What you expected to happen:
Expected load balancer created with 1 default ssl cert (first one listed will be default) & 1 additional cert.
NGINX Ingress controller version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.12.1
Build: 51c2b819690bbf1709b844dbf321a9acf6eda5a7
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.5
-------------------------------------------------------------------------------
Kubernetes version (use kubectl version
):
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.32.5-eks-5d4a308
Environment:
EKS running Bottlerocket OS 1.43.0 (aws-k8s-1.32) 6.1.141 containerd://1.7.27+bottlerocket
➜ ~ helm version
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}
-
How was the ingress-nginx-controller installed:
Installed via helm, see code block mentioned above -
Current State of the controller:
Warning SyncLoadBalancerFailed
service-controller Error syncing load balancer: failed to ensure load balancer: error updating load balancer listener: "ValidationError: Certificate ARN 'arn:aws:acm:us-east-1:12345:certificate/cert1,arn:aws:acm:us-east-1:12345:certificate/cert2 is not valid\n\tstatus code: 400, request id: xxxxx
-
Current state of ingress object, if applicable:
N/A -
Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
kubectl describe ...
of any custom configmap(s) created and in use- Any other related information that may help
- Any other related information like ;
How to reproduce this issue:
Using the ingress-nginx Helm chart, attempt to create with annotations for the controller.service.annotations.* and you will see that all values are coming in as quoted.
Quotes should be handled via the values.yaml per value and not pre-quoted.
Anything else we need to know:
We can see that in the documentation for the aws-load-balancer-controller that it does support the two certs, comma separated:
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.13/guide/service/annotations/#tls
The Helm template for rendering Service annotations globally applies the | quote
function. This causes an issue with annotations that rely on comma-separated values, such as service.beta.kubernetes.io/aws-load-balancer-ssl-cert, because the entire list is wrapped in a single set of quotes. The downstream Go parser is then unable to split the string into individual items (ParseStringSliceAnnotation
https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/pkg/service/model_build_listener.go#L181 ), leading to an AWS API ValidationError.
Here's a few issues that I believe might be related
Metadata
Metadata
Assignees
Labels
Type
Projects
Status