-
Notifications
You must be signed in to change notification settings - Fork 8.4k
Description
What happened:
After upgrading the helm-chart to 4.13.0, the ingresses (of applications) which were ip-restricted were all forbidden (giving 403 error). We digged further to identify what might be the problem, and found that the nginx.conf inside the ingress-controller pod is having 1 config difference in the newer version.
What you expected to happen:
Inside the new nginx.conf, within the map $http_x_forwarded_for $full_x_forwarded_for {
block and map $proxy_protocol_addr_trusted $forwarded_for_proxy_protocol {
block, we noticed that there is an extra config as listed below:
1 $http_x_forwarded_for;
After deleting this line from the pod running v4.13.0 nginx config and performing Nginx reload using the command nginx -s reload
, those applications were accessible again. So, I think this line is not doing what it should, which I suspect the logic introduced in v4.13.0 was trying to intelligently parse this list. The 1 $http_x_forwarded_for; part essentially acts as a default rule within that map. It says, "If none of the other specific conditions in this map are met, use the entire $http_x_forwarded_for header as the value."
NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version
):
-> NGINX Ingress controller
Release: v1.13.0
Build: 4cbb78a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.27.1
Kubernetes version (use kubectl version
):
-> kubeadm version: &version.Info{Major:"1", Minor:"31", GitVersion:"v1.31.2", GitCommit:"5864a4677267e6adeae276ad85882a8714d69d9d", GitTreeState:"clean", BuildDate:"2024-10-22T20:33:59Z", GoVersion:"go1.22.8", Compiler:"gc", Platform:"linux/amd64"}
Environment:
-
Cloud provider or hardware configuration: N/A
-
OS (e.g. from /etc/os-release): Debian 12
-
Kernel (e.g.
uname -a
): -
Install tools:
via HelmRelease, as part of FluxCD GitOps management
-
Basic cluster related info:
kubectl version
Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.7kubectl get nodes -o wide
N/A
-
How was the ingress-nginx-controller installed:
- If helm was used then please show output of
helm ls -A | grep -i ingress
-> ingress-nginx ingress-nginx 8 2025-08-07 12:04:35.993959567 +0000 UTC deployed ingress-nginx-4.13.0 1.13.0 - If helm was used then please show output of
helm -n <ingresscontrollernamespace> get values <helmreleasename>
->
publishService: enabled: true allowSnippetAnnotations: true config: use-forwarded-headers: "true" strict-validate-path-type: "false" compute-full-forwarded-for: "true" enable-real-ip: "true" use-proxy-protocol: "true" annotations-risk-level: "Critical" # https://github.com/kubernetes/ingress-nginx/issues/12648#issuecomment-2796248026 metrics: port: 10254 portName: metrics enabled: true service: enabled: true serviceMonitor: enabled: true namespace: "ingress-nginx" namespaceSelector: any: true scrapeInterval: 30s service: annotations: load-balancer.x.y/location: abc load-balancer.x.y/name: my-lb load-balancer.x.y/use-private-ip: "true" load-balancer.x.y/uses-proxyprotocol: "true" admissionWebhooks: # https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx#ingress-admission-webhooks enabled: false```
- If helm was used then please show output of
-
Current State of the controller:
kubectl describe ingressclasses
Name: nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.13.0
helm.sh/chart=ingress-nginx-4.13.0
helm.toolkit.fluxcd.io/name=ingress-nginx
helm.toolkit.fluxcd.io/namespace=ingress-nginx
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
Controller: k8s.io/ingress-nginx
Events:kubectl -n <ingresscontrollernamespace> get all -A -o wide
kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
-
Current state of ingress object, if applicable:
kubectl -n <appnamespace> get all,ing -o wide
kubectl -n <appnamespace> describe ing <ingressname>
- If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
How to reproduce this issue:
-> Please rollout the latest version of Ingress-Nginx controller v4.13.0, and make sure there are applications which have ingresses with annotation configured on LoadBalancer to be accessible only by specific ip-addresses. After rolling out the newer ingress-controller version, the applications give 403 error, because the ip-addresses which are identified are all internal to Kubernetes, and not the first value from the X-Forwarded-For header.
Anything else we need to know:
Please test the changes we have done, which is now working successfully. We removed the 1 $http_x_forwarded_for;
config from the nginx.conf, which resulted in the applications to work again. IMHO, this use-case (of application accessibility when restricted by certain ip-addresses) should be tested properly via test-cases before an actual release, as I feel this happened because of a missing test-case pertaining to this scenario.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status