Skip to content

The IP address from the X-Forwarded-For list has a bug which incorrectly selects the last IP address #13725

@am-i-helpful

Description

@am-i-helpful

What happened:

After upgrading the helm-chart to 4.13.0, the ingresses (of applications) which were ip-restricted were all forbidden (giving 403 error). We digged further to identify what might be the problem, and found that the nginx.conf inside the ingress-controller pod is having 1 config difference in the newer version.

What you expected to happen:

Inside the new nginx.conf, within the map $http_x_forwarded_for $full_x_forwarded_for { block and map $proxy_protocol_addr_trusted $forwarded_for_proxy_protocol { block, we noticed that there is an extra config as listed below:
1 $http_x_forwarded_for;
After deleting this line from the pod running v4.13.0 nginx config and performing Nginx reload using the command nginx -s reload, those applications were accessible again. So, I think this line is not doing what it should, which I suspect the logic introduced in v4.13.0 was trying to intelligently parse this list. The 1 $http_x_forwarded_for; part essentially acts as a default rule within that map. It says, "If none of the other specific conditions in this map are met, use the entire $http_x_forwarded_for header as the value."

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):
-> NGINX Ingress controller
Release: v1.13.0
Build: 4cbb78a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.27.1

Kubernetes version (use kubectl version):
-> kubeadm version: &version.Info{Major:"1", Minor:"31", GitVersion:"v1.31.2", GitCommit:"5864a4677267e6adeae276ad85882a8714d69d9d", GitTreeState:"clean", BuildDate:"2024-10-22T20:33:59Z", GoVersion:"go1.22.8", Compiler:"gc", Platform:"linux/amd64"}
Environment:

  • Cloud provider or hardware configuration: N/A

  • OS (e.g. from /etc/os-release): Debian 12

  • Kernel (e.g. uname -a):

  • Install tools:

    • via HelmRelease, as part of FluxCD GitOps management
  • Basic cluster related info:

    • kubectl version
      Client Version: v1.31.2
      Kustomize Version: v5.4.2
      Server Version: v1.31.7
    • kubectl get nodes -o wide
      N/A
  • How was the ingress-nginx-controller installed:

    • If helm was used then please show output of helm ls -A | grep -i ingress
      -> ingress-nginx ingress-nginx 8 2025-08-07 12:04:35.993959567 +0000 UTC deployed ingress-nginx-4.13.0 1.13.0
    • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
      ->
       publishService:
         enabled: true
       allowSnippetAnnotations: true
       config:
         use-forwarded-headers: "true"
         strict-validate-path-type: "false"
         compute-full-forwarded-for: "true"
         enable-real-ip: "true"
         use-proxy-protocol: "true"
         annotations-risk-level: "Critical" # https://github.com/kubernetes/ingress-nginx/issues/12648#issuecomment-2796248026
       metrics:
         port: 10254
         portName: metrics
         enabled: true
         service:
           enabled: true
         serviceMonitor:
           enabled: true
           namespace: "ingress-nginx"
           namespaceSelector:
             any: true
             scrapeInterval: 30s
       service:
         annotations:
           load-balancer.x.y/location: abc
           load-balancer.x.y/name: my-lb
           load-balancer.x.y/use-private-ip: "true"
           load-balancer.x.y/uses-proxyprotocol: "true"
     admissionWebhooks: # https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx#ingress-admission-webhooks
       enabled: false```
    
    
  • Current State of the controller:

    • kubectl describe ingressclasses
      Name: nginx
      Labels: app.kubernetes.io/component=controller
      app.kubernetes.io/instance=ingress-nginx
      app.kubernetes.io/managed-by=Helm
      app.kubernetes.io/name=ingress-nginx
      app.kubernetes.io/part-of=ingress-nginx
      app.kubernetes.io/version=1.13.0
      helm.sh/chart=ingress-nginx-4.13.0
      helm.toolkit.fluxcd.io/name=ingress-nginx
      helm.toolkit.fluxcd.io/namespace=ingress-nginx
      Annotations: meta.helm.sh/release-name: ingress-nginx
      meta.helm.sh/release-namespace: ingress-nginx
      Controller: k8s.io/ingress-nginx
      Events:
    • kubectl -n <ingresscontrollernamespace> get all -A -o wide
    • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
    • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
  • Current state of ingress object, if applicable:

    • kubectl -n <appnamespace> get all,ing -o wide
    • kubectl -n <appnamespace> describe ing <ingressname>
    • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

How to reproduce this issue:
-> Please rollout the latest version of Ingress-Nginx controller v4.13.0, and make sure there are applications which have ingresses with annotation configured on LoadBalancer to be accessible only by specific ip-addresses. After rolling out the newer ingress-controller version, the applications give 403 error, because the ip-addresses which are identified are all internal to Kubernetes, and not the first value from the X-Forwarded-For header.

Anything else we need to know:
Please test the changes we have done, which is now working successfully. We removed the 1 $http_x_forwarded_for; config from the nginx.conf, which resulted in the applications to work again. IMHO, this use-case (of application accessibility when restricted by certain ip-addresses) should be tested properly via test-cases before an actual release, as I feel this happened because of a missing test-case pertaining to this scenario.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions