Nginx Ingress TCP Connection Limit During Load Testing in Kubernetes

General Discussions
1

Kubernetes version: Latest
Cloud being used: (put bare-metal if not on a public cloud)
Installation method:
Host OS: Ubuntu 22.04
CNI and version: Calico
CRI and version:

I’m running a community site on a Kubernetes environment and using Nginx Ingress as below:
Nginx Ingress Controller → Nginx SVC[ Nginx Pods (serving JS files, etc.)]

During load testing with many concurrent users, the TCP connection count on the Ingress side goes up to around 1200, while each of the 3 Nginx Pods sees its TCP connection count rise above 300 but then gets capped at around 75.
It seems like certain connections are being limited when high traffic occurs.

I want the Nginx Pods to handle all incoming connections and process user requests quickly. However, tweaking the Ingress upstream count, worker connections, and other settings didn’t help. I also tried looking into the internal load balancer (like balancer.lua), but couldn’t find a solution.

If anyone knows how to address this issue or has suggestions on what to check, please let me know. Thanks in advance! :blush:

1 Like
2
  1. Do you have autoscaling enabled?
kubectl autoscale deployment nginx-ingress-controller \
    --cpu-percent=60 --min=2 --max=10
  1. Check ingress controller limits
    kubectl describe deployment nginx-ingress-controller -n ingress-nginx
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2000m"
    memory: "2Gi"
  1. check if nginx pods has cpu, memory or other resources limits.
  2. Nginx worker settings Nginx’s worker_processes and worker_connections settings directly impact how many concurrent connections can be processed.
worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 16384;
    multi_accept on;
}
  1. Ingress controller may have limits, try to see if you want to tune it and if it helps.
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30s"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600s"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600s"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    nginx.ingress.kubernetes.io/keep-alive-requests: "10000"
    nginx.ingress.kubernetes.io/upstream-keepalive-connections: "1024"
    nginx.ingress.kubernetes.io/upstream-keepalive-timeout: "300s"
    nginx.ingress.kubernetes.io/upstream-keepalive-requests: "10000"
1 Like