Hello!
I’m working on a IoT middleware project where we’re trying to make the gateway service elastic using kubernetes. This gateway service is a Java application and can receive around 5000 nodes connections using UDP. Note that this connections are persistent.
We’re using a private cloud and a single node kubernetes cluster running with minikube with none driver.
- Minikube version: v1.30.1
- Kubernetes version: 1.26.3
- Host OS: Ubuntu 16.04.7 LTS
Our first approach is a NodePort service, and HPA with metrics-server making autoscale based on CPU. At the first moment it worked, but then we have some trouble with connections balancing after an pod upscale. For example, in a scenario that I have 5000 nodes connected to a pod and an upscale happens I want to redirect all new connections to the newer pod, but since kube-proxy use iptables that behavior doesnt happen.
I was looking for some solutions and find that I can use kube-proxy in IPVS mode and select least connection as scheduler for load balancing. So, I run kube-proxy in IPVS mode and start have some issues with nodes getting disconnected (something that doesnt happen when I use kube-proxy with iptables).
I have two main questions here:
- This approach with kube-proxy in IPVS mode sounds like a good solution for the problem that I need to solve? If yes, there is a way to debug kube-proxy in IPVS mode to try understand what is happening with connections to be droped?
- There is any suggestion to solve this upscale balancing scenario that can use iptables, or any other kubernetes approach?