Issues with KubeProxy Network Programming Duration


I’m having an issue in all of my clusters. The metric "kubeproxy_network_programming_duration_seconds_bucket
" shows the amount of time that kube-proxy takes to sync the ipvs rules in the nodes to reflect the changes in etcd (for example, a pod being removed as an endpoint from a service)

This is causing timeouts on my requests, as some of them go to pods that are already dead.

I can reproduce the error when I restart a lot of pods, but I’ve checked the system metrics of etcd, controllers and nodes, and I can’t find any service being short of CPU of memory. There is no way to add more logging to kube-proxy. Where can I find more information about this? If I google this metric name, I can’t find anything helpful, so no one is using it :disappointed:

So yes, the p99 is 10!!! seconds. That means that a pod that’s no longer an endpoint of a service, can still receive traffic 10 seconds after the SIGTERM is sent to the pod.

I’ve confirmed this behavior by adding a PreStop exec command (a 12s sleep). This sleep allows the pod to keep receiving requests even after is marked as “terminating” and removed from the service endpoints

Cluster information:

Kubernetes version: 1.16
Cloud being used: AWS
Installation method: user-data scripts
Host OS: CoreOS
CNI and version: Flannel
CRI and version: Docker 18.06