Issues with KubeProxy Network Programming Duration

General Discussions
1

Hello.

I’m having an issue in all of my clusters. The metric "kubeproxy_network_programming_duration_seconds_bucket
" shows the amount of time that kube-proxy takes to sync the ipvs rules in the nodes to reflect the changes in etcd (for example, a pod being removed as an endpoint from a service)

This is causing timeouts on my requests, as some of them go to pods that are already dead.

I can reproduce the error when I restart a lot of pods, but I’ve checked the system metrics of etcd, controllers and nodes, and I can’t find any service being short of CPU of memory. There is no way to add more logging to kube-proxy. Where can I find more information about this? If I google this metric name, I can’t find anything helpful, so no one is using it :disappointed:

image
image2510×844 188 KB

So yes, the p99 is 10!!! seconds. That means that a pod that’s no longer an endpoint of a service, can still receive traffic 10 seconds after the SIGTERM is sent to the pod.

I’ve confirmed this behavior by adding a PreStop exec command (a 12s sleep). This sleep allows the pod to keep receiving requests even after is marked as “terminating” and removed from the service endpoints

Cluster information:

Kubernetes version: 1.16
Cloud being used: AWS
Installation method: user-data scripts
Host OS: CoreOS
CNI and version: Flannel
CRI and version: Docker 18.06

2

@pablokbs have you find any solutions to this? We have recently identified this. I wonder these two issues are related.

Update: It seems like this metric has issues with recording accurate values… As documented in SLI/SLO Caveats and here https://github.com/kubernetes/kubernetes/issues/82378