Asking for help? Comment out what you need so we can get more information to help you!
Cluster information:
Kubernetes version: v1.19.4
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: install by rancher
Host OS: Centos 7.9
CNI and version: Flannel with vxlan, and ipvs as proxy.
CRI and version: docker 20.10.5
Hello, I tried to connect to another pod by curl , but it took a long time to reponse:
$time curl http://172.20.64.35:8083
checkHeartBeat success
real 0m31.088s
user 0m0.003s
sys 0m0.002s
Then I tried to capture packets inside the container network namespace and host namespace, and I found that : The syn+ack reply soon on the host namespace, but it disappear on the network namespace. Then I tried to filter sequence number on the container packets, I found the packet transit to service ip.
Client pod ip is 172.20.67.18, server pod ip is 172.20.64.35. And the service rule as following:
$ ipvsadm -L -t 172.21.32.252:8083
Prot LocalAddress:Port Scheduler Flags
→ RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.21.32.252:us-srv rr
→ 172.20.60.32:us-srv Masq 1 23 5108
→ 172.20.61.25:us-srv Masq 1 23 5097
→ 172.20.63.27:us-srv Masq 1 23 5098
→ 172.20.64.35:us-srv Masq 1 30 5061
From the client pod, we can find:
And from the client host namespace, we can find the syn+ack already replied:
Then I used the server sequence number as filter rule on pod packets, then I find the ip been transit to service ip as below:
Since we have a large connect with the service ip, so i thought that there might be some sessions left on conntrack, so the reply packets hit the session and transit to service ip. Then I capture and collect the conntrack table on the same time. But I can’t get a session with service ip on the fault port.
How I can proceed? Thank you very much.
