Unable to connect to pod ips from inside pod

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: v1.19.4
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: install by rancher
Host OS: Centos 7.9
CNI and version: Flannel with vxlan, and ipvs as proxy.
CRI and version: docker 20.10.5

Hello, I tried to connect to another pod by curl , but it took a long time to reponse:
$time curl
checkHeartBeat success
real 0m31.088s
user 0m0.003s
sys 0m0.002s

Then I tried to capture packets inside the container network namespace and host namespace, and I found that : The syn+ack reply soon on the host namespace, but it disappear on the network namespace. Then I tried to filter sequence number on the container packets, I found the packet transit to service ip.

Client pod ip is, server pod ip is And the service rule as following:
$ ipvsadm -L -t
Prot LocalAddress:Port Scheduler Flags
→ RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP rr
→ Masq 1 23 5108
→ Masq 1 23 5097
→ Masq 1 23 5098
→ Masq 1 30 5061

From the client pod, we can find:

And from the client host namespace, we can find the syn+ack already replied:

Then I used the server sequence number as filter rule on pod packets, then I find the ip been transit to service ip as below:

Since we have a large connect with the service ip, so i thought that there might be some sessions left on conntrack, so the reply packets hit the session and transit to service ip. Then I capture and collect the conntrack table on the same time. But I can’t get a session with service ip on the fault port.

How I can proceed? Thank you very much.

Captures inside pods.

Captures from the host namespace:

I tried to get a conntack like bellow, but I failed.
tcp 6 0 CLOSE src= dst= sport=57559 dport=8083 src= dst= sport=8083 dport=57559 mark=0 use=1