Cluster information:
Kubernetes version: v1.16.7
Host OS: Linux
I have a couple of services in my application on kubernetes that communicate with each other. Recently while checking the network traffic using tcpdump , I saw a service was throwing a reset every time it was involved in a communication.
I see the below output every few seconds:
root@querying1:~# tcpdump -a -s 0 -tt -v
tcpdump: listening on cali8eec3d9c0ff, link-type EN10MB (Ethernet), capture size 262144 bytes
1623958319.504851 IP (tos 0x0, ttl 64, id 57704, offset 0, flags [DF], proto TCP (6), length 281)
10.1.58.62.57162 > 10.1.58.47.27017: Flags [P.], cksum 0x897a (incorrect -> 0x7482), seq 1980143344:1980143573, ack 1710182524, win 502, options [nop,nop,TS val 3094663568 ecr 595693379], length 229
1623958319.505243 IP (tos 0x0, ttl 63, id 20674, offset 0, flags [DF], proto TCP (6), length 921)
10.1.58.47.27017 > 10.1.58.62.57162: Flags [P.], cksum 0x8bfa (incorrect -> 0x9bc8), seq 1:870, ack 229, win 6148, options [nop,nop,TS val 595703380 ecr 3094663568], length 869
1623958319.505254 IP (tos 0x0, ttl 64, id 57705, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.62.57162 > 10.1.58.47.27017: Flags [.], cksum 0x8895 (incorrect -> 0x77fc), ack 870, win 502, options [nop,nop,TS val 3094663568 ecr 595703380], length 0
1623958319.536019 IP (tos 0x0, ttl 63, id 49352, offset 0, flags [DF], proto TCP (6), length 60)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [S], cksum 0x887a (incorrect -> 0x77bb), seq 2370570482, win 65330, options [mss 1390,sackOK,TS val 3415999428 ecr 0,nop,wscale 7], length 0
1623958319.536036 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.1.58.62.4500 > 10.1.58.12.53720: Flags [S.], cksum 0x887a (incorrect -> 0x465f), seq 2681670834, ack 2370570483, win 64766, options [mss 1390,sackOK,TS val 2168718769 ecr 3415999428,nop,wscale 7], length 0
1623958319.536052 IP (tos 0x0, ttl 63, id 49353, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [.], cksum 0x8872 (incorrect -> 0x6fe5), ack 1, win 511, options [nop,nop,TS val 3415999428 ecr 2168718769], length 0
1623958319.536262 IP (tos 0x0, ttl 63, id 49354, offset 0, flags [DF], proto TCP (6), length 525)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [P.], cksum 0x8a4b (incorrect -> 0x10ce), seq 1:474, ack 1, win 511, options [nop,nop,TS val 3415999428 ecr 2168718769], length 473
1623958319.536268 IP (tos 0x0, ttl 64, id 8244, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.62.4500 > 10.1.58.12.53720: Flags [.], cksum 0x8872 (incorrect -> 0x6e14), ack 474, win 503, options [nop,nop,TS val 2168718769 ecr 3415999428], length 0
1623958319.536702 IP (tos 0x0, ttl 64, id 8245, offset 0, flags [DF], proto TCP (6), length 193)
10.1.58.62.4500 > 10.1.58.12.53720: Flags [P.], cksum 0x88ff (incorrect -> 0x74ea), seq 1:142, ack 474, win 503, options [nop,nop,TS val 2168718770 ecr 3415999428], length 141
1623958319.536744 IP (tos 0x0, ttl 63, id 49355, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [.], cksum 0x8872 (incorrect -> 0x6d7e), ack 142, win 510, options [nop,nop,TS val 3415999429 ecr 2168718770], length 0
1623958319.537105 IP (tos 0x0, ttl 63, id 49356, offset 0, flags [DF], proto TCP (6), length 236)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [P.], cksum 0x892a (incorrect -> 0xb478), seq 474:658, ack 142, win 510, options [nop,nop,TS val 3415999429 ecr 2168718770], length 184
1623958319.537115 IP (tos 0x0, ttl 64, id 8246, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.62.4500 > 10.1.58.12.53720: Flags [.], cksum 0x8872 (incorrect -> 0x6cce), ack 658, win 502, options [nop,nop,TS val 2168718770 ecr 3415999429], length 0
1623958319.537843 IP (tos 0x0, ttl 64, id 8247, offset 0, flags [DF], proto TCP (6), length 574)
10.1.58.62.4500 > 10.1.58.12.53720: Flags [P.], cksum 0x8a7c (incorrect -> 0x7bbb), seq 142:664, ack 658, win 502, options [nop,nop,TS val 2168718771 ecr 3415999429], length 522
1623958319.537871 IP (tos 0x0, ttl 63, id 49357, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [.], cksum 0x8872 (incorrect -> 0x6abe), ack 664, win 506, options [nop,nop,TS val 3415999430 ecr 2168718771], length 0
1623958319.537997 IP (tos 0x0, ttl 64, id 8248, offset 0, flags [DF], proto TCP (6), length 83)
10.1.58.62.4500 > 10.1.58.12.53720: Flags [P.], cksum 0x8891 (incorrect -> 0x1f1c), seq 664:695, ack 658, win 502, options [nop,nop,TS val 2168718771 ecr 3415999430], length 31
1623958319.538012 IP (tos 0x0, ttl 63, id 49358, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [.], cksum 0x8872 (incorrect -> 0x6a9f), ack 695, win 506, options [nop,nop,TS val 3415999430 ecr 2168718771], length 0
1623958319.538031 IP (tos 0x0, ttl 64, id 8249, offset 0, flags [DF], proto TCP (6), length 52)
10.1.58.62.4500 > 10.1.58.12.53720: Flags [F.], cksum 0x8872 (incorrect -> 0x6aa2), seq 695, ack 658, win 502, options [nop,nop,TS val 2168718771 ecr 3415999430], length 0
1623958319.538761 IP (tos 0x0, ttl 63, id 49359, offset 0, flags [DF], proto TCP (6), length 83)
10.1.58.12.53720 > 10.1.58.62.4500: Flags [P.], cksum 0x8891 (incorrect -> 0x6019), seq 658:689, ack 696, win 506, options [nop,nop,TS val 3415999431 ecr 2168718771], length 31
It seems the connection resets mostly when the F flag has been set (to indicate the sender’s intention to terminate the connection to the receiving host) and instead of establishing a new TCP connection it is trying to use the same connection to send some data which causes the reset.
The packets are not getting lost and neither the application is having any kind of issues. I am just not sure why the reset [R] is being thrown. I checked the code and that seems to be fine as well.
After doing some research online I found kube-proxy Subtleties: Debugging an Intermittent Connection Reset | Kubernetes which felt like similar to my issue but after applying the work around that didn’t work for me.
Any ideas or suggestions no how to work through this?!