What does "--to-destination :0" mean in iptables rules generated by kube-proxy

Cluster information:

Kubernetes version: 1.22.5
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: centos7 (Linux shanghai 6.2.10-1.el7.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 4 13:58:32 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux)

CNI and version: flannel v0.19.2
CRI and version: docker 18.06

I happened to find that kube-proxy in my cluster output KUBE-SEP rules like this:

iptables -t nat -S | grep to-destination
-A KUBE-SEP-3N5FIJS2DHSEENTB -p tcp -m comment --comment "openelb-system/layer2-svc:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --random --persistent
-A KUBE-SEP-4U5LVJUBLXSJ5MTI -p tcp -m comment --comment "fabedge/service-hub" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --persistent
-A KUBE-SEP-7XYRAQHCDMWOB56C -p tcp -m comment --comment "fabedge/fabedge-operator" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --persistent
-A KUBE-SEP-ADM7IEO333MANCKX -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 
-A KUBE-SEP-AICPCLO2RM743DP7 -p tcp -m comment --comment "kubeedge/cloudcore:cloudhub-quic" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --random
-A KUBE-SEP-B4EZC3GTFACZCIHF -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 
-A KUBE-SEP-BY2PGPM3WYHNPOOJ -p tcp -m comment --comment "kubeedge/cloudcore:tunnelport" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --random
-A KUBE-SEP-GFU26SYT7KV3U67A -p tcp -m comment --comment "default/nginx:https" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0
-A KUBE-SEP-HM343LOZLYWXUGFZ -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0 --persistent
-A KUBE-SEP-HQEZVK3DRSQIM56O -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 
-A KUBE-SEP-HWLPML2HRDEHBAFK -p tcp -m comment --comment "default/nginx:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 
-A KUBE-SEP-I5JKKND3RXBIZWK7 -p tcp -m comment --comment "kubeedge/cloudcore:cloudhub-https" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --random
-A KUBE-SEP-JJ2LDLIAIZGYJX77 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 
-A KUBE-SEP-O52MRVHRAQJS3OJY -p tcp -m comment --comment "default/nginx:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 
-A KUBE-SEP-P7EKVAR4RJO622DO -p tcp -m comment --comment "kubeedge/cloudcore:cloudhub" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --random
-A KUBE-SEP-QBB7I64OUWAJZV3L -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0
-A KUBE-SEP-QNLFKXKINW5DMADC -p tcp -m comment --comment "kubeedge/cloudcore:cloudstream" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0 --random
-A KUBE-SEP-YI3PTOL7SCBZQ23K -p tcp -m comment --comment "openelb-system/openelb-admission:https-webhook" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0
-A KUBE-SEP-YXWM7XVI4J4II4AY -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0:0
-A KUBE-SEP-ZLW6P3TMQ6G24ULY -p tcp -m comment --comment "default/nginx:https" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination 0.0.0.0

As above displayed, the “–to-desination” parameter of DNAT target is “:0” or “0.0.0.0”, and appeared multiple times on each KUBE-SEP rule,

What does “:0” mean here? How did rules like those can forward traffic to a endpoint?

I read the source code of kube-proxy 1.22.5 and found this:

args = append(args, "-m", protocol, "-p", protocol, "-j", "DNAT", "--to-destination", epInfo.Endpoint)

It seems there is no chance to generate rules like above.

I also checked another cluster which deployed in similar enviroment except kernel is “3.10.0-1160.el7.x86_64”, it’s KUBE-SEP rules are what I expected:

 iptables -t nat -S | grep "to-destination"
-A KUBE-SEP-2BSL7AUA5YFD4HRD -p tcp -m comment --comment "fabedge-e2e-test/cloud-nginx-617:https" -m tcp -j DNAT --to-destination 10.233.102.152:30443
-A KUBE-SEP-45XJUG3YHUJ7V7XK -p tcp -m comment --comment "kubeedge/cloudcore:cloudhub-https" -m tcp -j DNAT --to-destination 10.22.46.11:10002
-A KUBE-SEP-4IR55IEMV6ID5FDL -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.233.102.138:53
-A KUBE-SEP-4IVK6V6RGNDRYL6Q -p tcp -m comment --comment "calico-apiserver/calico-api:apiserver" -m tcp -j DNAT --to-destination 10.233.75.18:5443
-A KUBE-SEP-4VUKK2VECAAMVKDP -p tcp -m comment --comment "fabedge/fabedge-operator" -m tcp -j DNAT --to-destination 10.233.116.88:3030
-A KUBE-SEP-4YXNTTQG3OUFS55S -p tcp -m comment --comment "fabedge-e2e-test/cloud-nginx-617:https" -m tcp -j DNAT --to-destination 10.233.75.31:30443
-A KUBE-SEP-6CJPTSKABJ5EOYJX -p tcp -m comment --comment "fabedge-e2e-test/edge-nginx-908:http" -m tcp -j DNAT --to-destination 10.237.195.16:30080
-A KUBE-SEP-6ELEAKTABD6HG7AY -p tcp -m comment --comment "fabedge-e2e-test/host-cloud-nginx-143:https" -m tcp -j DNAT --to-destination 10.22.46.11:30443
-A KUBE-SEP-6KVJ2LFJLWIPSUZI -p tcp -m comment --comment "fabedge-e2e-test/host-cloud-nginx-143:http" -m tcp -j DNAT --to-destination 10.22.46.25:30080
-A KUBE-SEP-6UWXO4ZXY463KUOG -p tcp -m comment --comment "fabedge-e2e-test/cloud-nginx-617:http" -m tcp -j DNAT --to-destination 10.233.116.103:30080
-A KUBE-SEP-6YYGNU4B54RRUVJI -p tcp -m comment --comment "default/nginx:https" -m tcp -j DNAT --to-destination 10.233.68.22:443
-A KUBE-SEP-AG6A2TE2TFTV5GA6 -p tcp -m comment --comment "fabedge-e2e-test/host-edge-nginx-367:https" -m tcp -j DNAT --to-destination 10.22.46.6:30443
-A KUBE-SEP-B7NMJGBOPUM2N7HH -p tcp -m comment --comment "calico-apiserver/calico-api:apiserver" -m tcp -j DNAT --to-destination 10.233.75.13:5443
-A KUBE-SEP-BAG5WD64LLF72XDL -p tcp -m comment --comment "kubeedge/cloudcore:tunnelport" -m tcp -j DNAT --to-destination 10.22.46.11:10004
-A KUBE-SEP-CGKQ7TMHICEMSUQX -p tcp -m comment --comment "default/nginx:http" -m tcp -j DNAT --to-destination 10.233.75.19:80
-A KUBE-SEP-CWUEIU3QRBVKEQ4Z -p tcp -m comment --comment "calico-system/calico-typha:calico-typha" -m tcp -j DNAT --to-destination 10.22.46.25:5473
-A KUBE-SEP-F3ITHY2UIMF5RYSY -p tcp -m comment --comment "kubeedge/cloudcore:cloudhub-quic" -m tcp -j DNAT --to-destination 10.22.46.11:10001
-A KUBE-SEP-FCK6UPK4D6AGZY2M -p tcp -m comment --comment "fabedge-e2e-test/host-cloud-nginx-143:https" -m tcp -j DNAT --to-destination 10.22.46.23:30443
-A KUBE-SEP-FYIT4BIQTU737E4Q -p tcp -m comment --comment "fabedge/fabdns:dns-tcp" -m tcp -j DNAT --to-destination 10.233.75.12:53
-A KUBE-SEP-JJNORG74KR6U55N7 -p tcp -m comment --comment "default/nginx:http" -m tcp -j DNAT --to-destination 10.233.75.20:80
-A KUBE-SEP-KBZ4O3HWKUHEXIEV -p tcp -m comment --comment "fabedge-e2e-test/host-cloud-nginx-143:http" -m tcp -j DNAT --to-destination 10.22.46.23:30080
-A KUBE-SEP-LQT22CWUGFFHSVXG -p tcp -m comment --comment "kubeedge/cloudcore:cloudstream" -m tcp -j DNAT --to-destination 10.22.46.11:10003
-A KUBE-SEP-NIIHHGFNJZ5CJEYD -p tcp -m comment --comment "fabedge-e2e-test/cloud-nginx-617:http" -m tcp -j DNAT --to-destination 10.233.102.152:30080
-A KUBE-SEP-O4QQ7PYHCSAD3DDV -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.233.102.138:53
-A KUBE-SEP-OBHFBUFD3JD5LGNZ -p tcp -m comment --comment "fabedge-e2e-test/edge-nginx-908:https" -m tcp -j DNAT --to-destination 10.237.195.16:30443
-A KUBE-SEP-OHELGRNTYNMFQ4ME -p tcp -m comment --comment "kubeedge/cloudcore:cloudhub" -m tcp -j DNAT --to-destination 10.22.46.11:10000
-A KUBE-SEP-OMPSVYWSYL6OGUAQ -p tcp -m comment --comment "default/nginx:http" -m tcp -j DNAT --to-destination 10.233.68.22:80
-A KUBE-SEP-P6P54WOUT5D5KIPA -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.233.102.136:53
-A KUBE-SEP-QJN4UMBCZOSUHAMX -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.233.102.136:9153
-A KUBE-SEP-S3LJCMSJURSPV66C -p tcp -m comment --comment "calico-system/calico-typha:calico-typha" -m tcp -j DNAT --to-destination 10.22.46.23:5473
-A KUBE-SEP-SPOEOSRKJ7CERS52 -p tcp -m comment --comment "calico-system/calico-typha:calico-typha" -m tcp -j DNAT --to-destination 10.22.46.11:5473
-A KUBE-SEP-TEY5JZWZXB3GZVR5 -p tcp -m comment --comment "fabedge-e2e-test/host-cloud-nginx-143:https" -m tcp -j DNAT --to-destination 10.22.46.25:30443
-A KUBE-SEP-UAGVYUCFFQBWFXPN -p tcp -m comment --comment "default/nginx:http" -m tcp -j DNAT --to-destination 10.233.116.87:80
-A KUBE-SEP-UFPK3ZHRPQM35HRS -p tcp -m comment --comment "default/nginx:https" -m tcp -j DNAT --to-destination 10.233.75.20:443
-A KUBE-SEP-UOBSJAPI7DZLBTPR -p tcp -m comment --comment "default/nginx:https" -m tcp -j DNAT --to-destination 10.233.75.19:443
-A KUBE-SEP-USW67C33ZIYU4F5G -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 10.22.46.11:6443
-A KUBE-SEP-UUY5KIIT7PRZECUW -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.233.102.136:53
-A KUBE-SEP-UYGPVTDY5P4T7CVN -p tcp -m comment --comment "fabedge-e2e-test/host-cloud-nginx-143:http" -m tcp -j DNAT --to-destination 10.22.46.11:30080
-A KUBE-SEP-XC6LYREWBA6L2MK2 -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 10.233.102.138:9153
-A KUBE-SEP-XEZU3EBQ2WW4MLUB -p tcp -m comment --comment "default/nginx:https" -m tcp -j DNAT --to-destination 10.233.116.87:443
-A KUBE-SEP-XPEYKKEBDZSIJHW5 -p udp -m comment --comment "fabedge/fabdns:dns-udp" -m udp -j DNAT --to-destination 10.233.75.12:53
-A KUBE-SEP-XT27XZITR6CMTMQB -p tcp -m comment --comment "fabedge/service-hub" -m tcp -j DNAT --to-destination 10.233.116.86:3000
-A KUBE-SEP-Y3O5BDI3CQOEDQOD -p tcp -m comment --comment "fabedge-e2e-test/cloud-nginx-617:https" -m tcp -j DNAT --to-destination 10.233.116.103:30443
-A KUBE-SEP-YKTS33H3MUEAGOO7 -p tcp -m comment --comment "fabedge-e2e-test/cloud-nginx-617:http" -m tcp -j DNAT --to-destination 10.233.75.31:30080
-A KUBE-SEP-YN6VQZ2MOGFBD53Y -p tcp -m comment --comment "calico-system/calico-kube-controllers-metrics:metrics-port" -m tcp -j DNAT --to-destination 10.233.75.16:9094
-A KUBE-SEP-ZKAWUXOV35PNOVEC -p tcp -m comment --comment "fabedge-e2e-test/host-edge-nginx-367:http" -m tcp -j DNAT --to-destination 10.22.46.6:30080

Can anyone help me figure out what happened in first cluster?

It seems wierd compatiblity problem, after I executed iptables command in kube-proxy container, the output is normal. I upgraded the kernel of all nodes of the first cluster, this might cause some problem.

I got the same output. That confused me a few days to understand how the K8S service sends traffic to backend pods. Finally I found the iptables version difference between kube-proxy container (which is 1.8) and the K8S node(which is 1.4). If running the kube-proxy iptables binary on K8S node, it will give the expected results.
e.g. # /run/containerd/io.containerd.runtime.v2.task/k8s.io/3d5b95000c749a6a91cbe7fdbcad0c97d3c7e83aa15076629fa2dd35dcc2b711/rootfs/usr/sbin/iptables -t nat -L -n

Cheers!