I sometimes see network traffic with dst address and port pointing to an actual kubernetes service “leaving” the kubernetes node ending up at the firewall. If I inspect the rules which kubernetes add I don’t see how this can even happen. I saw that even for the case where there are no PODs for a specific service IP a rule is added which REJECTS packets.
So the question would be what leads to the situation which let’s me observe network packets that has as destination ip a ip address from the cluster ip range (ip address from the range configured as --service-cluster-ip-range at the apiserver).