Network policies block only cross nodes traffic

General Discussions
1

Cluster information:

Kubernetes version:1.13
Cloud being used: bare-metal
Installation method: Hard Way
Host OS: CoreOS
CNI and version: Flannel 3.6.0, Calico 0.11.0
CRI and version:

Since the upgrade to the above version of Canal, ingress network policies fail to allow traffic when the destination POD is on a different node than the originator POD.
The policies were working fine before the upgrade. I.e. they were allowing traffic from specific pods.
Now they only allow the traffic if the source POD is on the same node as the destination POD.
I cannot see any error anywhere. Even the iptables on all nodes do not show any rejection.

Any idea where I can look?

Thanks,
Michele

2

Don’t know about canal, but a little about calico. Assuming the rules are okay, by any chance the hosts/interfaces changed during the upgrade? Can you make sure the hostendpoint objects exist and are using the proper interface?

3

Would you mind being more specific on how I can check? I’m not an expert on Calico.
Thanks,
Michele

4

You can use kubectl get hostendpoints to see the endpoints in calico. But you really need to use calicoctl.

However, as I’m not familiar with canal, not sure if that is the case in canal too :-/

5

kubectl get hostendpoints returns nothing. I tried for all namespaces as well, but nothing is returned.

6

Oh, okay, just reading again and is not communicating to the node that fails. Sorry, misread.

I’d check network policies and global network policies, but not sure how those interact with canal :-/.

Also, you can see iptables manually in the node and see if any rule is dropping that. Or run tcpdump in both nodes and see if the packet arrives, etc.