I was running a HA cluster with 5 identical nodes just fine. We needed more resources, so I went to add a new node, with identical OS build and microk8s install (1.25.0)
The problem is pods running on the new node, cannot communicate with pods running on any other node and vice versa. The inter-pod communication is working just fine on the original five nodes.
I have been trying to figure out what makes the new node different, but I have been unable to. All nodes run on the same network and subnet, the same OS build (Ubuntu 20.04) and Microk8s version. Calico-node logs does not indicate any issues, and microk8s inspect also says everything is fine.
Microk8s inspection archive can be found here if relevant.