Debugging metallb problems

I have been following https://mswis.com/load-balancing-with-microk8s-kubernetes/ to try and get an end to end example of an app with load balancing running. I deployed everything using the manifests from the blog and everything looks good, however it does not work. So as I dig deeper, it doesn’t seem like metallb is the problem. I’ve deployed 2 replicas of helloworld using the manifests in the blog (this is a 4 node micrk8s cluster 1.27 on rpi). I’ve deployed everything in the blog and no obvious issues… but I get nothing from the metallb external ip address - or almost nothing, it has worked a couple of times… looked in all the logs, but not seeing anything stand out. So finally I try hitting the helloworld-service from the node that the pod is running on and it works about 50% of the time, it just hangs when it doesn’t work. So I am assuming that is a pretty big problem that needs to be tackled first… but I have no idea why something would work intermittently like this. The hello world-service is node port and this is the manifest:

apiVersion: v1
kind: Service
metadata:
  name: helloworld
spec:
  type: NodePort
  selector:
    app: helloworld
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80

pretty simple - it all seems pretty simple, but now that I am down to the node port service being unreliable I don’t know how to further debug this at the moment. Any help on what to look at next would be appreciated.

Thanks
Bill

Ok, I dig more digging overnight…

NAMESPACE     LAST SEEN   TYPE      REASON         OBJECT                  MESSAGE
ingress       90s         Normal    nodeAssigned   service/ingress         announcing from node "uk8s04" with protocol "layer2"
kube-system   5m6s        Warning   Unhealthy      pod/calico-node-t65b2   Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
kube-system   96s         Warning   Unhealthy      pod/calico-node-fmtmr   Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
kube-system   3m4s        Warning   Unhealthy      pod/calico-node-f8mrh   Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
kube-system   97s         Warning   Unhealthy      pod/calico-node-x295h   Readiness probe failed: calico/node is not ready: felix is not ready: Get "http://localhost:9099/readiness": dial tcp 127.0.0.1:9099: connect: connection refused

looks like calico is my problem - and that makes sense, but I just installed via snap, so now I am really unsure of how to fix this - networking in microk8s is kind of a black box to me -

and finally the answer…

in Calico fails on 21.10 when running on raspberry pi · Issue #2680 · canonical/microk8s · GitHub

 sudo apt install linux-modules-extra-raspi

I don’t believe I saw this in the install instructions anywhere… but I was just following a random blog - so maybe it is in the docs…

yes, my bad, it is in the docs… The blog I was following was probably pre ubuntu 21.10 I guess.