Why all the master nodes has to be restarted for Cilium to come back

Cluster information:

Kubernetes version: 1.20
Cloud being used: bare-metal
Installation method: kops
Host OS: Ubuntu 18.04.6 LTS
CNI and version: cilium - v1.9.7
CRI and version: Docker - 19.3.15

We recently faced an issue with our Kubernetes cluster managed using kops. The cilium pods went down and refused to come up. We managed to re-install cilium but still, the pods were not coming up and as the cilium was down the kubectl commands were not working as well.

There was only one option of restarting the Master node to solve this issue so we restarted the master node leader but that too did not make the cilium pods to come up. It was only after restarting all three master nodes the cilium pods started to come up.

Do anyone have any idea why we had to restart all three master nodes for this? Why restarting only the leader of the Master node did not solve this issue? What was the reason for the need to restart all the Master nodes?

We were getting repetitive logs like the below from kube-system namespace:

msg=“Unable to update ipcache map entry on pod add” error=“ipcache entry for podIP 172.16.1.144 owned by kvstore or agent” hostIP=172.16.1.144 k8sNamespace=newrelic
msg=“failed to resolve identity” error=“identity 2780 not found” identity=2780 subsys=hubble"
msg=“Unable to update ipcache map entry on pod add” error=“ipcache entry for podIP 172.16.6.8 owned by kvstore or agent” hostIP=172.16.6.8 k8sNamespace=lacework

“message”: “2023-02-21 16:30:18.866104 I | embed: rejected connection from “127.0.0.1:41638” (error “tls: client didn’t provide a certificate”, ServerName “localhost”)n”,
“message”: “I0221 16:30:27.923074 1 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = “transport is closing"n”,
“message”: “E0221 16:31:42.226340 1 controller.go:302] controller-runtime/manager/controller/node “msg”=“Reconciler error” “error”=“error identifying node “ip-10-200-32-204.ec2.internal”: found instance “i-0f96b91ef5279dca2”, but state is “shutting-down”” “name”=“ip-10-200-32-204.ec2.internal” “namespace”=”” “reconciler group”=“” “reconciler kind”=“Node” n"