Kube-controller-manager is missing on GKE


So, here’s the story. We are using GKE and are running a few services in a couple of clusters with a couple namespaces, etc. Today, I was trying to do some troubleshooting of certain things and tried to access kube-controller-manager as described in many various sources over the internet.

  1. So I have connected to a cluster using gcloud cli
  2. I have tried performing kubectl get pods -namespace=kube-system
  3. I observed no kube-controller-manager there at all
  4. I have connected to the VM that is running k8s node (I have actually tried connecting to all of them with no luck) and tried searching for /var/logs/kube-controller-manager.log - no luck
  5. I have also checked the /etc/kubernetes/manifests and found only kube-proxy manifests there.

I am certain that not that long ago I have tried doing the same and kube-controller-manager pod was there and I could find all the logs.
Can someone point me to the right direction here?
Is this something related to the fact that I am not connecting to the correct node? (I can’t see any special nodes to be honest, I tried connecting to them all, none of them seemed anyhow master)
Is it something new in terms of how k8s is set up on GKE?
Am I missing some IAM permissions?

Please help.


Controller manager runs on the master machine. That machine is not a
node in the cluster.