Please help, I can't make sense of RBAC issues in my cluster

General Discussions
1

Hi everyone,

I am trying to install Promtail on minikube using the official Helm chart. The Promtail pod can’t list pods and gives me the following errors message:

E0401 17:00:54.581785 1 reflector.go:127] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:451: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "[https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0](https://10.96.0.1/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0)": dial tcp 10.96.0.1:443: connect: connection refused
E0401 17:00:55.787988 1 reflector.go:127] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:451: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "[https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0](https://10.96.0.1/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0)": dial tcp 10.96.0.1:443: connect: connection refused

I don’t understand why, because I can curl the kubernetes api from the Promtail pod:

$ kubectl exec -it promtail-zs6r2 -c promtail -- curl -ks "https://kubernetes:443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0"
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "pods"
},
"code": 403
}

The second problem is that it seems the Promtail pod uses the system:anonymous user instead of its service account:

$ kubectl get pod promtail-zs6r2 -o jsonpath='{.spec.serviceAccountName}'
promtail
$ kubectl auth can-i --as=system:serviceaccount:default:promtail list pod
yes

Could anyone please explain to me what is going? Thanks a lot

Cluster information:

Kubernetes version: v1.20.2
Cloud being used: (put bare-metal if not on a public cloud): minikube v1.18.1
Installation method: minikube start --network-plugin cni
Host OS: VirtualBox
CNI and version: Calico (very recent version, not sure exactly which one)
CRI and version: Docker