Kubectl fail all commands - forbidden

General Discussions
1

Hello everyone,

I want to start saying that I’m not an expert but i’m facing a big problem with a production environment.
More people touched this system so I will explain just what I know.
A few weeks ago we had a problem with all certificates expired.
I know a colleague used the command kubeadm alpha certs to renew them and he solved the problem but, since then, all kubectl command fail with errors messages like:

  • Error from server (Forbidden): services is forbidden: User “kubernetes-admin” cannot list services in the namespace “kube-system”
    OR
  • Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User “kubernetes-admin” cannot ist clusterroles.rbac.authorization.k8s.io at the cluster scope
    OR
  • configmaps “kubeadm-config” is forbidden: User “kubernetes-admin” cannot get configmaps in the namespace “kube-systm”

I think probably we have somes issue maybe with a role or something like that but without kubectl I can’t modify anything. The dashboard we have installed is an old version and it not permit to modify roles cluster or roles etc.
The problem is probably something “stupid” that simply I can’t see. Anyone with patience to help me solve this problem please?

Thanks in advance.

Cluster information:

Kubernetes version: v1.13
Cloud being used: OVH
Installation method: kubeadm
Host OS: Ubuntu 16.04
CNI and version: v1.13
CRI and version: v1.13.

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

2

I’m not familiar with kubeadm, but have you tried using the kubeconfig files on the nodes (maybe the ones in the master are more powerful)?

I really don’t know what certs kubeadm uses for the controllers/workers, but that can’t hurt to try if you have more permissions with that.

3

Hello rata, first of all thx for answering.
i suppose that somewhere (into roles o secret or whatever) there are configured the old certs and for this reason any kubectl command fail with the error message “forbidden”.
Kubeadm is the way the cluster was installed at the beginning and the whole cluster worked fine until 1 year after, when certs expired and we had to renew all of them.
But, since then, now all kubectl commands fails as I said before, this mean I can’t modify any configuration via the usual way, using kubectl.

4

Yes, I think I got the picture. Have you tried what I suggested in the previous message?