Hi,
I Have kuberenetes cluster with one master node and 4 worker nodes and its is scaled up with jenkins
In my existing kubernetes cluster certificates were expired, so renewed using below commands.
kubeadm certs renew all
cp /etc/kubernetes/admin.conf config
sudo chown $(id -u):$(id -g) config
after this i did
systemctl restart kubelet && systemctl enable kubelet
systemctl status kubelet
here im getting error as below,
root@si-rd22-kbnode:# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2023-08-23 08:11:09 UTC; 1min 39s ago
Docs: kubernetes. io doc /home/
Main PID: 124290 (kubelet)
Tasks: 35 (limit: 77157)
Memory: 52.1M
CGroup: /system.slice/kubelet.service
└─124290 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.7
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: I0823 08:11:12.686574 124290 request.go:601] Waited for 1.098730384s due to client-side throttling, not priority and fairness, request: POST:https:// …:/api/v1/namespaces/kube-system/pods
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.693978 124290 kubelet.go:1690] “Failed creating a mirror pod for” err=“pods "kube-controller-manager-si-rd22-kbnode" already exists” pod=“kube-system/kube-controller-manager-si-rd22-kbnode”
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.804661 124290 configmap.go:193] Couldn’t get configMap kube-system/coredns: failed to sync configmap cache: timed out waiting for the condition
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.804758 124290 configmap.go:193] Couldn’t get configMap kube-system/kube-proxy: failed to sync configmap cache: timed out waiting for the condition
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.804783 124290 nestedpendingoperations.go:335] Operation for “{volumeName:kubernetes.io/configmap/0ef2b42e-615f-4e48-9e47-38d681b3096b-config-volume podName:0ef2b42e-615f-4e48-9e47-38d681b3096b nodeName:}” failed. No retries permitted until 2023-08-23 08:11:13.304744674 +0000 UTC m=+3.683800931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume “config-volume” (UniqueName: “kubernetes.io/configmap/0ef2b42e-615f-4e48-9e47-38d681b3096b-config-volume”) pod “coredns-6d4b75cb6d-djmvw” (UID: “0ef2b42e-615f-4e48-9e47-38d681b3096b”) : failed to sync configmap cache: timed out waiting for the condition
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.804844 124290 nestedpendingoperations.go:335] Operation for “{volumeName:kubernetes.io/configmap/82ada0a8-7ea9-47d2-86aa-425a0d913c4d-kube-proxy podName:82ada0a8-7ea9-47d2-86aa-425a0d913c4d nodeName:}” failed. No retries permitted until 2023-08-23 08:11:13.304815798 +0000 UTC m=+3.683872055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume “kube-proxy” (UniqueName: “kubernetes.io/configmap/82ada0a8-7ea9-47d2-86aa-425a0d913c4d-kube-proxy”) pod “kube-proxy-4ffz7” (UID: “82ada0a8-7ea9-47d2-86aa-425a0d913c4d”) : failed to sync configmap cache: timed out waiting for the condition
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.804895 124290 configmap.go:193] Couldn’t get configMap kube-flannel/kube-flannel-cfg: failed to sync configmap cache: timed out waiting for the condition
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.804894 124290 configmap.go:193] Couldn’t get configMap kube-system/coredns: failed to sync configmap cache: timed out waiting for the condition
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.804976 124290 nestedpendingoperations.go:335] Operation for “{volumeName:kubernetes.io/configmap/08036020-78bf-4ea3-a681-303b19a7a35f-flannel-cfg podName:08036020-78bf-4ea3-a681-303b19a7a35f nodeName:}” failed. No retries permitted until 2023-08-23 08:11:13.304936397 +0000 UTC m=+3.683992654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume “flannel-cfg” (UniqueName: “kubernetes.io/configmap/08036020-78bf-4ea3-a681-303b19a7a35f-flannel-cfg”) pod “kube-flannel-ds-9lhnt” (UID: “08036020-78bf-4ea3-a681-303b19a7a35f”) : failed to sync configmap cache: timed out waiting for the condition
Aug 23 08:11:12 si-rd22-kbnode kubelet[124290]: E0823 08:11:12.805093 124290 nestedpendingoperations.go:335] Operation for “{volumeName:kubernetes.io/configmap/aa653b1f-e15f-4378-b794-711c79580836-config-volume podName:aa653b1f-e15f-4378-b794-711c79580836 nodeName:}” failed. No retries permitted until 2023-08-23 08:11:13.305060999 +0000 UTC m=+3.684117266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume “config-volume” (UniqueName: “kubernetes.io/configmap/aa653b1f-e15f-4378-b794-711c79580836-config-volume”) pod “coredns-6d4b75cb6d-2x2kc” (UID: “aa653b1f-e15f-4378-b794-711c79580836”) : failed to sync configmap cache: timed out waiting for the condition
Cluster information:
Installation method: apt
Host OS: ubuntu 18.04
CNI and version: kubernetes-cni- 0.8.7-00
KubeLet and version: kubelet - 1.24.4-00