RecentStats: unable to find data in memory cache | Error getting data for container race condition

General Discussions
1

We recently upgraded our server to the 1.21.10 version of k8s.
The master and worker nodes are both up and in Ready status.
However, I could see the following logs that are piling up both in master and worker nodes.

Mar 16 08:08:05 champa kubelet[2045]: W0316 08:08:05.949461    2045 manager.go:696] Error getting data for container /kubesvc.slice/docker.service because of race condition

Mar 16 08:08:06 champa kubelet[2045]: E0316 08:08:06.728558    2045 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubesvc.slice/docker.service\": RecentStats: unable to find data in memory cache], [\"/kubesvc.slice/kubelet.service\": RecentStats: unable to find data in memory cache]"
Mar 16 08:08:06 champa kubelet[2045]: E0316 08:08:06.728862    2045 summary_sys_containers.go:82] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubesvc.slice/kubelet.service\": failed to get container info for \"/kubesvc.slice/kubelet.service\": partial failures: [\"/kubesvc.slice/kubelet.service\": RecentStats: unable to find data in memory cache]" containerName="/kubesvc.slice/kubelet.service"

I saw blogs to check about symlink docker but i don’t see any.

root@champa:~# ll /var/lib/ | grep docker
drwx--x--x 15 root      root      4096 Mar 16 07:16 docker/
drwxr-xr-x  2 root      root      4096 Mar 16 05:10 docker-engine/
drwxr-xr-x  3 root      root      4096 Mar 16 05:10 dockershim/

Any help/pointers will be really helpful

Cluster information:

docker information:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.0-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 55
  Running: 49
  Paused: 0
  Stopped: 6
 Images: 84
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc version: v1.0.3-0-gf46b6ba
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1068-aws
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.58GiB
 Name: champa
 ID: random
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Kubernetes version:

root@champa:~# dpkg -l | grep kube
hi  kubectl                           1.21.10-00                          amd64        Kubernetes Command Line Tool
hi  kubelet                           1.21.10-00                          amd64        Kubernetes Node Agent
hi  kubernetes-cni                    0.8.7-00                            amd64        Kubernetes CNI
root@champa:~# uname  -a
Linux champa 5.4.0-1068-aws #72~18.04.1-Ubuntu SMP Thu Mar 3 08:49:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
2

I have upgraded the k8s cluster to 1.21.11 and the docker version to 20.10.13 but still, the error persists.
Any pointers/directions is really appreciated.

3

The versions are almost the same when encountering the same problems :upside_down_face:

4

solve


cd /sys/fs/cgroup

find -name "kubelet.service"

rmdir {dir}

It seems that cgroups cannot be nested. You must create a cgroup of the same level, such as

mkdir -p /sys/fs/cgroup/hugetlb/kubelet.slice

mkdir -p /sys/fs/cgroup/hugetlb/system.slice