Can’t install microk8s on RHEL 7.8

joerg.keith · November 10, 2020, 2:09am

I can’t get microk8s 1.19.2 to start on RHEL 7.8, using snap. Same works fine on another machine running 7.7.

Lots of errors like:
Nov 10 04:50:25 177 microk8s.daemon-kubelet: E1110 04:50:25.789334 9288 pod_workers.go:191] Error syncing pod cb666780-24cd-4a40-bf17-9c9245204acc (“hostpath-provisioner-5c65fbdb4f-gvrfn_kube-system(cb666780-24cd-4a40-bf17-9c9245204acc)”), skipping: failed to ensure that the pod: cb666780-24cd-4a40-bf17-9c9245204acc cgroups exist and are correctly applied: failed to create container for [kubepods besteffort podcb666780-24cd-4a40-bf17-9c9245204acc] : mkdir /sys/fs/cgroup/memory/kubepods/besteffort/podcb666780-24cd-4a40-bf17-9c9245204acc: cannot allocate memory

The pods are stuck in ContainerCreating. Both machines (v7.7 and v7.8) were running clustered with microk8s 1.19.0 for several month without any problem, then I made some configuration changes to the workload (not microk8s) and couldn’t bring the cluster back up, ended up doing a fresh install of 1.19.2.

How do I diagnose this further?

Thanks,
Joerg

Cluster information:

Kubernetes version: microk8s 1.19.2
Cloud being used: bare-metal
Installation method: snap
Host OS: RHEL 7.8
CNI and version:
CRI and version:

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

kjackal · November 17, 2020, 6:53am

It is hard to say what is failing here.

Could you file an issue at [1] and attach the microk8s.inspect tarball?

Thanks

kjackal · November 17, 2020, 6:56am

Maybe this will address the problem:

github.com/docker/for-linux

Docker fails to start containers with cgroup memory allocation error.

opened 02:39PM - 29 Oct 19 UTC

JakeBonek

* [x] This is a bug report * [ ] This is a feature request * [x] I searched existing issues before opening this one  ### Expected behavior Docker should successfully start hello-world container. ### Actual behavior After a certain amount of time, docker fails to start any containers on a host with the following error: `[root@REDACTED]# docker run hello-world docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/memory/docker/fe4159ed6f4ec16af63ba0c2af53ec9c6b0c0c2ac42ff96f6816d5e28a821b4e: cannot allocate memory\"": unknown. ERRO[0000] error waiting for container: context canceled ` This issue has been fixed in the past by restarting the docker daemon or rebooting the machine although the docker daemon is active and running at the time of running the container. The machine has ample available memory and cpus and should have no problem starting the container. ### Steps to reproduce the behavior  **Output of `docker version`:** ``` Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:23:03 2018 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:25:29 2018 OS/Arch: linux/amd64 Experimental: false ``` **Output of `docker info`:** ``` Containers: 39 Running: 17 Paused: 0 Stopped: 22 Images: 39 Server Version: 18.06.1-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: 69663f0bd4b60df09991c08812a60108003fa340 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 3.10.0-957.1.3.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 56 Total Memory: 503.6GiB Name: REDACTED ID: UK7O:GWIS:TFRJ:JDUB:5SS7:GH6W:TA4K:NBQC:7W4V:YLZJ:Q2AV:UBXA Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: bridge-nf-call-ip6tables is disabled ``` **Additional environment details (AWS, VirtualBox, physical, etc.)** At the time of running the container, the host has 500GB of available memory and around 50+free cores.

Topic		Replies	Views
Microk8s.status not running; microk8s.inspect looks good General Discussions	4	3237	February 5, 2020
MicroK8s v1.17 released! microk8s	0	1130	December 11, 2019
MicroK8s v1.18 released! microk8s	0	2305	March 26, 2020
Microk8s not starting in Debian 10 VM microk8s	0	962	February 4, 2023
Microk8s.daemon-kubelite[2610202]: E0926 microk8s	0	35	September 26, 2024

Can’t install microk8s on RHEL 7.8

Cluster information:

Related topics