Kubernetes loops process kill infinitely

I have installed openstack-helm, but when the daemonset pod goes down, the process disappears, and the following message is output to the log. The new pod will not be able to start.

The process has already been killed, so what can I do to interrupt the process kill?
Should I clear the information I have somewhere?

Cluster information:

Kubernetes version:1.20 or later
Cloud being used: (put bare-metal if not on a public cloud)
Installation method:snap install microk8s --classic --channel=1.21
Host OS: ubuntu 20.04.2
CNI and version:calico 1.19.0
CRI and version: containerd v1.4.4

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.


May  7 13:34:52 os-com-02 microk8s.daemon-kubelite[2431787]: I0507 13:34:52.389092 2431787 pod_container_manager_linux.go:188] "Failed to kill all the processes attached to cgroup" cgroupName=[kubepods besteffort pod3c9ba130-b492-4cd1-a655-c8c247ac23f1] err="permission denied"
May  7 13:34:54 os-com-02 kernel: [163609.182768] kauditd_printk_skb: 5 callbacks suppressed
May  7 13:34:54 os-com-02 kernel: [163609.182770] audit: type=1400 audit(1620362094.381:12337): apparmor="DENIED" operation="signal" profile="libvirtd" pid=2431787 comm="kubelite" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.microk8s.daemon-kubelite"
May  7 13:34:54 os-com-02 microk8s.daemon-kubelite[2431787]: I0507 13:34:54.387153 2431787 pod_container_manager_linux.go:188] "Failed to kill all the processes attached to cgroup" cgroupName=[kubepods besteffort pod3c9ba130-b492-4cd1-a655-c8c247ac23f1] err="permission denied"
May  7 13:34:54 os-com-02 kernel: [163609.182996] audit: type=1400 audit(1620362094.385:12338): apparmor="DENIED" operation="signal" profile="libvirtd" pid=2431787 comm="kubelite" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.microk8s.daemon-kubelite"
May  7 13:34:54 os-com-02 kernel: [163609.183091] audit: type=1400 audit(1620362094.385:12339): apparmor="DENIED" operation="signal" profile="libvirtd" pid=2431787 comm="kubelite" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.microk8s.daemon-kubelite"
May  7 13:34:54 os-com-02 kernel: [163609.183176] audit: type=1400 audit(1620362094.385:12340): apparmor="DENIED" operation="signal" profile="libvirtd" pid=2431787 comm="kubelite" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.microk8s.daemon-kubelite"
May  7 13:34:54 os-com-02 kernel: [163609.183264] audit: type=1400 audit(1620362094.385:12341): apparmor="DENIED" operation="signal" profile="libvirtd" pid=2431787 comm="kubelite" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.microk8s.daemon-kubelite"

This is an apparmor denial.

I am not an apparmor specialist of some sort, but i think you need to edit the libvirt profile in /etc/apparmor.d/libvirt/TEMPLATE.qemu and add something like these.

signal (receive) peer=snap.microk8s.daemon-kubelite,
  signal (receive) peer=snap.microk8s.daemon-containerd,

Please note that i have not tried this at all. :blush:

thanks dear.

Check if the setting works when the same event occurs next time.
This time I reinstalled it.

The problem is that the process has already been killed.
But the process is called the kill function from kubenetes.
I thought that if the kill function was an error, it would retry indefinitely.