After the server restarts, the slave node cannot join the master node

General Discussions
,
1

After the server restarts, it cannot join the main node. All components are switching back and forth between running and CrashLoopBackOff, and there is no specific explanation for the events displayed on the components. I hope experts can help take a look
CoreDNS, kube proxy, kube flannel, hami scheduler, hami device plugin component events are as follows:

kubectl describe pod -n kube-system coredns-674b8bbfcf-h7q4l
Events:
Type Reason Age From Message

Normal Scheduled 14m default-scheduler Successfully assigned kube-system/coredns-674b8bbfcf-h7q4l to phy-10-19-196-71
Normal Created 11m (x5 over 14m) kubelet Created container: coredns
Normal Started 11m (x5 over 14m) kubelet Started container coredns
Normal Killing 10m (x5 over 14m) kubelet Stopping container coredns
Normal SandboxChanged 10m (x5 over 13m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning Unhealthy 10m kubelet Readiness probe failed: Get “”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning BackOff 3m41s (x58 over 13m) kubelet Back-off restarting failed container coredns in pod coredns-674b8bbfcf-h7q4l_kube-system(0fbfab24-420c-4316-bead-fbe00c9411aa)
Normal Pulled 64s (x8 over 14m) kubelet Container image “registry.k8s.io/coredns/coredns:v1.12.0” already present on machine

kubectl describe pod -n kube-system coredns-674b8bbfcf-qj78k
Events:
Type Reason Age From Message

Normal Scheduled 13m default-scheduler Successfully assigned kube-system/coredns-674b8bbfcf-qj78k to phy-10-19-196-71
Normal Started 11m (x5 over 13m) kubelet Started container coredns
Normal Killing 3m54s (x5 over 13m) kubelet Stopping container coredns
Warning Unhealthy 3m51s kubelet Readiness probe failed: Get “”: dial tcp 10.244.1.65:8181: connect: connection refused
Normal SandboxChanged 3m48s (x5 over 13m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning BackOff 2m34s (x26 over 13m) kubelet Back-off restarting failed container coredns in pod coredns-674b8bbfcf-qj78k_kube-system(d51b857d-3daa-444e-b12b-07722d6d0fa8)
Normal Pulled 2m23s (x6 over 13m) kubelet Container image “” already present on machine
Normal Created 2m23s (x6 over 13m) kubelet Created container: coredns

kubectl describe pod -n kube-system hami-device-plugin-hc6vm
Events:
Type Reason Age From Message

Normal Scheduled 10m default-scheduler Successfully assigned kube-system/hami-device-plugin-hc6vm to phy-10-19-196-71
Normal SandboxChanged 7m36s (x2 over 9m4s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 7m22s (x3 over 10m) kubelet Container image “docker.io/projecthami/hami:v2.7.1” already present on machine
Normal Created 7m22s (x3 over 10m) kubelet Created container: device-plugin
Normal Started 7m14s (x3 over 10m) kubelet Started container device-plugin
Normal Pulled 7m14s (x3 over 10m) kubelet Container image “docker.io/projecthami/hami:v2.7.1” already present on machine
Normal Created 7m14s (x3 over 10m) kubelet Created container: vgpu-monitor
Normal Started 7m5s (x3 over 10m) kubelet Started container vgpu-monitor
Normal Killing 5m50s (x3 over 9m5s) kubelet Stopping container device-plugin
Normal Killing 5m50s (x3 over 9m5s) kubelet Stopping container vgpu-monitor
Warning BackOff 2m35s (x13 over 7m36s) kubelet Back-off restarting failed container device-plugin in pod hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)
Warning BackOff 2m35s (x13 over 7m36s) kubelet Back-off restarting failed container vgpu-monitor in pod hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)

kubectl describe pod -n kube-system hami-scheduler-7d5bd56886-jg4dn
Events:
Type Reason Age From Message

Normal Scheduled 10m default-scheduler Successfully assigned kube-system/hami-scheduler-7d5bd56886-jg4dn to phy-10-19-196-71
Normal SandboxChanged 4m35s (x2 over 4m37s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Created 4m21s (x3 over 10m) kubelet Created container: kube-scheduler
Normal Started 4m21s (x3 over 10m) kubelet Started container kube-scheduler
Normal Pulled 4m21s (x3 over 10m) kubelet Container image “” already present on machine
Normal Created 4m21s (x3 over 10m) kubelet Created container: vgpu-scheduler-extender
Normal Started 4m21s (x3 over 10m) kubelet Started container vgpu-scheduler-extender
Normal Killing 4m20s (x3 over 4m38s) kubelet Stopping container kube-scheduler
Normal Killing 4m20s (x3 over 4m38s) kubelet Stopping container vgpu-scheduler-extender
Warning BackOff 100s (x13 over 4m35s) kubelet Back-off restarting failed container kube-scheduler in pod hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)
Warning BackOff 100s (x12 over 4m35s) kubelet Back-off restarting failed container vgpu-scheduler-extender in pod hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)
Normal Pulled 21s (x6 over 10m) kubelet Container image “” already present on machine

kubectl describe pod -n kube-system kube-proxy-2xv2v
Events:
Type Reason Age From Message

Normal Scheduled 4m17s default-scheduler Successfully assigned kube-system/kube-proxy-2xv2v to phy-10-19-196-71
Normal Pulled 14s (x2 over 4m17s) kubelet Container image “” already present on machine
Normal Created 14s (x2 over 4m17s) kubelet Created container: kube-proxy
Normal Started 14s (x2 over 4m17s) kubelet Started container kube-proxy
Normal Killing 13s (x2 over 15s) kubelet Stopping container kube-proxy
Normal SandboxChanged 12s (x2 over 14s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning BackOff 10s (x3 over 12s) kubelet Back-off restarting failed container kube-proxy in pod kube-proxy-2xv2v_kube-system(e726839f-0bd7-4baa-8606-f573082735cd)

kubectl describe pod -n kube-flannel kube-flannel-ds-4qv88
Events:
Type Reason Age From Message

Normal Scheduled 7m17s default-scheduler Successfully assigned kube-flannel/kube-flannel-ds-4qv88 to phy-10-19-196-71
Normal Pulled 4m22s (x2 over 7m16s) kubelet Container image “ ” already present on machine
Normal Started 4m22s (x2 over 7m16s) kubelet Started container install-cni
Normal Created 4m22s (x2 over 7m16s) kubelet Created container: install-cni
Normal Started 4m21s (x2 over 7m15s) kubelet Started container kube-flannel
Normal Created 4m21s (x2 over 7m15s) kubelet Created container: kube-flannel
Normal Killing 4m20s (x2 over 4m24s) kubelet Stopping container kube-flannel
Normal SandboxChanged 4m19s (x2 over 4m23s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning BackOff 4m19s kubelet Back-off restarting failed container install-cni-plugin in pod kube-flannel-ds-4qv88_kube-flannel(92613278-7191-460b-ad10-56c0107168a1)
Normal Started 4m18s (x3 over 7m16s) kubelet Started container install-cni-plugin
Normal Created 4m18s (x3 over 7m17s) kubelet Created container: install-cni-plugin
Normal Pulled 4m18s (x3 over 7m17s) kubelet Container image “ ” already present on machine
Normal Pulled 2m15s (x4 over 7m15s) kubelet Container image “ ” already present on machine
Warning BackOff 6s (x9 over 4m16s) kubelet Back-off restarting failed container kube-flannel in pod kube-flannel-ds-4qv88_kube-flannel(92613278-7191-460b-ad10-56c0107168a1)

k8s log:
sudo journalctl -u kubelet --no-pager | tail -n 100
Jan 15 21:42:33 phy-10-19-196-71 kubelet[163303]: E0115 21:42:33.490332 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“[failed to “StartContainer” for “kube-scheduler” with CrashLoopBackOff: “back-off 5m0s restarting failed container=kube-scheduler pod=hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)”, failed to “StartContainer” for “vgpu-scheduler-extender” with CrashLoopBackOff: “back-off 5m0s restarting failed container=vgpu-scheduler-extender pod=hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)”]” pod=“kube-system/hami-scheduler-7d5bd56886-jg4dn” podUID=“04519158-ad96-4e76-a3bc-46bcaa22576b”
Jan 15 21:42:33 phy-10-19-196-71 kubelet[163303]: I0115 21:42:33.518765 163303 scope.go:117] “RemoveContainer” containerID=“356c0a67fa91f0be7f89e9b561f07fbbc1544673ad0dd2bff31dd9a498d40b1c”
Jan 15 21:42:33 phy-10-19-196-71 kubelet[163303]: E0115 21:42:33.519089 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=kube-green-controller-manager-7b7cf6b8cc-p8r8l_kube-green(b5f89fb1-ccac-451b-a6dd-770fc4b510e1)”” pod=“kube-green/kube-green-controller-manager-7b7cf6b8cc-p8r8l” podUID=“b5f89fb1-ccac-451b-a6dd-770fc4b510e1”
Jan 15 21:42:33 phy-10-19-196-71 kubelet[163303]: E0115 21:42:33.541767 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:34 phy-10-19-196-71 kubelet[163303]: E0115 21:42:34.542292 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:35 phy-10-19-196-71 kubelet[163303]: I0115 21:42:35.489790 163303 scope.go:117] “RemoveContainer” containerID=“a39c6310624f83a0028cb6f59a00c358c1a93c92409573dc495ffa0aa076d904”
Jan 15 21:42:35 phy-10-19-196-71 kubelet[163303]: E0115 21:42:35.490114 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=eraser-controller-manager-5b8f646476-67spl_eraser-system(7b532785-af5f-40af-90ff-021a5520e8f9)”” pod=“eraser-system/eraser-controller-manager-5b8f646476-67spl” podUID=“7b532785-af5f-40af-90ff-021a5520e8f9”
Jan 15 21:42:35 phy-10-19-196-71 kubelet[163303]: I0115 21:42:35.530213 163303 scope.go:117] “RemoveContainer” containerID=“76164b40db20140b3bb12e87dad559103aeee28290a66b4e48713cf06638db93”
Jan 15 21:42:35 phy-10-19-196-71 kubelet[163303]: E0115 21:42:35.530537 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-controller” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-controller pod=cert-manager-5f7b5dbfbc-lw2pz_cert-manager(f9835c2c-7bf9-44a8-b36f-e52e8e9f15c2)”” pod=“cert-manager/cert-manager-5f7b5dbfbc-lw2pz” podUID=“f9835c2c-7bf9-44a8-b36f-e52e8e9f15c2”
Jan 15 21:42:35 phy-10-19-196-71 kubelet[163303]: E0115 21:42:35.543311 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:36 phy-10-19-196-71 kubelet[163303]: E0115 21:42:36.544380 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:37 phy-10-19-196-71 kubelet[163303]: E0115 21:42:37.544739 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:38 phy-10-19-196-71 kubelet[163303]: E0115 21:42:38.545906 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:39 phy-10-19-196-71 kubelet[163303]: I0115 21:42:39.516961 163303 scope.go:117] “RemoveContainer” containerID=“be6bc1b003d6773279666c38366576e6efdbdf3031a4b7bffbe5889e437daa08”
Jan 15 21:42:39 phy-10-19-196-71 kubelet[163303]: E0115 21:42:39.517269 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “coredns” with CrashLoopBackOff: “back-off 1m20s restarting failed container=coredns pod=coredns-674b8bbfcf-lv6r7_kube-system(b597e5f9-9715-489e-b0a0-bc21b1a0e0de)”” pod=“kube-system/coredns-674b8bbfcf-lv6r7” podUID=“b597e5f9-9715-489e-b0a0-bc21b1a0e0de”
Jan 15 21:42:39 phy-10-19-196-71 kubelet[163303]: E0115 21:42:39.546394 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:40 phy-10-19-196-71 kubelet[163303]: E0115 21:42:40.447032 163303 file.go:104] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:40 phy-10-19-196-71 kubelet[163303]: E0115 21:42:40.547576 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:41 phy-10-19-196-71 kubelet[163303]: E0115 21:42:41.548137 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:42 phy-10-19-196-71 kubelet[163303]: E0115 21:42:42.549342 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:43 phy-10-19-196-71 kubelet[163303]: E0115 21:42:43.550177 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:44 phy-10-19-196-71 kubelet[163303]: E0115 21:42:44.551344 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:45 phy-10-19-196-71 kubelet[163303]: I0115 21:42:45.514494 163303 scope.go:117] “RemoveContainer” containerID=“9633c42b5f8bf40406109d663a5e2e755a3e9172227f840bf81c92440c689e9b”
Jan 15 21:42:45 phy-10-19-196-71 kubelet[163303]: E0115 21:42:45.514902 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-cainjector” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-7d5b44bb96-rsfv6_cert-manager(444946a1-c568-45ee-9e58-b525c007da27)”” pod=“cert-manager/cert-manager-cainjector-7d5b44bb96-rsfv6” podUID=“444946a1-c568-45ee-9e58-b525c007da27”
Jan 15 21:42:45 phy-10-19-196-71 kubelet[163303]: E0115 21:42:45.551516 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: I0115 21:42:46.489814 163303 scope.go:117] “RemoveContainer” containerID=“6f4893511a0df487f9d098feb01e0a868c4b637de73e7cf047b84d9c9f9f26a5”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: I0115 21:42:46.489856 163303 scope.go:117] “RemoveContainer” containerID=“57d385d6807348609c7e87abc382ca5d6ff36afbcc85e016722430476819e191”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: I0115 21:42:46.490119 163303 scope.go:117] “RemoveContainer” containerID=“bc10e6ceeb223bc90d33cd5d412b327e39bf5f68762dad26d913bf9d3db27d5a”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: E0115 21:42:46.490222 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“[failed to “StartContainer” for “kube-scheduler” with CrashLoopBackOff: “back-off 5m0s restarting failed container=kube-scheduler pod=hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)”, failed to “StartContainer” for “vgpu-scheduler-extender” with CrashLoopBackOff: “back-off 5m0s restarting failed container=vgpu-scheduler-extender pod=hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)”]” pod=“kube-system/hami-scheduler-7d5bd56886-jg4dn” podUID=“04519158-ad96-4e76-a3bc-46bcaa22576b”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: E0115 21:42:46.490478 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=capsule-controller-manager-584d6c96d9-b8l7k_capsule-system(833e1bde-25dc-4611-82bc-bbe2e0eb6bc4)”” pod=“capsule-system/capsule-controller-manager-584d6c96d9-b8l7k” podUID=“833e1bde-25dc-4611-82bc-bbe2e0eb6bc4”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: I0115 21:42:46.490625 163303 scope.go:117] “RemoveContainer” containerID=“7c83006d097765cae782775d8d43f4b3d15503765a49ffa81fb3e03306071397”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: I0115 21:42:46.490669 163303 scope.go:117] “RemoveContainer” containerID=“b09ba39cef0090ed8a7513328d0232d9df1fd50aa71c0fb4b73e027d490d0c6a”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: E0115 21:42:46.491035 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“[failed to “StartContainer” for “device-plugin” with CrashLoopBackOff: “back-off 5m0s restarting failed container=device-plugin pod=hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)”, failed to “StartContainer” for “vgpu-monitor” with CrashLoopBackOff: “back-off 5m0s restarting failed container=vgpu-monitor pod=hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)”]” pod=“kube-system/hami-device-plugin-hc6vm” podUID=“0e107896-30bf-406e-8ef4-4ccab4008c56”
Jan 15 21:42:46 phy-10-19-196-71 kubelet[163303]: E0115 21:42:46.552528 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:47 phy-10-19-196-71 kubelet[163303]: I0115 21:42:47.518393 163303 scope.go:117] “RemoveContainer” containerID=“356c0a67fa91f0be7f89e9b561f07fbbc1544673ad0dd2bff31dd9a498d40b1c”
Jan 15 21:42:47 phy-10-19-196-71 kubelet[163303]: E0115 21:42:47.518684 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=kube-green-controller-manager-7b7cf6b8cc-p8r8l_kube-green(b5f89fb1-ccac-451b-a6dd-770fc4b510e1)”” pod=“kube-green/kube-green-controller-manager-7b7cf6b8cc-p8r8l” podUID=“b5f89fb1-ccac-451b-a6dd-770fc4b510e1”
Jan 15 21:42:47 phy-10-19-196-71 kubelet[163303]: E0115 21:42:47.552951 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:48 phy-10-19-196-71 kubelet[163303]: E0115 21:42:48.553406 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:49 phy-10-19-196-71 kubelet[163303]: I0115 21:42:49.273053 163303 pod_container_deletor.go:80] “Container not found in pod’s containers” containerID=“b85cb36db73bf9061a0cdb964c96354eed22c68c802902187c99f10231c6f7b6”
Jan 15 21:42:49 phy-10-19-196-71 kubelet[163303]: I0115 21:42:49.273104 163303 scope.go:117] “RemoveContainer” containerID=“5514e9844b083564a4368239d1702bf5813d7dfbb718af0803fd460cb1dccb57”
Jan 15 21:42:49 phy-10-19-196-71 kubelet[163303]: E0115 21:42:49.409777 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-webhook” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-webhook pod=cert-manager-webhook-69459b8974-bgdbx_cert-manager(36f533fd-6fe2-41b9-9cfe-eefa18459e63)”” pod=“cert-manager/cert-manager-webhook-69459b8974-bgdbx” podUID=“36f533fd-6fe2-41b9-9cfe-eefa18459e63”
Jan 15 21:42:49 phy-10-19-196-71 kubelet[163303]: E0115 21:42:49.554126 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:50 phy-10-19-196-71 kubelet[163303]: I0115 21:42:50.281487 163303 scope.go:117] “RemoveContainer” containerID=“f2fe6c4aa6033c18e850ca81112c7a8c50eaf41eaac1170d607d9fae8fd51e23”
Jan 15 21:42:50 phy-10-19-196-71 kubelet[163303]: E0115 21:42:50.281797 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-webhook” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-webhook pod=cert-manager-webhook-69459b8974-bgdbx_cert-manager(36f533fd-6fe2-41b9-9cfe-eefa18459e63)”” pod=“cert-manager/cert-manager-webhook-69459b8974-bgdbx” podUID=“36f533fd-6fe2-41b9-9cfe-eefa18459e63”
Jan 15 21:42:50 phy-10-19-196-71 kubelet[163303]: I0115 21:42:50.496521 163303 scope.go:117] “RemoveContainer” containerID=“76164b40db20140b3bb12e87dad559103aeee28290a66b4e48713cf06638db93”
Jan 15 21:42:50 phy-10-19-196-71 kubelet[163303]: E0115 21:42:50.496800 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-controller” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-controller pod=cert-manager-5f7b5dbfbc-lw2pz_cert-manager(f9835c2c-7bf9-44a8-b36f-e52e8e9f15c2)”” pod=“cert-manager/cert-manager-5f7b5dbfbc-lw2pz” podUID=“f9835c2c-7bf9-44a8-b36f-e52e8e9f15c2”
Jan 15 21:42:50 phy-10-19-196-71 kubelet[163303]: I0115 21:42:50.540314 163303 scope.go:117] “RemoveContainer” containerID=“a39c6310624f83a0028cb6f59a00c358c1a93c92409573dc495ffa0aa076d904”
Jan 15 21:42:50 phy-10-19-196-71 kubelet[163303]: E0115 21:42:50.540558 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=eraser-controller-manager-5b8f646476-67spl_eraser-system(7b532785-af5f-40af-90ff-021a5520e8f9)”” pod=“eraser-system/eraser-controller-manager-5b8f646476-67spl” podUID=“7b532785-af5f-40af-90ff-021a5520e8f9”
Jan 15 21:42:50 phy-10-19-196-71 kubelet[163303]: E0115 21:42:50.555309 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:51 phy-10-19-196-71 kubelet[163303]: I0115 21:42:51.490132 163303 scope.go:117] “RemoveContainer” containerID=“be6bc1b003d6773279666c38366576e6efdbdf3031a4b7bffbe5889e437daa08”
Jan 15 21:42:51 phy-10-19-196-71 kubelet[163303]: E0115 21:42:51.490396 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “coredns” with CrashLoopBackOff: “back-off 1m20s restarting failed container=coredns pod=coredns-674b8bbfcf-lv6r7_kube-system(b597e5f9-9715-489e-b0a0-bc21b1a0e0de)”” pod=“kube-system/coredns-674b8bbfcf-lv6r7” podUID=“b597e5f9-9715-489e-b0a0-bc21b1a0e0de”
Jan 15 21:42:51 phy-10-19-196-71 kubelet[163303]: E0115 21:42:51.555882 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:52 phy-10-19-196-71 kubelet[163303]: E0115 21:42:52.556020 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:53 phy-10-19-196-71 kubelet[163303]: I0115 21:42:53.245074 163303 scope.go:117] “RemoveContainer” containerID=“f2fe6c4aa6033c18e850ca81112c7a8c50eaf41eaac1170d607d9fae8fd51e23”
Jan 15 21:42:53 phy-10-19-196-71 kubelet[163303]: E0115 21:42:53.245433 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-webhook” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-webhook pod=cert-manager-webhook-69459b8974-bgdbx_cert-manager(36f533fd-6fe2-41b9-9cfe-eefa18459e63)”” pod=“cert-manager/cert-manager-webhook-69459b8974-bgdbx” podUID=“36f533fd-6fe2-41b9-9cfe-eefa18459e63”
Jan 15 21:42:53 phy-10-19-196-71 kubelet[163303]: E0115 21:42:53.557008 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:53 phy-10-19-196-71 kubelet[163303]: I0115 21:42:53.778525 163303 scope.go:117] “RemoveContainer” containerID=“f2fe6c4aa6033c18e850ca81112c7a8c50eaf41eaac1170d607d9fae8fd51e23”
Jan 15 21:42:53 phy-10-19-196-71 kubelet[163303]: E0115 21:42:53.778836 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-webhook” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-webhook pod=cert-manager-webhook-69459b8974-bgdbx_cert-manager(36f533fd-6fe2-41b9-9cfe-eefa18459e63)”” pod=“cert-manager/cert-manager-webhook-69459b8974-bgdbx” podUID=“36f533fd-6fe2-41b9-9cfe-eefa18459e63”
Jan 15 21:42:54 phy-10-19-196-71 kubelet[163303]: E0115 21:42:54.557744 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:55 phy-10-19-196-71 kubelet[163303]: E0115 21:42:55.557954 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:56 phy-10-19-196-71 kubelet[163303]: E0115 21:42:56.558872 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:57 phy-10-19-196-71 kubelet[163303]: E0115 21:42:57.559549 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:58 phy-10-19-196-71 kubelet[163303]: E0115 21:42:58.560503 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:42:59 phy-10-19-196-71 kubelet[163303]: I0115 21:42:59.489276 163303 scope.go:117] “RemoveContainer” containerID=“7c83006d097765cae782775d8d43f4b3d15503765a49ffa81fb3e03306071397”
Jan 15 21:42:59 phy-10-19-196-71 kubelet[163303]: I0115 21:42:59.489312 163303 scope.go:117] “RemoveContainer” containerID=“b09ba39cef0090ed8a7513328d0232d9df1fd50aa71c0fb4b73e027d490d0c6a”
Jan 15 21:42:59 phy-10-19-196-71 kubelet[163303]: E0115 21:42:59.489642 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“[failed to “StartContainer” for “device-plugin” with CrashLoopBackOff: “back-off 5m0s restarting failed container=device-plugin pod=hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)”, failed to “StartContainer” for “vgpu-monitor” with CrashLoopBackOff: “back-off 5m0s restarting failed container=vgpu-monitor pod=hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)”]” pod=“kube-system/hami-device-plugin-hc6vm” podUID=“0e107896-30bf-406e-8ef4-4ccab4008c56”
Jan 15 21:42:59 phy-10-19-196-71 kubelet[163303]: E0115 21:42:59.560802 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:00 phy-10-19-196-71 kubelet[163303]: E0115 21:43:00.446615 163303 file.go:104] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:00 phy-10-19-196-71 kubelet[163303]: I0115 21:43:00.496701 163303 scope.go:117] “RemoveContainer” containerID=“6f4893511a0df487f9d098feb01e0a868c4b637de73e7cf047b84d9c9f9f26a5”
Jan 15 21:43:00 phy-10-19-196-71 kubelet[163303]: I0115 21:43:00.496738 163303 scope.go:117] “RemoveContainer” containerID=“57d385d6807348609c7e87abc382ca5d6ff36afbcc85e016722430476819e191”
Jan 15 21:43:00 phy-10-19-196-71 kubelet[163303]: I0115 21:43:00.496971 163303 scope.go:117] “RemoveContainer” containerID=“9633c42b5f8bf40406109d663a5e2e755a3e9172227f840bf81c92440c689e9b”
Jan 15 21:43:00 phy-10-19-196-71 kubelet[163303]: E0115 21:43:00.497104 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“[failed to “StartContainer” for “kube-scheduler” with CrashLoopBackOff: “back-off 5m0s restarting failed container=kube-scheduler pod=hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)”, failed to “StartContainer” for “vgpu-scheduler-extender” with CrashLoopBackOff: “back-off 5m0s restarting failed container=vgpu-scheduler-extender pod=hami-scheduler-7d5bd56886-jg4dn_kube-system(04519158-ad96-4e76-a3bc-46bcaa22576b)”]” pod=“kube-system/hami-scheduler-7d5bd56886-jg4dn” podUID=“04519158-ad96-4e76-a3bc-46bcaa22576b”
Jan 15 21:43:00 phy-10-19-196-71 kubelet[163303]: E0115 21:43:00.497263 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-cainjector” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-7d5b44bb96-rsfv6_cert-manager(444946a1-c568-45ee-9e58-b525c007da27)”” pod=“cert-manager/cert-manager-cainjector-7d5b44bb96-rsfv6” podUID=“444946a1-c568-45ee-9e58-b525c007da27”
Jan 15 21:43:00 phy-10-19-196-71 kubelet[163303]: E0115 21:43:00.561391 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:01 phy-10-19-196-71 kubelet[163303]: I0115 21:43:01.490001 163303 scope.go:117] “RemoveContainer” containerID=“bc10e6ceeb223bc90d33cd5d412b327e39bf5f68762dad26d913bf9d3db27d5a”
Jan 15 21:43:01 phy-10-19-196-71 kubelet[163303]: I0115 21:43:01.490193 163303 scope.go:117] “RemoveContainer” containerID=“76164b40db20140b3bb12e87dad559103aeee28290a66b4e48713cf06638db93”
Jan 15 21:43:01 phy-10-19-196-71 kubelet[163303]: E0115 21:43:01.490306 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=capsule-controller-manager-584d6c96d9-b8l7k_capsule-system(833e1bde-25dc-4611-82bc-bbe2e0eb6bc4)”” pod=“capsule-system/capsule-controller-manager-584d6c96d9-b8l7k” podUID=“833e1bde-25dc-4611-82bc-bbe2e0eb6bc4”
Jan 15 21:43:01 phy-10-19-196-71 kubelet[163303]: E0115 21:43:01.490501 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-controller” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-controller pod=cert-manager-5f7b5dbfbc-lw2pz_cert-manager(f9835c2c-7bf9-44a8-b36f-e52e8e9f15c2)”” pod=“cert-manager/cert-manager-5f7b5dbfbc-lw2pz” podUID=“f9835c2c-7bf9-44a8-b36f-e52e8e9f15c2”
Jan 15 21:43:01 phy-10-19-196-71 kubelet[163303]: E0115 21:43:01.562395 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:02 phy-10-19-196-71 kubelet[163303]: I0115 21:43:02.496663 163303 scope.go:117] “RemoveContainer” containerID=“356c0a67fa91f0be7f89e9b561f07fbbc1544673ad0dd2bff31dd9a498d40b1c”
Jan 15 21:43:02 phy-10-19-196-71 kubelet[163303]: E0115 21:43:02.496958 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=kube-green-controller-manager-7b7cf6b8cc-p8r8l_kube-green(b5f89fb1-ccac-451b-a6dd-770fc4b510e1)”” pod=“kube-green/kube-green-controller-manager-7b7cf6b8cc-p8r8l” podUID=“b5f89fb1-ccac-451b-a6dd-770fc4b510e1”
Jan 15 21:43:02 phy-10-19-196-71 kubelet[163303]: E0115 21:43:02.562873 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:03 phy-10-19-196-71 kubelet[163303]: E0115 21:43:03.563598 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:04 phy-10-19-196-71 kubelet[163303]: I0115 21:43:04.490203 163303 scope.go:117] “RemoveContainer” containerID=“a39c6310624f83a0028cb6f59a00c358c1a93c92409573dc495ffa0aa076d904”
Jan 15 21:43:04 phy-10-19-196-71 kubelet[163303]: E0115 21:43:04.490491 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “manager” with CrashLoopBackOff: “back-off 5m0s restarting failed container=manager pod=eraser-controller-manager-5b8f646476-67spl_eraser-system(7b532785-af5f-40af-90ff-021a5520e8f9)”” pod=“eraser-system/eraser-controller-manager-5b8f646476-67spl” podUID=“7b532785-af5f-40af-90ff-021a5520e8f9”
Jan 15 21:43:04 phy-10-19-196-71 kubelet[163303]: E0115 21:43:04.564250 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:05 phy-10-19-196-71 kubelet[163303]: I0115 21:43:05.531953 163303 scope.go:117] “RemoveContainer” containerID=“f2fe6c4aa6033c18e850ca81112c7a8c50eaf41eaac1170d607d9fae8fd51e23”
Jan 15 21:43:05 phy-10-19-196-71 kubelet[163303]: E0115 21:43:05.532270 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “cert-manager-webhook” with CrashLoopBackOff: “back-off 5m0s restarting failed container=cert-manager-webhook pod=cert-manager-webhook-69459b8974-bgdbx_cert-manager(36f533fd-6fe2-41b9-9cfe-eefa18459e63)”” pod=“cert-manager/cert-manager-webhook-69459b8974-bgdbx” podUID=“36f533fd-6fe2-41b9-9cfe-eefa18459e63”
Jan 15 21:43:05 phy-10-19-196-71 kubelet[163303]: E0115 21:43:05.565155 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:06 phy-10-19-196-71 kubelet[163303]: I0115 21:43:06.533860 163303 scope.go:117] “RemoveContainer” containerID=“be6bc1b003d6773279666c38366576e6efdbdf3031a4b7bffbe5889e437daa08”
Jan 15 21:43:06 phy-10-19-196-71 kubelet[163303]: E0115 21:43:06.534137 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“failed to “StartContainer” for “coredns” with CrashLoopBackOff: “back-off 1m20s restarting failed container=coredns pod=coredns-674b8bbfcf-lv6r7_kube-system(b597e5f9-9715-489e-b0a0-bc21b1a0e0de)”” pod=“kube-system/coredns-674b8bbfcf-lv6r7” podUID=“b597e5f9-9715-489e-b0a0-bc21b1a0e0de”
Jan 15 21:43:06 phy-10-19-196-71 kubelet[163303]: E0115 21:43:06.566175 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:07 phy-10-19-196-71 kubelet[163303]: E0115 21:43:07.566700 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:08 phy-10-19-196-71 kubelet[163303]: E0115 21:43:08.567138 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:09 phy-10-19-196-71 kubelet[163303]: E0115 21:43:09.568239 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:10 phy-10-19-196-71 kubelet[163303]: I0115 21:43:10.540085 163303 scope.go:117] “RemoveContainer” containerID=“7c83006d097765cae782775d8d43f4b3d15503765a49ffa81fb3e03306071397”
Jan 15 21:43:10 phy-10-19-196-71 kubelet[163303]: I0115 21:43:10.540125 163303 scope.go:117] “RemoveContainer” containerID=“b09ba39cef0090ed8a7513328d0232d9df1fd50aa71c0fb4b73e027d490d0c6a”
Jan 15 21:43:10 phy-10-19-196-71 kubelet[163303]: E0115 21:43:10.540480 163303 pod_workers.go:1301] “Error syncing pod, skipping” err=“[failed to “StartContainer” for “device-plugin” with CrashLoopBackOff: “back-off 5m0s restarting failed container=device-plugin pod=hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)”, failed to “StartContainer” for “vgpu-monitor” with CrashLoopBackOff: “back-off 5m0s restarting failed container=vgpu-monitor pod=hami-device-plugin-hc6vm_kube-system(0e107896-30bf-406e-8ef4-4ccab4008c56)”]” pod=“kube-system/hami-device-plugin-hc6vm” podUID=“0e107896-30bf-406e-8ef4-4ccab4008c56”
Jan 15 21:43:10 phy-10-19-196-71 kubelet[163303]: E0115 21:43:10.569145 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”
Jan 15 21:43:11 phy-10-19-196-71 kubelet[163303]: E0115 21:43:11.569474 163303 file_linux.go:61] “Unable to read config path” err=“path does not exist, ignoring” path=“/etc/kubernetes/manifests”