All probes (liveness, readiness) fail on Debian 11 / CRI-O 1.25.1 / K8S 1.25.4

General Discussions
1

I’m probably missing something important but could’t find anything in either k8s not cri-o docs:

After a fresh k8s install all probes are failing in my setup. Everything else is running fine, but not the probes.

What I did:

Installed vanilla K8S 1.25.4 on vanilla Debian 11 with vanilla cri-o 1.25.1 as container runtime. (also tried 1.24, same result)

Then applied test pod from here

https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/pods/probe/exec-liveness.yaml

And then I get the following from the logs:

1202 13:06:06.257580 17325 remote_runtime.go:734] “ExecSync cmd from runtime service failed” err=“rpc error: code = Unknown desc = command error: EOF, stdout: , stderr: , exit code -1” containerID=“ec9eb6391c79e079c4ee6ecdede3f8916a0ef3e6d74aefb069048171db41b455” cmd=[cat /tmp/healthy]

It seems it doesn’t matter how I redefine the probes, they always fail.

Client Version: version.Info{Major:“1”, Minor:“25”, GitVersion:“v1.25.4”, GitCommit:“872a965c6c6526caa949f0c6ac028ef7aff3fb78”, GitTreeState:“archive”, BuildDate:“2022-11-10T22:18:49Z”, GoVersion:“go1.19.3”, Compiler:“gc”, Platform:“linux/amd64”}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:“1”, Minor:“25”, GitVersion:“v1.25.4”, GitCommit:“872a965c6c6526caa949f0c6ac028ef7aff3fb78”, GitTreeState:“clean”, BuildDate:“2022-11-09T13:29:58Z”, GoVersion:“go1.19.3”, Compiler:“gc”, Platform:“linux/amd64”}

Os is Debian 11, kernel 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux

I installed k8s as follows

kubeadm init --config seeBelowFileContents

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
criSocket: “unix:///var/run/crio/crio.sock”

Pods are starting, DNS is working, even complex deployments are running if I just disable the probes, but probes autofail no matter what. I can exec into the pod and (kubectl) execute what the probes did, working as well.

What am I missing?

2 

kubelet[948680]: I1205 08:17:08.507645  948680 prober.go:163] "Exec-Probe runProbe" pod="default/liveness-exec" containerName="liveness" execCommand=[cat /tmp/healthy]
systemd[1]: run-runc-726d4c2010b67629535c1e2560664592546fa50f97ab0502e8592e7e4628dd31-runc.5SNNUr.mount: Succeeded.
systemd[928226]: run-runc-726d4c2010b67629535c1e2560664592546fa50f97ab0502e8592e7e4628dd31-runc.5SNNUr.mount: Succeeded.
kubelet[948680]: E1205 08:17:08.557373  948680 remote_runtime.go:711] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: EOF, stdout: , stderr: , exit code -1" containerID="726d4c2010b67629535c1e2560664592546fa50f97ab0502e8592e7e4628dd31" cmd=[cat /tmp/healthy]
kubelet[948680]: I1205 08:17:08.557404  948680 exec.go:62] Exec probe response: ""
kubelet[948680]: E1205 08:17:08.557424  948680 prober.go:118] "Probe errored" err="rpc error: code = Unknown desc = command error: EOF, stdout: , stderr: , exit code -1" probeType="Liveness" pod="default/liveness-exec" podUID=d5e694d2-1dc0-433c-b652-7b2a8c59bbbd containerName="liveness"
kubelet[948680]: I1205 08:17:08.557587  948680 event.go:294] "Event occurred" object="default/liveness-exec" fieldPath="spec.containers{liveness}" kind="Pod" apiVersion="v1" type="Warning" reason="Unhealthy" message="Liveness probe errored: rpc error: code = Unknown desc = command error: EOF, stdout: , stderr: , exit code -1"
3

It probably has to do with defective cni config.
All the snippets I can find regarding flannel and cri-o are a bit confusing to say the least.

Does anyone have a working cni config for bare metal + cri-o + flannel?

4

If anyone else lands here: it’s related to a bug in conmon that hits on Debian Busters version: