CNI flannel in the wrong path

General Discussions
1

Hello!

I’m deploying my first lab k8s cluster with kubeadm in debian and I’ve choosed flannel as CNI because I found it as the simplest cni.

I’ve installed it with the commands

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl apply -f kube-flannel.yml

The problem was that the cluster wasn’t ready because the coredns pods couldn’t be created.

The command kubectl -n kube-system describe pods corednsgives me the information that it was looking for the cni binaries at the wrong path:

kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "71430bfda001180bb1dd10693e354858bf566201ff5b83018c6cfa24f401293e": plugin type="flannel" failed (add): failed to find plugin "flannel" in path [/usr/lib/cni]

I’ve solved it dirtily with a symbolic link ln -s /opt/cni/bin /usr/lib/cni, but I want to understand why happened this misunderstanding between the flannnel and the kubelet configuration and what is to be done to solve it in a more elegant way.

Cluster information:

Kubernetes version: v1.34.0
Cloud being used: bare-metal
Installation method: kubeadm init
Host OS: debian 13
CNI and version: flannel v1.9.0
CRI and version: containerd 1.7.24

2

Hello Jardi,

This is because the flannel manifest that you used to install flannel, directs the installation to /opt/bin/cni which is a standard path for cni plugins bin.

Since you are using containerd:1.7 on Debian-13 I am not sure why it’s looking for the plugin bin in /usr/lib/cni. Has this been configured somewhere, probably in your containerd config.toml?

There are two methods to fix this (or, in your terms, solve it in an elegant way ;D)

  1. (RECOMMENDED) Update your containerd config.toml to look for CNI plugins at the correct path.
[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  1. (NOT RECOMMENDED) If your container runtime containerd is unable to update the path or there are too many nodes to do this config, you can install flannel to a different path.
    1. If you open your flannel manifest, you will find the installation path
      initContainers:
      - args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        command:
        - cp
        image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.0-flannel1
        name: install-cni-plugin
        volumeMounts:
        - mountPath: /opt/cni/bin
          name: cni-plugin

And the volume mount:

      volumes:
      - hostPath:
          path: /opt/cni/bin
        name: cni-plugin

And the main container:

      containers:
      - args:
        - --ip-masq
        - --kube-subnet-mgr
        command:
        - /opt/bin/flanneld

You need to change the path from /opt/bin/' to the one that your container runtime is looking for.
This is complicated and risky method and you’ll prolly find your self lost in it.

Another option is to redo the CRI and CNI both, resetup the containerd, probably use containerd 2.

Background:

The most important thing to note is that prior to Kubernetes 1.24, the CNI plugins could also be managed by the kubelet using the cni-bin-dir and network-plugin command-line parameters. These command-line parameters were removed in Kubernetes 1.24, with management of the CNI no longer in scope for kubelet.
I am stating it here, because some AI may suggest you do something like the above (using cni-bin-dir and you’ll waste precious time figuring out what’s wrong.

Let me know if this helps!