Cluster information:
Kubernetes version: 1.25.4
Cloud being used: Azure VM with Ubuntu 22.04
I’m having trouble with networking problems with an Ubuntu 22.04. I’m using kubernetes. I first tried weave, then Flannel.
In both cases the syslog has messages every few minutes with errors.
When using Weave, there’s a lot of errors for weave in the syslog. When using Flannel the cni0 is degraded and it has a lot of errors in the syslog. I’m not sure where to find help understanding how it should look when healthy.
Here are some lines from the syslog, systemctl status systemd-networkd.service and the networkctl.
Aug 13 11:50:00 ubuntu networkd-dispatcher[739]: WARNING:Unknown index 1086 seen, reloading interface list
Aug 13 11:50:00 ubuntu systemd-networkd[20424]: veth89866373: Link UP
Aug 13 11:50:00 ubuntu systemd-udevd[3451192]: Using default interface naming scheme 'v249'.
Aug 13 11:50:00 ubuntu kernel: [155362.174770] cni0: port 44(veth89866373) entered blocking state
Aug 13 11:50:00 ubuntu kernel: [155362.174774] cni0: port 44(veth89866373) entered disabled state
Aug 13 11:50:00 ubuntu kernel: [155362.174862] device veth89866373 entered promiscuous mode
Aug 13 11:50:00 ubuntu systemd-networkd[20424]: veth89866373: Gained carrier
Aug 13 11:50:00 ubuntu containerd[15085]: map[string]interface {}{"cniVersion":"0.3.1", "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[string]interface {}{map[string]interface {}{"subnet":"10.32.0.0/24"}}}, "routes":[]types.Route{types.Route{Dst:net.IPNet{IP:net.IP{0xa, 0x20, 0x0, 0x0}, Mask:net.IPMask{0xff, 0xff, 0xf0, 0x0}}, GW:net.IP(nil)}}, "type":"host-local"}, "isDefaultGateway":true, "isGateway":true, "mtu":(*uint)(0xc0000ae8e8), "name":"cbr0", "type":"bridge"}
Aug 13 11:50:00 ubuntu containerd[15085]: delegateAdd: netconf sent to delegate plugin:
Aug 13 11:50:00 ubuntu kernel: [155362.178259] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 13 11:50:00 ubuntu kernel: [155362.178296] IPv6: ADDRCONF(NETDEV_CHANGE): veth89866373: link becomes ready
Aug 13 11:50:00 ubuntu kernel: [155362.178360] cni0: port 44(veth89866373) entered blocking state
Aug 13 11:50:00 ubuntu kernel: [155362.178364] cni0: port 44(veth89866373) entered forwarding state
Aug 13 11:50:00 ubuntu systemd-udevd[3451222]: Using default interface naming scheme 'v249'.
Aug 13 11:50:00 ubuntu systemd-networkd[20424]: veth12d40373: Link UP
Aug 13 11:50:00 ubuntu kernel: [155362.182585] cni0: port 46(veth12d40373) entered blocking state
Aug 13 11:50:00 ubuntu kernel: [155362.182589] cni0: port 46(veth12d40373) entered disabled state
Aug 13 11:50:00 ubuntu kernel: [155362.182662] device veth12d40373 entered promiscuous mode
Aug 13 11:50:00 ubuntu kernel: [155362.182709] cni0: port 46(veth12d40373) entered blocking state
Aug 13 11:50:00 ubuntu kernel: [155362.182711] cni0: port 46(veth12d40373) entered forwarding state
Aug 13 11:50:00 ubuntu networkd-dispatcher[739]: WARNING:Unknown index 1087 seen, reloading interface list
Aug 13 11:50:00 ubuntu systemd-networkd[20424]: veth12d40373: Gained carrier
Aug 13 11:50:00 ubuntu kernel: [155362.199672] IPv6: ADDRCONF(NETDEV_CHANGE): veth12d40373: link becomes ready
Aug 13 11:50:00 ubuntu containerd[15085]: {"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.32.0.0/24"}]],"routes":[{"dst":"10.32.0.0/20"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}
Aug 13 11:50:00 ubuntu containerd[15085]: map[string]interface {}{"cniVersion":"0.3.1", "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[string]interface {}{map[string]interface {}{"subnet":"10.32.0.0/24"}}}, "routes":[]types.Route{types.Route{Dst:net.IPNet{IP:net.IP{0xa, 0x20, 0x0, 0x0}, Mask:net.IPMask{0xff, 0xff, 0xf0, 0x0}}, GW:net.IP(nil)}}, "type":"host-local"}, "isDefaultGateway":true, "isGateway":true, "mtu":(*uint)(0xc000020948), "name":"cbr0", "type":"bridge"}
Aug 13 11:50:00 ubuntu containerd[15085]: delegateAdd: netconf sent to delegate plugin:
Aug 13 11:50:00 ubuntu containerd[15085]: {"cniVersion":"0.3.1","hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.32.0.0/24"}]],"routes":[{"dst":"10.32.0.0/20"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}time="2023-08-13T11:50:00.536356484Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.536414239Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.536427229Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.536558344Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/351e08f685569e958557e91796d77cb0008db907275901d457e07aa703679075 pid=3451276 runtime=io.containerd.runc.v2
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.543607068Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.543653262Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.543665841Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.543797989Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/2444fa9c5af731f0bfa1207cca2dc508001e3e6b1cc8c50745e4bd51a2f30afe pid=3451304 runtime=io.containerd.runc.v2
Aug 13 11:50:00 ubuntu systemd[1]: Started libcontainer container 351e08f685569e958557e91796d77cb0008db907275901d457e07aa703679075.
Aug 13 11:50:00 ubuntu systemd[1]: Started libcontainer container 2444fa9c5af731f0bfa1207cca2dc508001e3e6b1cc8c50745e4bd51a2f30afe.
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.604097075Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to create inotify fd"
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.622061606Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to create inotify fd"
Aug 13 11:50:00 ubuntu systemd[1]: Started libcontainer container 6acdb17109ab72f4d27b05520cbb9398c77b255ebbb5073a17940b4a504a1ab9.
Aug 13 11:50:00 ubuntu systemd[1]: Started libcontainer container 07dd7f1499c69873febfe26f4270c1c7d39e1e8656b96dabf04749b22b7efa1f.
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.726864451Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to create inotify fd"
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.727530961Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to create inotify fd"
Aug 13 11:50:00 ubuntu systemd[1]: cri-containerd-07dd7f1499c69873febfe26f4270c1c7d39e1e8656b96dabf04749b22b7efa1f.scope: Deactivated successfully.
Aug 13 11:50:00 ubuntu systemd[1]: cri-containerd-6acdb17109ab72f4d27b05520cbb9398c77b255ebbb5073a17940b4a504a1ab9.scope: Deactivated successfully.
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.858204170Z" level=warning msg="cleaning up after shim disconnected" id=6acdb17109ab72f4d27b05520cbb9398c77b255ebbb5073a17940b4a504a1ab9 namespace=k8s.io
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.858233130Z" level=warning msg="cleaning up after shim disconnected" id=07dd7f1499c69873febfe26f4270c1c7d39e1e8656b96dabf04749b22b7efa1f namespace=k8s.io
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.865106569Z" level=warning msg="cleanup warnings time=\"2023-08-13T11:50:00Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=3451456 runtime=io.containerd.runc.v2\n"
Aug 13 11:50:00 ubuntu containerd[15085]: time="2023-08-13T11:50:00.865219910Z" level=warning msg="cleanup warnings time=\"2023-08-13T11:50:00Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=3451455 runtime=io.containerd.runc.v2\n"
Aug 13 11:50:02 ubuntu systemd-networkd[20424]: veth12d40373: Gained IPv6LL
Aug 13 11:50:02 ubuntu systemd-networkd[20424]: veth89866373: Gained IPv6LL
Aug 13 11:50:02 ubuntu systemd[1]: run-containerd-io.containerd.grpc.v1.cri-sandboxes-2444fa9c5af731f0bfa1207cca2dc508001e3e6b1cc8c50745e4bd51a2f30afe-shm.mount: Deactivated successfully.
Aug 13 11:50:02 ubuntu systemd[1]: run-containerd-io.containerd.grpc.v1.cri-sandboxes-351e08f685569e958557e91796d77cb0008db907275901d457e07aa703679075-shm.mount: Deactivated successfully.
Aug 13 11:50:02 ubuntu systemd[1]: cri-containerd-351e08f685569e958557e91796d77cb0008db907275901d457e07aa703679075.scope: Deactivated successfully.
Aug 13 11:50:02 ubuntu systemd[1]: cri-containerd-2444fa9c5af731f0bfa1207cca2dc508001e3e6b1cc8c50745e4bd51a2f30afe.scope: Deactivated successfully.
Aug 13 11:50:02 ubuntu containerd[15085]: time="2023-08-13T11:50:02.606953798Z" level=error msg="failed to handle sandbox TaskExit event &TaskExit{ContainerID:2444fa9c5af731f0bfa1207cca2dc508001e3e6b1cc8c50745e4bd51a2f30afe,ID:2444fa9c5af731f0bfa1207cca2dc508001e3e6b1cc8c50745e4bd51a2f30afe,Pid:3451334,ExitStatus:137,ExitedAt:2023-08-13 11:50:02.581316764 +0000 UTC,XXX_unrecognized:[],}" error="failed to stop sandbox: unknown error after kill: runc did not terminate successfully: exit status 1: lstat /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47116815_e6a7_44df_9085_5c2e1e9a5be6.slice/cri-containerd-2444fa9c5af731f0bfa1207cca2dc508001e3e6b1cc8c50745e4bd51a2f30afe.scope: no such file or directory\n: unknown"
Aug 13 11:50:02 ubuntu systemd[1]: run-containerd-io.containerd.runtime.v2.task-k8s.io-351e08f685569e958557e91796d77cb0008db907275901d457e07aa703679075-rootfs.mount: Deactivated successfully.
Aug 13 11:50:02 ubuntu containerd[15085]: time="2023-08-13T11:50:02.657499846Z" level=warning msg="cleaning up after shim disconnected" id=351e08f685569e958557e91796d77cb0008db907275901d457e07aa703679075 namespace=k8s.io
Aug 13 11:50:02 ubuntu containerd[15085]: time="2023-08-13T11:50:02.664489904Z" level=warning msg="cleanup warnings time=\"2023-08-13T11:50:02Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=3451544 runtime=io.containerd.runc.v2\n"
Aug 13 11:50:02 ubuntu systemd-networkd[20424]: veth89866373: Link DOWN
Aug 13 11:50:02 ubuntu systemd-networkd[20424]: veth89866373: Lost carrier
Aug 13 11:50:02 ubuntu kernel: [155364.341038] cni0: port 44(veth89866373) entered disabled state
Aug 13 11:50:02 ubuntu kernel: [155364.342241] device veth89866373 left promiscuous mode
Aug 13 11:50:02 ubuntu kernel: [155364.342245] cni0: port 44(veth89866373) entered disabled state
Aug 13 11:50:02 ubuntu networkctl[3451580]: Interface "veth89866373" not found.
Aug 13 11:50:02 ubuntu systemd[1]: networkd-dispatcher.service: Got notification message from PID 3451580, but reception only permitted for main PID 739
Aug 13 11:50:02 ubuntu networkd-dispatcher[739]: ERROR:Failed to get interface "veth89866373" status: Command '['/usr/bin/networkctl', 'status', '--no-pager', '--no-legend', '--', 'veth89866373']' returned non-zero exit status 1.
Aug 13 11:50:02 ubuntu systemd[1]: run-netns-cni\x2d59d2cbde\x2d6c56\x2d492b\x2d641e\x2d4d5afdc651c8.mount: Deactivated successfully.
Aug 13 11:50:04 ubuntu systemd-networkd[20424]: veth12d40373: Link DOWN
Aug 13 11:50:04 ubuntu systemd-networkd[20424]: veth12d40373: Lost carrier
Aug 13 11:50:04 ubuntu kernel: [155365.901702] cni0: port 46(veth12d40373) entered disabled state
Aug 13 11:50:04 ubuntu kernel: [155365.902603] device veth12d40373 left promiscuous mode
Aug 13 11:50:04 ubuntu kernel: [155365.902607] cni0: port 46(veth12d40373) entered disabled state
Aug 13 11:50:04 ubuntu networkctl[3451639]: Interface "veth12d40373" not found.
Aug 13 11:50:04 ubuntu systemd[1]: networkd-dispatcher.service: Got notification message from PID 3451639, but reception only permitted for main PID 739
Aug 13 11:50:04 ubuntu networkd-dispatcher[739]: ERROR:Failed to get interface "veth12d40373" status: Command '['/usr/bin/networkctl', 'status', '--no-pager', '--no-legend', '--', 'veth12d40373']' returned non-zero exit status 1.
Aug 13 11:50:04 ubuntu systemd[1]: run-netns-cni\x2dd8250508\x2df97d\x2dd9e7\x2d9575\x2dad8c4ae21944.mount: Deactivated successfully.
…
Aug 13 11:55:00 ubuntu kernel: [155662.201619] cni0: port 46(vethd72c1c0d) entered blocking state
Aug 13 11:55:00 ubuntu kernel: [155662.201624] cni0: port 46(vethd72c1c0d) entered disabled state
Aug 13 11:55:00 ubuntu kernel: [155662.201717] device vethd72c1c0d entered promiscuous mode
Aug 13 11:55:00 ubuntu kernel: [155662.201772] cni0: port 46(vethd72c1c0d) entered blocking state
Aug 13 11:55:00 ubuntu kernel: [155662.201774] cni0: port 46(vethd72c1c0d) entered forwarding state
Aug 13 11:55:00 ubuntu systemd-networkd[20424]: vethd72c1c0d: Gained carrier
rgadmin@ubuntu:/var/log$ networkctl
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
3 enP43331s1 ether enslaved unmanaged
4 kube-ipvs0 ether off unmanaged
5 flannel.1 vxlan routable unmanaged
6 cni0 bridge degraded-carrier unmanaged
7 veth1452caf8 ether degraded unmanaged
8 veth39c4b0ec ether degraded unmanaged
12 vethf70bac30 ether degraded unmanaged
15 veth6d0ddae4 ether degraded unmanaged
17 vethaa4b9c57 ether degraded unmanaged
19 veth722d044e ether degraded unmanaged
systemctl status systemd-networkd.service
- systemd-networkd.service - Network Configuration
Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-08-11 17:29:33 UTC; 35min ago
TriggeredBy: - systemd-networkd.socket
Docs: man:systemd-networkd.service(8)
Main PID: 20424 (systemd-network)
Status: "Processing requests..."
Tasks: 1 (limit: 77151)
Memory: 2.0M
CPU: 249ms
CGroup: /system.slice/systemd-networkd.service
└─20424 /lib/systemd/systemd-networkd
Aug 11 17:55:02 ubuntu systemd-networkd[20424]: vethd8f8d04b: Lost carrier
Aug 11 18:00:00 ubuntu systemd-networkd[20424]: vethd34da19a: Link UP
Aug 11 18:00:00 ubuntu systemd-networkd[20424]: veth6eb806f2: Link UP
Aug 11 18:00:00 ubuntu systemd-networkd[20424]: vethd34da19a: Gained carrier
Aug 11 18:00:00 ubuntu systemd-networkd[20424]: veth6eb806f2: Gained carrier
Aug 11 18:00:02 ubuntu systemd-networkd[20424]: veth6eb806f2: Link DOWN
Aug 11 18:00:02 ubuntu systemd-networkd[20424]: veth6eb806f2: Lost carrier
Aug 11 18:00:02 ubuntu systemd-networkd[20424]: vethd34da19a: Gained IPv6LL
Aug 11 18:00:02 ubuntu systemd-networkd[20424]: vethd34da19a: Link DOWN
Aug 11 18:00:02 ubuntu systemd-networkd[20424]: vethd34da19a: Lost carrier