Kubernetes communication between pods and external machines is sometimes blocked by UFW

General Discussions
1

Cluster information:

Kubernetes version: 1.21.3
Cloud being used: bare-metal
Installation method: apt && kube init
Host OS: Ubuntu 20.04
CNI and version: Calico 3.20.0
CRI and version: Docker 10.10.8

I installed kubernetes on bare metal with configuration as below:

  • k8s-master - virtual machine attached to private network - 10.35.2.36
  • k8s-node1 - virtual machine attached to private network - 10.35.2.37
  • db - virtual machine attached to private network - 10.35.2.34
  • networking - calico
  • container runtime: docker

db is outside kubernetes cluster (not attached as worker), k8s cluster is:

STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
Ready    control-plane,master   72m   v1.21.3   10.35.2.36    <none>        Ubuntu 20.04.3 LTS   5.4.0-84-generic   docker://20.10.8
Ready    <none>                 49m   v1.21.3   10.35.2.37    <none>        Ubuntu 20.04.3 LTS   5.4.0-84-generic   docker://20.10.8

UWF is set to allow all communication on private network:

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW IN    10.35.2.32/27/tcp

Kubernetes was initialized with:

--cluster-cidr=192.168.0.0/16

Calico with config:

 name: CALICO_IPV4POOL_CIDR
              value: "192.168.133.0/24"

My applications deployed to cluster generally workes fine - app can connect from docker (from k8s pod) to db host, pods sees each other.

The problem is: in k8s-node1 machine I see a lot of errors logged from UFW.
Mainly those are errors targeting coredns containers port 53 (DNS):

[UFW BLOCK] IN=calib678d9f2863 OUT=enp1s0 MAC=ee:ee:ee:ee:ee:ee:ce:09:88:b5:fa:70:08:00 SRC=192.168.193.68 DST=10.35.2.34 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=41444 DF PROTO=TCP SPT=35646 DPT=1433 WINDOW=64800 RES=0x00 SYN URGP=0 MARK=0x10000

But there are also entries telling me that communication to my db server is blocked.

[UFW BLOCK] IN=cali39580615a9a OUT=enp1s0 MAC=ee:ee:ee:ee:ee:ee:0a:0c:af:04:cc:e5:08:00 SRC=192.168.133.131 DST=10.35.2.34 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=40304 DF PROTO=TCP SPT=34274 DPT=1433 WINDOW=64800 RES=0x00 SYN URGP=0 MARK=0x10000

Those ufw block are not permanent - as I said my app generally works, only some of connections are dropped and I can’t find the reason.

1 Like
2

Hi, @Andrzej
Seeing the details provided, I understand that you are having those issues not always.
Those are intermittent ones if I understand you correctly.

I would agree that it could be not k8s related, but some general UFW or DNS-related issues.
By any chance, have you already tried to explicitly allow port 53 traffic in UFW & increase log level?

Indeed, seems quite strange that those issues happening only time to time.

thanks, regards, Yermek

3