Kubernetes install on EC2, communication failure(kubeadm init, containerd)

General Discussions
1

Cluster information:

Kubernetes version: v1.29.10
Cloud being used: AWS EC2
Installation method: kubeadm init
Host OS: rhel 8.10
CNI and version: Calico 3.29.0
CRI and version: Containerd 1.7.23

Hi, I’m having a problem.
Could you please help?
Thanks in advance.
(They said you can only add a maximum of 5 links, so I wrote it a little strangely.)

Desired result

Build the next version of Kubernetes Cluster with two AWS EC2 units.

kube_version=1.29.10
coredns_version=1.11.1
pause_version=3.9
etcd_version=3.5.15-0
containerd_version=1.7.23
runc_version=1.1.14
calio_version=3.29.0

For version compatibility, refer to the following documents.

Containerd(for Kubernetes)
h-----ttps://containerd.io/releases/#kubernetes-support
Runc(for Containerd)
h-----ttps://github.com/containerd/containerd/blob/v1.7.23/docs/RUNC.md
h-----ttps://github.com/containerd/containerd/blob/v1.7.23/script/setup/runc-version
Calio(for Kubernetes)
h-----ttps://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#kubernetes-requirements
kubernets Each Manifest
kubeadm config images list --kubernetes-version=v1.29.10
-----registry.k8s.io/kube-apiserver:v1.29.10
-----registry.k8s.io/kube-controller-manager:v1.29.10
-----registry.k8s.io/kube-scheduler:v1.29.10
-----registry.k8s.io/kube-proxy:v1.29.10
-----registry.k8s.io/coredns/coredns:v1.11.1
-----registry.k8s.io/pause:3.9
-----registry.k8s.io/etcd:3.5.15-0

What’s the problem?

After completing cluster construction, creating and connecting to Ubuntu Pod and performing ‘apt update’ does not seem to allow communication.
Sometimes it doesn’t work, sometimes it works and then it doesn’t work and repeats.
For reference, when the cluster was built using kubespray, it worked normally. But I need to use kubeadm.

The environment is as follows:

VPC: 10.0.0.0/16 (DNS hostnames:Enabled, DNS resolution: Enabled)
Public Subnet: 10.0.0.0/20 (Routing 0.0.0.0-> Internet Gateway)
NACL: All Allow

EC2

  • Master Server : 10.0.4.188 (in Public Subnet)(RHEL 8.10)(ami-03439f5ccc1eeb443)
  • Worker Server : 10.0.9.252 (in Public Subnet)(RHEL 8.10)(ami-03439f5ccc1eeb443)

Security Group

  • Inbound : All Traffic 0.0.0.0 Allow
  • Outboud: All Traffic 0.0.0.0 Allow

Return value when executing the following command in EC2
stat -fc %T /sys/fs/cgroup
==> tmpfs
uname -m
==> x86_64

Mater Server

#!/bin/bash
ls /usr/share/zoneinfo/Asia | grep Seoul
sudo ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
date

yum update -y
dnf update -y

echo "====Building a Kubernetes environment===="
echo "=Forwarding IPv4 and letting iptables see bridged traffic="
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system


echo "=================Swap off================="
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

echo "=============Firewall Disable============="
systemctl stop firewalld
systemctl disable firewalld

echo "=========SELinux Permissive Mode========="
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

echo "============Install Containerd============"
curl -LO https://github.com/containerd/containerd/releases/download/v1.7.23/containerd-1.7.23-linux-amd64.tar.gz
tar Cxzvf /usr/local containerd-1.7.23-linux-amd64.tar.gz
curl -LO https://raw.githubusercontent.com/containerd/containerd/v1.7.23/containerd.service
mkdir -p /usr/local/lib/systemd/system
mv containerd.service /usr/local/lib/systemd/system/containerd.service

systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd --no-pager

echo "===============Install Runc==============="
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.14/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc
runc --version

echo "========Containerd Configure Edit========"
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sed -i 's|sandbox_image = "registry.k8s.io/pause:3.8"|sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml
systemctl restart containerd

echo "=============Kubernetes repo============="
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

echo "====Install kubeadm, kubelet, kubectl===="
sudo yum install -y kubelet-1.29.10 kubeadm-1.29.10 kubectl-1.29.10 --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
kubeadm version
kubelet --version
kubectl version
systemctl status kubelet --no-pager

echo "========Setting crictl Endpoint========"
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
EOF

echo "===============kubeadm init==============="
kubeadm init --pod-network-cidr=192.168.0.0/16 >> kubeadm_ini_output.txt
systemctl restart kubelet
systemctl status kubelet --no-pager

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

echo "==============Install Calico=============="
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/custom-resources.yaml
sleep 20
kubectl get pods -n calico-system
kubectl get nodes -o wide

echo "============Install Calicoctl============"
curl -L https://github.com/projectcalico/calico/releases/download/v3.29.0/calicoctl-linux-amd64 -o calicoctl
chmod +x ./calicoctl
mv calicoctl /usr/local/bin

Worker Server

#!/bin/bash
ls /usr/share/zoneinfo/Asia | grep Seoul
sudo ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
date

yum update -y
dnf update -y

echo "====Building a Kubernetes environment===="
echo "=Forwarding IPv4 and letting iptables see bridged traffic="
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system


echo "=================Swap off================="
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

echo "=============Firewall Disable============="
systemctl stop firewalld
systemctl disable firewalld

echo "=========SELinux Permissive Mode========="
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

echo "============Install Containerd============"
curl -LO https://github.com/containerd/containerd/releases/download/v1.7.23/containerd-1.7.23-linux-amd64.tar.gz
tar Cxzvf /usr/local containerd-1.7.23-linux-amd64.tar.gz
curl -LO https://raw.githubusercontent.com/containerd/containerd/v1.7.23/containerd.service
mkdir -p /usr/local/lib/systemd/system
mv containerd.service /usr/local/lib/systemd/system/containerd.service

systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd --no-pager

echo "===============Install Runc==============="
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.14/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc
runc --version

echo "========Containerd Configure Edit========"
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sed -i 's|sandbox_image = "registry.k8s.io/pause:3.8"|sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml
systemctl restart containerd

echo "=============Kubernetes repo============="
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

echo "====Install kubeadm, kubelet, kubectl===="
sudo yum install -y kubelet-1.29.10 kubeadm-1.29.10 kubectl-1.29.10 --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
kubeadm version
kubelet --version
kubectl version
systemctl status kubelet --no-pager

echo "========Setting crictl Endpoint========"
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
EOF

kubeadm join {---Please Change here---}

apt update test (Execute commands on Master Server)

vi sample.yaml

apiVersion: v1
kind: Pod
metadata:
name: ubuntu
spec:
containers:
- name: ubuntu
    image: ubuntu:latest
    command: ["sleep", "infinity"]
    ports:
    - containerPort: 80


kubectl apply -f sample.yaml
kubectl exec ubuntu -it -- bash
apt update

It comes out like this:

Ign:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Ign:2 http://archive.ubuntu.com/ubuntu noble InRelease  
Ign:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease  
Ign:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Ign:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease  
Ign:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Ign:2 http://archive.ubuntu.com/ubuntu noble InRelease  
Err:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Temporary failure resolving 'security.ubuntu.com'  
Ign:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease  
Ign:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease  
Ign:2 http://archive.ubuntu.com/ubuntu noble InRelease  
Ign:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease  
0% [Connecting to archive.ubuntu.com]

However, if I do the following, communication will be normal.
Modify the Master Node’s taint without joining the Worker Node.

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Create an Ubuntu Pod on the Master Node and perform apt update.

It operates normally.

Why is there no problem when using only one Master Server, but a problem occurs when a Worker Server is added?
To say it was a problem with AWS EC2 worked normally when built with kubespray.
Thank you.

Next, I will create environments that can be used as a reference.

# kubectl get all -n kube-system
NAME                                                                        READY   STATUS    RESTARTS   AGE
pod/coredns-76f75df574-dxlh2                                                1/1     Running   0          56m
pod/coredns-76f75df574-xwpbh                                                1/1     Running   0          56m
pod/etcd-ip-10-0-4-188.ap-northeast-2.compute.internal                      1/1     Running   0          56m
pod/kube-apiserver-ip-10-0-4-188.ap-northeast-2.compute.internal            1/1     Running   0          56m
pod/kube-controller-manager-ip-10-0-4-188.ap-northeast-2.compute.internal   1/1     Running   0          56m
pod/kube-proxy-ff4mj                                                        1/1     Running   0          56m
pod/kube-proxy-tg64g                                                        1/1     Running   0          14m
pod/kube-scheduler-ip-10-0-4-188.ap-northeast-2.compute.internal            1/1     Running   0          56m

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   56m

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/kube-proxy   2         2         2       2            2           kubernetes.io/os=linux   56m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   2/2     2            2           56m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-76f75df574   2         2         2       56m

kubectl get nodes -o wide
NAME                                            STATUS   ROLES           AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                KERNEL-VERSION                  CONTAINER-RUNTIME
ip-10-0-4-188.ap-northeast-2.compute.internal   Ready    control-plane   69m   v1.29.10   10.0.4.188    <none>        Red Hat Enterprise Linux 8.10 (Ootpa)   4.18.0-553.27.1.el8_10.x86_64   containerd://1.7.23
ip-10-0-9-252.ap-northeast-2.compute.internal   Ready    <none>          26m   v1.29.10   10.0.9.252    <none>        Red Hat Enterprise Linux 8.10 (Ootpa)   4.18.0-553.27.1.el8_10.x86_64   containerd://1.7.23

# kubectl logs pod/coredns-76f75df574-dxlh2 -n kube-system
.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2

# kubectl logs pod/coredns-76f75df574-xwpbh  -n kube-system
.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2

# cat /etc/kubernetes/manifests/* |grep cidr
    - --allocate-node-cidrs=true
    - --cluster-cidr=192.168.0.0/16
# cat /etc/kubernetes/manifests/* |grep range
    - --service-cluster-ip-range=10.96.0.0/12
    - --service-cluster-ip-range=10.96.0.0/12

Please… Help me…