# Kubernetes install on EC2, communication failure(kubeadm init, containerd)

**URL:** <https://discuss.kubernetes.io/t/kubernetes-install-on-ec2-communication-failure-kubeadm-init-containerd/30564>\
**Category:** General Discussions\
**Tags:** network\
**Created:** [November 20, 2024, 8:41am UTC](https://discuss.kubernetes.io/t/kubernetes-install-on-ec2-communication-failure-kubeadm-init-containerd/30564 "2024-11-20T08:41:13Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jinhee](https://avatars.discourse-cdn.com/v4/letter/j/848f3c/32.png) [@jinhee](https://discuss.kubernetes.io/u/jinhee)\
**Post date:** [November 20, 2024, 8:41am UTC](https://discuss.kubernetes.io/t/kubernetes-install-on-ec2-communication-failure-kubeadm-init-containerd/30564/1 "2024-11-20T08:41:13Z")

</div>

### Cluster information:

Kubernetes version: v1.29.10  
Cloud being used: AWS EC2  
Installation method: kubeadm init  
Host OS: rhel 8.10  
CNI and version: Calico 3.29.0  
CRI and version: Containerd 1.7.23

Hi, I’m having a problem.  
Could you please help?  
Thanks in advance.  
(They said you can only add a maximum of 5 links, so I wrote it a little strangely.)

## **Desired result**

Build the next version of Kubernetes Cluster with two AWS EC2 units.

> kube\_version=1.29.10  
> coredns\_version=1.11.1  
> pause\_version=3.9  
> etcd\_version=3.5.15-0  
> containerd\_version=1.7.23  
> runc\_version=1.1.14  
> calio\_version=3.29.0

For version compatibility, refer to the following documents.

> **Containerd(for Kubernetes)**  
> h-----ttps://containerd.io/releases/#kubernetes-support  
> **Runc(for Containerd)**  
> h-----ttps://github.com/containerd/containerd/blob/v1.7.23/docs/RUNC.md  
> h-----ttps://github.com/containerd/containerd/blob/v1.7.23/script/setup/runc-version  
> **Calio(for Kubernetes)**  
> h-----ttps://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#kubernetes-requirements  
> **kubernets Each Manifest**  
> kubeadm config images list --kubernetes-version=v1.29.10  
> -----registry.k8s.io/kube-apiserver:v1.29.10  
> -----registry.k8s.io/kube-controller-manager:v1.29.10  
> -----registry.k8s.io/kube-scheduler:v1.29.10  
> -----registry.k8s.io/kube-proxy:v1.29.10  
> -----registry.k8s.io/coredns/coredns:v1.11.1  
> -----registry.k8s.io/pause:3.9  
> -----registry.k8s.io/etcd:3.5.15-0

## **What’s the problem?**

After completing cluster construction, creating and connecting to Ubuntu Pod and performing ‘apt update’ does not seem to allow communication.  
Sometimes it doesn’t work, sometimes it works and then it doesn’t work and repeats.  
For reference, when the cluster was built using kubespray, it worked normally. But I need to use kubeadm.

## The environment is as follows:

* * *

**VPC** : 10.0.0.0/16 (DNS hostnames:Enabled, DNS resolution: Enabled)  
**Public Subnet** : 10.0.0.0/20 (Routing 0.0.0.0-\> Internet Gateway)  
**NACL** : All Allow

**EC2**

- Master Server : 10.0.4.188 (in Public Subnet)(RHEL 8.10)(ami-03439f5ccc1eeb443)
- Worker Server : 10.0.9.252 (in Public Subnet)(RHEL 8.10)(ami-03439f5ccc1eeb443)

**Security Group**

- Inbound : All Traffic 0.0.0.0 Allow
- Outboud: All Traffic 0.0.0.0 Allow

**Return value when executing the following command in EC2**  
stat -fc %T /sys/fs/cgroup  
==\> tmpfs  
uname -m  
==\> x86\_64

### Mater Server

```
#!/bin/bash
ls /usr/share/zoneinfo/Asia | grep Seoul
sudo ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
date

yum update -y
dnf update -y

echo "====Building a Kubernetes environment===="
echo "=Forwarding IPv4 and letting iptables see bridged traffic="
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

echo "=================Swap off================="
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

echo "=============Firewall Disable============="
systemctl stop firewalld
systemctl disable firewalld

echo "=========SELinux Permissive Mode========="
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

echo "============Install Containerd============"
curl -LO https://github.com/containerd/containerd/releases/download/v1.7.23/containerd-1.7.23-linux-amd64.tar.gz
tar Cxzvf /usr/local containerd-1.7.23-linux-amd64.tar.gz
curl -LO https://raw.githubusercontent.com/containerd/containerd/v1.7.23/containerd.service
mkdir -p /usr/local/lib/systemd/system
mv containerd.service /usr/local/lib/systemd/system/containerd.service

systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd --no-pager

echo "===============Install Runc==============="
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.14/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc
runc --version

echo "========Containerd Configure Edit========"
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sed -i 's|sandbox_image = "registry.k8s.io/pause:3.8"|sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml
systemctl restart containerd

echo "=============Kubernetes repo============="
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

echo "====Install kubeadm, kubelet, kubectl===="
sudo yum install -y kubelet-1.29.10 kubeadm-1.29.10 kubectl-1.29.10 --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
kubeadm version
kubelet --version
kubectl version
systemctl status kubelet --no-pager

echo "========Setting crictl Endpoint========"
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
EOF

echo "===============kubeadm init==============="
kubeadm init --pod-network-cidr=192.168.0.0/16 >> kubeadm_ini_output.txt
systemctl restart kubelet
systemctl status kubelet --no-pager

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

echo "==============Install Calico=============="
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/custom-resources.yaml
sleep 20
kubectl get pods -n calico-system
kubectl get nodes -o wide

echo "============Install Calicoctl============"
curl -L https://github.com/projectcalico/calico/releases/download/v3.29.0/calicoctl-linux-amd64 -o calicoctl
chmod +x ./calicoctl
mv calicoctl /usr/local/bin

```

### Worker Server

```
#!/bin/bash
ls /usr/share/zoneinfo/Asia | grep Seoul
sudo ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
date

yum update -y
dnf update -y

echo "====Building a Kubernetes environment===="
echo "=Forwarding IPv4 and letting iptables see bridged traffic="
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

echo "=================Swap off================="
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

echo "=============Firewall Disable============="
systemctl stop firewalld
systemctl disable firewalld

echo "=========SELinux Permissive Mode========="
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

echo "============Install Containerd============"
curl -LO https://github.com/containerd/containerd/releases/download/v1.7.23/containerd-1.7.23-linux-amd64.tar.gz
tar Cxzvf /usr/local containerd-1.7.23-linux-amd64.tar.gz
curl -LO https://raw.githubusercontent.com/containerd/containerd/v1.7.23/containerd.service
mkdir -p /usr/local/lib/systemd/system
mv containerd.service /usr/local/lib/systemd/system/containerd.service

systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd --no-pager

echo "===============Install Runc==============="
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.14/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc
runc --version

echo "========Containerd Configure Edit========"
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sed -i 's|sandbox_image = "registry.k8s.io/pause:3.8"|sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml
systemctl restart containerd

echo "=============Kubernetes repo============="
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

echo "====Install kubeadm, kubelet, kubectl===="
sudo yum install -y kubelet-1.29.10 kubeadm-1.29.10 kubectl-1.29.10 --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
kubeadm version
kubelet --version
kubectl version
systemctl status kubelet --no-pager

echo "========Setting crictl Endpoint========"
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
EOF

kubeadm join {---Please Change here---}

```

### apt update test (Execute commands on Master Server)

```
vi sample.yaml

apiVersion: v1
kind: Pod
metadata:
name: ubuntu
spec:
containers:
- name: ubuntu
    image: ubuntu:latest
    command: ["sleep", "infinity"]
    ports:
    - containerPort: 80

kubectl apply -f sample.yaml
kubectl exec ubuntu -it -- bash
apt update

```

It comes out like this:

```bash
Ign:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Ign:2 http://archive.ubuntu.com/ubuntu noble InRelease  
Ign:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease  
Ign:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Ign:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease  
Ign:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Ign:2 http://archive.ubuntu.com/ubuntu noble InRelease  
Err:1 http://security.ubuntu.com/ubuntu noble-security InRelease  
Temporary failure resolving 'security.ubuntu.com'  
Ign:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease  
Ign:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease  
Ign:2 http://archive.ubuntu.com/ubuntu noble InRelease  
Ign:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease  
0% [Connecting to archive.ubuntu.com]  

```

However, if I do the following, communication will be normal.  
Modify the Master Node’s taint without joining the Worker Node.

```
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

```

Create an Ubuntu Pod on the Master Node and perform apt update.

It operates normally.

Why is there no problem when using only one Master Server, but a problem occurs when a Worker Server is added?  
To say it was a problem with AWS EC2 worked normally when built with kubespray.  
Thank you.

Next, I will create environments that can be used as a reference.

```bash
# kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-76f75df574-dxlh2 1/1 Running 0 56m
pod/coredns-76f75df574-xwpbh 1/1 Running 0 56m
pod/etcd-ip-10-0-4-188.ap-northeast-2.compute.internal 1/1 Running 0 56m
pod/kube-apiserver-ip-10-0-4-188.ap-northeast-2.compute.internal 1/1 Running 0 56m
pod/kube-controller-manager-ip-10-0-4-188.ap-northeast-2.compute.internal 1/1 Running 0 56m
pod/kube-proxy-ff4mj 1/1 Running 0 56m
pod/kube-proxy-tg64g 1/1 Running 0 14m
pod/kube-scheduler-ip-10-0-4-188.ap-northeast-2.compute.internal 1/1 Running 0 56m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 56m

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-proxy 2 2 2 2 2 kubernetes.io/os=linux 56m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 2/2 2 2 56m

NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-76f75df574 2 2 2 56m

```

```bash
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-4-188.ap-northeast-2.compute.internal Ready control-plane 69m v1.29.10 10.0.4.188 <none> Red Hat Enterprise Linux 8.10 (Ootpa) 4.18.0-553.27.1.el8_10.x86_64 containerd://1.7.23
ip-10-0-9-252.ap-northeast-2.compute.internal Ready <none> 26m v1.29.10 10.0.9.252 <none> Red Hat Enterprise Linux 8.10 (Ootpa) 4.18.0-553.27.1.el8_10.x86_64 containerd://1.7.23

```

```bash
# kubectl logs pod/coredns-76f75df574-dxlh2 -n kube-system
.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2

# kubectl logs pod/coredns-76f75df574-xwpbh -n kube-system
.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2

```

```bash
# cat /etc/kubernetes/manifests/* |grep cidr
    - --allocate-node-cidrs=true
    - --cluster-cidr=192.168.0.0/16
# cat /etc/kubernetes/manifests/* |grep range
    - --service-cluster-ip-range=10.96.0.0/12
    - --service-cluster-ip-range=10.96.0.0/12

```

Please… Help me..