I’m new to k8s and I’ve been trying to learn it. I faced a problem with trying to setup aws-vpc-k8s-cni on my fresh k8s cluster with coredns
. Here’s the problem in detail.
Cluster & Network Architecture
The cluster setup consists of 1 master
(ec2
instance) node only for now. The network is built of 1 VPC
with CIDR range 10.0.0.0/16
and 4 Subnets
with CIDR ranges (public_1, 10.0.0.0/20)
, (public_2, 10.0.16.0/20)
, (private_1, 10.0.32.0/20)
, (private_2, 10.0.48.0/20)
. The public subnets route outbound traffic via an internet gateway
to the internet and private subnets route outbound traffic via a NAT
proxy (ec2
instance) to the internet. The private subnets are also connected to each other via a VPC Endpoint
. I access machines on my private subnet via another ec2
instance that acts as a bastion. Hence, There are a total of 3 machines.
- K8S Master Instance.
- Bastion Instance (SSH only)
- NAT Proxy Instance (Private Subnet → Internet [Outbound Only])
AWS Setup Master Instance
I’ve attached a security group that forwards the ports listed in kubernetes.io/docs/reference/networking/ports-and-protocols. I’ve also created an IAM role (ec2 instance profile) and attached the policies listed cloud-provider-aws.sigs.k8s.io/prerequisites.
On Master Instance
- I setup the hostname of the instance to the private ipv4 dns name with
curl http://169.254.169.254/latest/meta-data/hostname
. - I setup the iptables with
iptables -P FORWARD ACCEPT
. - I installed
containerd
,kubelet
,kubectl
,kubeadm
. - I loaded the modules
br_netfilter
&overlay
. - I setup kernal flags namely
net.bridge.bridge-nf-call-iptables = 1
,net.bridge.bridge-nf-call-ip6tables = 1
,net.ipv4.ip_forward = 1
. - I added
KUBELET_EXTRA_ARGS=--node-ip=<ipv4_address> --cloud-provider=external
in/etc/systemd/system/kubelet.service.d/20-aws.conf
. - I added
--cloud-provider=external
to/etc/kubernetes/manifests/kubernetes-controller-manager.yaml
&/etc/kubernetes/manifests/kube-apiserver.yaml
. - I initialized a fresh k8s cluster with
kubeadm init --token--ttl 0
and it initialised successfully. - I deployed
aws-vpc-cni-k8s
viakubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.16.4/config/master/aws-k8s-cni.yaml
. - I deployed
aws-cloud-controller-manager
viakubectl apply -k 'github.com/kubernetes/cloud-provider-aws/examples/existing-cluster/base/?ref=master'
.
Problem
Both aws-cloud-controller-manager
and aws-vpc-cni
deployed successfully and showed no errors. While the node shows up as “Ready”, I’m unable to get coredns
working. It’s unable to start. Neither journalctl -u kubelet.service
nor journalctl -u containerd.service
reveal anything about the source of problem. I believe my ports are forwarded correctly and I’m unsure what’s causing corends
to not run.
My assumption is that coredns
pod is directing dns lookups from it’s /etc/resolv.conf
to my master node /etc/resolv.conf
because of the config map forward . /etc/resolv.conf
. The master node is is unaware of the cluster & therefore no lookup is successfull. I could be wrong here. I’m not sure to resolve this.
Here’s relevant logs,
kubectl logs coredns-76f75df574-465x5 -n kube-system
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:39074->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:47632->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:34254->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:39274->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:49990->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:54792->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:46469->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:33147->10.0.0.2:53: i/o timeout
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/kubernetes: Trace[597383834]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231 (16-Mar-2024 09:09:06.572) (total time: 30002ms):
Trace[597383834]: ---"Objects listed" error:Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout 30001ms (09:09:36.574)
Trace[597383834]: [30.002080972s] [30.002080972s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/kubernetes: Trace[1358557007]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231 (16-Mar-2024 09:09:06.573) (total time: 30001ms):
Trace[1358557007]: ---"Objects listed" error:Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout 30001ms (09:09:36.574)
Trace[1358557007]: [30.001364092s] [30.001364092s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/kubernetes: Trace[784641658]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231 (16-Mar-2024 09:09:06.574) (total time: 30001ms):
Trace[784641658]: ---"Objects listed" error:Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout 30000ms (09:09:36.575)
Trace[784641658]: [30.001209965s] [30.001209965s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:33770->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 908260726288404915.5102928999241227370. HINFO: read udp 10.0.43.148:54282->10.0.0.2:53: i/o timeout
kubectl describe pod coredns-76f75df574-465x5 -n kube-system
admin@ip-10-0-32-163:~$ kubectl describe pod coredns-76f75df574-465x5 -n kube-system
Name: coredns-76f75df574-465x5
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: coredns
Node: ip-10-0-32-163.ap-south-1.compute.internal/10.0.32.163
Start Time: Sat, 16 Mar 2024 09:09:05 +0000
Labels: k8s-app=kube-dns
pod-template-hash=76f75df574
Annotations: <none>
Status: Running
IP: 10.0.43.148
IPs:
IP: 10.0.43.148
Controlled By: ReplicaSet/coredns-76f75df574
Containers:
coredns:
Container ID: containerd://cb07b34ca26b0b5d51dfcf658ba03fed0979346da243abd98ab8190a6df46800
Image: registry.k8s.io/coredns/coredns:v1.11.1
Image ID: registry.k8s.io/coredns/coredns@sha256:1eeb4c7316bacb1d4c8ead65571cd92dd21e27359f0d4917f1a5822a73b75db1
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Running
Started: Sat, 16 Mar 2024 09:31:42 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 16 Mar 2024 09:24:46 +0000
Finished: Sat, 16 Mar 2024 09:26:40 +0000
Ready: False
Restart Count: 8
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5l9f6 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
kube-api-access-5l9f6:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/control-plane:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 27m default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Normal Scheduled 23m default-scheduler Successfully assigned kube-system/coredns-76f75df574-465x5 to ip-10-0-32-163.ap-south-1.compute.internal
Normal Created 23m kubelet Created container coredns
Normal Started 23m kubelet Started container coredns
Warning Unhealthy 23m kubelet Readiness probe failed: Get "http://10.0.43.148:8181/ready": dial tcp 10.0.43.148:8181: i/o timeout (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 21m (x5 over 22m) kubelet Liveness probe failed: Get "http://10.0.43.148:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal Killing 21m kubelet Container coredns failed liveness probe, will be restarted
Normal Pulled 21m (x2 over 23m) kubelet Container image "registry.k8s.io/coredns/coredns:v1.11.1" already present on machine
Warning Unhealthy 13m (x80 over 23m) kubelet Readiness probe failed: Get "http://10.0.43.148:8181/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning BackOff 3m36s (x28 over 10m) kubelet Back-off restarting failed container coredns in pod coredns-76f75df574-465x5_kube-system(73e7c967-cf8f-4ee5-b525-1839d64e5058)
kubectl get pods -A -o wide
admin@ip-10-0-32-163:~$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default dnsutils 1/1 Running 1 (65m ago) 10h 10.0.35.236 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system aws-cloud-controller-manager-czwrm 1/1 Running 4 (65m ago) 19h 10.0.32.163 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system aws-node-4rtrh 2/2 Running 8 (65m ago) 19h 10.0.32.163 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system coredns-76f75df574-465x5 0/1 CrashLoopBackOff 11 (4m24s ago) 43m 10.0.43.148 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system coredns-76f75df574-b9r5t 0/1 CrashLoopBackOff 11 (2m53s ago) 43m 10.0.44.228 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system etcd-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 4 (65m ago) 19h 10.0.32.163 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system kube-apiserver-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 4 (65m ago) 19h 10.0.32.163 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system kube-controller-manager-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 4 (65m ago) 19h 10.0.32.163 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system kube-proxy-6cg92 1/1 Running 4 (65m ago) 19h 10.0.32.163 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kube-system kube-scheduler-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 4 (65m ago) 19h 10.0.32.163 ip-10-0-32-163.ap-south-1.compute.internal <none> <none>
kubectl get all -A
admin@ip-10-0-32-163:~$ kubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/aws-cloud-controller-manager-8d4m9 1/1 Running 3 (2m17s ago) 17h
kube-system pod/aws-node-lvl5p 2/2 Running 4 (3m7s ago) 17h
kube-system pod/coredns-76f75df574-jqjk4 0/1 Running 13 (38s ago) 10h
kube-system pod/coredns-76f75df574-nbd9c 0/1 Running 13 (50s ago) 10h
kube-system pod/dnsutils 1/1 Running 1 (3m7s ago) 10h
kube-system pod/etcd-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 14 (3m7s ago) 17h
kube-system pod/kube-apiserver-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 2 (3m7s ago) 17h
kube-system pod/kube-controller-manager-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 2 (3m7s ago) 17h
kube-system pod/kube-proxy-r2sh2 1/1 Running 2 (3m7s ago) 17h
kube-system pod/kube-scheduler-ip-10-0-32-163.ap-south-1.compute.internal 1/1 Running 5 (3m7s ago) 17h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.97.0.1 <none> 443/TCP 17h
kube-system service/kube-dns ClusterIP 10.97.0.10 <none> 53/UDP,53/TCP,9153/TCP 17h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/aws-cloud-controller-manager 1 1 1 1 1 node-role.kubernetes.io/control-plane= 17h
kube-system daemonset.apps/aws-node 1 1 1 1 1 <none> 17h
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 17h
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 0/2 2 0 17h
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-76f75df574 2 2 0 10h
sudo iptables -vnL
admin@ip-10-0-32-163:~$ sudo iptables -vnL
Chain INPUT (policy ACCEPT 576K packets, 102M bytes)
pkts bytes target prot opt in out source destination
5175 311K KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */
571K 100M KUBE-NODEPORTS 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes health check service ports */
5175 311K KUBE-EXTERNAL-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
575K 102M KUBE-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */
0 0 KUBE-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
0 0 KUBE-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
0 0 KUBE-EXTERNAL-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 580K packets, 110M bytes)
pkts bytes target prot opt in out source destination
7345 442K KUBE-PROXY-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */
7345 442K KUBE-SERVICES 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
579K 110M KUBE-FIREWALL 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-EXTERNAL-SERVICES (2 references)
pkts bytes target prot opt in out source destination
Chain KUBE-FIREWALL (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP 0 -- * * !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
Chain KUBE-FORWARD (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP 0 -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding conntrack rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-PROXY-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-PROXY-FIREWALL (3 references)
pkts bytes target prot opt in out source destination
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 REJECT 6 -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:metrics has no endpoints */ tcp dpt:9153 reject-with icmp-port-unreachable
0 0 REJECT 17 -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns has no endpoints */ udp dpt:53 reject-with icmp-port-unreachable
0 0 REJECT 6 -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp has no endpoints */ tcp dpt:53 reject-with icmp-port-unreachable
kubectl cluster-info
Kubernetes control plane is running at https://10.0.32.163:6443
CoreDNS is running at https://10.0.32.163:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'
cat /etc/resolve.conf
(on my master node)
admin@ip-10-0-32-163:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.
nameserver 10.0.0.2
kubectl -n kube-system exec dnsutils -- cat /etc/resolv.conf
(on a test pod deployed to kube-system to debug dns)
admin@ip-10-0-32-163:~$ kubectl -n kube-system exec dnsutils -- cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.97.0.10
options ndots:5
kubectl describe cm corends -n kube-system
admin@ip-10-0-32-163:~$ kubectl describe cm coredns -n kube-system
Name: coredns
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
Corefile:
----
.:53 {
log
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
BinaryData
====
Events: <none>
Cluster information:
- Kubernetes version:
v1.29.2
- Cloud being used: (put bare-metal if not on a public cloud)
AWS
Installation method:kubeadm init --pod-network-cidr=10.96.0.0/16 --service-cidr=10.97.0.0/16 --token-ttl 0
- Host OS:
Linux ip-10-0-32-163.ap-south-1.compute.internal 6.1.0-18-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux
- CNI and version: aws-vpc-k8s-cni v1.16.4
- CRI and version:
containerd containerd.io 1.6.28 ae07eda36dd25f8a1b98dfbf587313b99c0190bb