We are setting up a Kubernetes cluster within a Private virtual Network. All master nodes and worker nodes are in a same private subnet but in different Availability Zones and they are accessible through a Public Load balancer.
All Kubernetes service pods are up and running but in kubernetes core-dns and api-server pod logs we are seeing below errors related to Azure DNS:
Core DNS Logs:
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7 linux/amd64, go1.13.6, da7f65b
[ERROR] plugin/errors: 2 1898679200261753048.7155559289384615854. HINFO: read udp 192.168.16.4:37176->168.63.129.16:53: i/o timeout
[ERROR] plugin/errors: 2 1898679200261753048.7155559289384615854. HINFO: read udp 192.168.16.4:41483->168.63.129.16:53: i/o timeout
[ERROR] plugin/errors: 2 1898679200261753048.7155559289384615854. HINFO: read udp 192.168.16.4:46000->168.63.129.16:53: i/o timeout
API Server Logs:
E0901 04:17:35.747411 1 available_controller.go:420] failed with: failing
or missing response from https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: Get
https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting
for connection (Client.Timeout exceeded while awaiting headers)
I0901 04:17:38.490026 1 log.go:172] http: TLS handshake error from 168.63.129.16:53930: EOF
E0901 04:17:40.750482 1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing
or missing response from https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: Get
https://10.100.127.203:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting
for connection (Client.Timeout exceeded while awaiting headers)
W0901 04:17:43.815365 1 lease.go:224] Resetting endpoints for master service “kubernetes” to
[190.160.2.4 190.160.2.5]
I0901 04:17:44.493715 1 log.go:172] http: TLS handshake error from 168.63.129.16:53947: EOF
Security Group Rules that we added in private subnet for our Master and Worker Nodes :
Inbound rules:
Port 0-65535
Protocol TCP
Source Any
Destination Any
Action Allow
Port 0-65535
Protocol UDP
Source Any
Destination Any
Action Allow
Outbound Rules:
Port Any
Protocol Any
Source Any
Destination Any
Action Allow
Could someone please help us in understanding and resolving these issues. We stucked due to these issues from long time.
Thank you in advance !!