Help needed to connect pods to external network in kubernetes

General Discussions
1

Hi All,

We have setup two VMS 1 Linux VM and 1 Windows 2019 VM and we went a head and made Linux VM as Master and Windows 2019 as the worker node. We used the Flannel Network mode (l2bridge). We were able to create a deployment which runs two pods. And we were able to run the service also.
The container which we deployed in the POD runs a python script which connects to google. When we run the same container in the docker run command it is able to connect to google and get response. And when we are trying to access the same with the POD deployed we are getting connection error which we think could be due to DNS issue.

We are new to set up kubertenes. Could any one please help us in resolving this issue.

We followed the below link for support:

Any help on this is appreciated.

2

Is coredns working as expected or giving any errors in the logs?

3

Hi macintoshprime,

Thanks for the reply. Below are the commands we ran for the codedns logs and it shows no errors:

VM versions we are using:
Master: (UBUNTU) (16.04.6 LTS)
Node : (Windows 2019 1809 10.0.17763.316)

/home/ubuntu/kube/yaml# kubectl get pod --all-namespaces -o wide

NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-86c58d9df4-qfm2c 1/1 Running 0 3h15m 10.244.0.36 ip-10-0-2-179
kube-system coredns-86c58d9df4-r6mdk 1/1 Running 0 3h15m 10.244.0.35 ip-10-0-2-179
kube-system etcd-ip-10-0-2-179 1/1 Running 0 3h14m 10.0.2.179 ip-10-0-2-179

kubectl get svc --namespace=kube-system

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 3h17m

kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c coredns

log:

.:53
2019-03-01T17:44:44.434Z [INFO] CoreDNS-1.2.6
2019-03-01T17:44:44.434Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
[INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
[INFO] Reloading
[INFO] plugin/reload: Running configuration MD5 = 2394cf331ea25e9aacc36ddf69fafcdb
[INFO] Reloading complete
2019-03-01T19:34:58.522Z [INFO] 127.0.0.1:52595 - 63598 “HINFO IN 4423589095444525296.1811339747229552696. udp 57 false 512” NXDOMAIN qr,rd,ra 133 0.011971797s

4

What do you have in /etc/resolves.conf in the pod? If you switch to 8.8.8.8 does it work?

And does ping 8.8.8.8 works?

Kubernetes uses an internal DNS server. Checking that we will know if it might be the kubernetes DBS server or not (i.e.if it works using 8.8.8.8 it seems related to something internal to kubernetes).

And if it doesn’t work with 8.8.8.8, the ping will let us know if connectivity (icmp at least) works, letting DNS servers aside.

Once we know which fails and which doesn’t, we can dig forward on that direction :slight_smile:

1 Like
5

Hi Rata,
Thanks for your response. As per your suggestions below are the details retrieved
Could you please let us know what could be the issue

My pod does not have etc/resolves.conf as it is a windows base container.
We tried to ping 8.8.8.8 from master and worker node(windows node) and we got a response but when we tried to log into the pod and tried to ping 8.8.8.8 there was request time out

In the master(linux) etc/resolv.conf file below is the content
nameserver 10.0.2.5
nameserver 10.0.3.28
search abc.qwe.com

Below is our core DNS config file
Name: coredns
Namespace: kube-system
Labels:
Annotations: 

		Data
		====
		Corefile:
		----
		.:53 {
			log
			errors
			health
			kubernetes cluster.local in-addr.arpa ip6.arpa {
			   pods insecure
			   upstream 10.0.2.5 8.8.8.8
			   fallthrough in-addr.arpa ip6.arpa
			}
			prometheus :9153
			proxy . /etc/resolv.conf
			cache 30
			loop
			reload
			loadbalance
		}

		Events:  <none>
6

Oh, okay. If you can’t ping to 8.8.8.8 from the pod, then it’s not only DNS but IP connectivity is failing.

That might be related to some firewall, network overlay you use or some networking thing. But at a lower layer, IP, no DNS

So, I’d look into those things (firewalls, network overlay, if connection to the from the pod to the node works, etc to see where are packet dropped).

But I don’t know about windows. So I can’t really help you on the specifics on how to debug on windows. Sorry :frowning:

Nevertheless, those are the things I’d look at. And please report back with your findings :slight_smile:

7

Just looking for a simple answer to the solution but when you ran the join command on the windows side what did you use as the -KubeDnsServiceIP <Kube-dns Service IP> arg?

8

Hi
Thanks for your reply
Please find the below start command we have used to join the worker node

.\start.ps1 -ManagementIP -NetworkMode l2bridge -ClusterCIDR 10.244.0.0/16 -ServiceCIDR 10.96.0.0/12 -KubeDnsServiceIP < kube-dns service cluster ip> -LogDir C:\k\logs\

We have got the kube-dns service ip by running

$ kubectl get service -o wide --all-namespaces
Which gave below output

kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 46h k8s-app=kube-dns

And we used 10.96.0.10 in the join command

Please let me know in case of any corrections

9

That looks good, just wanted to rule that one out :slight_smile:

Are there any errors in the kubelet or kube-api logs on the windows host?

10

Hi Thanks for your response.
There are no specific logs which I could find. Is there any specific location where the logs can be found.

But I could find the below lines on the powershell which is running for kubelet not sure if this helps

Flag –cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet’s --config flag. See Set Kubelet parameters via a config file - Kubernetes for more information.
Flag –cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet’s --config flag. See Set Kubelet parameters via a config file - Kubernetes for more information.

22608 docker_sandbox.go:697] ResolvConfPath is empty.

6224 proxier.go:1041] Endpoint information not available for service default/ds-p-resol:. Not applying any policy

There are two ways we tried to connect the worker node:

  1. " .\start.ps1 -ManagementIP 10.0.2.95 -NetworkMode l2bridge -ClusterCIDR 10.244.0.0/16 -ServiceCIDR 10.96.0.0/12 -KubeDnsServiceIP 10.96.0.10 -LogDir C:\k\logs" – 10.96.0.10 is the kube-dns service ip also shared previously

In this case when we have tried to debug into the running container and tried to do a nslookup we got the below error "DNS Request timed out "
and the ipconfig inside the container says the below

And we are unable to ping anything inside the container which was created for the pod.

And then we tried another approach
2. " .\start.ps1 -ManagementIP 10.0.2.95 -NetworkMode l2bridge -ClusterCIDR 10.244.0.0/16 -ServiceCIDR 10.96.0.0/12 -KubeDnsServiceIP 8.8.8.8 -LogDir C:\k\logs" – which is google public DNS ip

It seems to reslove and give output for publicly exposed ips only and the corresponding ip config is as below

Reference screenshots:

image
image.png1252Ă—495 55 KB

But when we are using domain name instead of ips, they are not getting resolved.

But the same was tried in the ubuntu(master node) in which we deployed a sample container “busybox” to check the DNS resolving.
We found that when we get into the container of this pod, the ubuntu container was able to resolve the domain names.

So in a similar way how to configure a worker node(windows host) to resolve the domain names

And other issue which persists only in windows is when we try running command on pods as below it gives connection timeout error

PS C:\kubernetes test> kubectl exec -ti py-google-5d78fff-5jdfx – nslookup google.com
Error from server: error dialing backend: dial tcp 10.244.2.2:10250: i/o timeout

where 10.244.2.2 is the windows node internal ip which is different from the actual windows ip

There were some warning message found when we described the coredns pod similar to the below

Warning FailedCreatePodSandBox 36m (x2 over 36m) kubelet, ip-10-0-2-179 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “2d84d4079a5f909784a3c54eb5201fec4f8bcf401c3bbbd6e994e4923ff260de” network for pod “coredns-86c58d9df4-r6mdk”: NetworkPlugin cni failed to set up pod “coredns-86c58d9df4-r6mdk_kube-system” network: open /run/flannel/subnet.env: no such file or directory

11

Hi Surendra
I am trying to add a windows node to K8s cluster. Though in response to start.ps1 command execution, I am getting following error …

Generated CNI Config [@{cniVersion=0.2.0; name=cbr0; type=flannel; delegate=}]
Generated net-conf Config [@{Network=10.102.0.0/16; Backend=}]
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kubernetes”)
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kubernetes”)

– Any idea how can we fix this ?

I am using Hype-V nodes to build K8s cluster. MAC Spoofing is enabled on all nodes.
Windows Server 2019 is being used as an OS for a windows node

12

Hi @surendravts , i am trying to Join Windows worker node to the cluster and at the final when i try to join the node i get the followin error as below , can you please let me know where iam doing wrong.

k8s error_LI (2
k8s error_LI (21885Ă—594 278 KB