[K8S 1.13.1] Programs running inside PODs cannot connect to internet

General Discussions
1

I just got K8S 1.13.1 installed on my Linux box, and I could install helm chart successfully. But the programs running inside containers can’t connect to internet.

Note that the linux box is behind a company proxy, and I could connect to internet successfully on the node via the proxy.

export http_proxy=“http://www-proxy.mycompany.com:80”
export https_proxy=“http://www-proxy.mycompany.com:80”
curl www.google.com

Do anyone have any suggestion? Thanks.

The OS is Oracle Linux Server release 7.5. The network add-on installed is flannel.

2

Hi
I also run same setup of kubernetes on Oracle Linux behind corporate firewall.

I have tested installing package from yum by setting proxy in yum.conf

Just make sure you can ping the corporate network gateway from within the pod and see which application is trying to connect to internet,for each application specifing proxy is different

3

@boksskumar Thanks for the response.

Unfortunately, the corporate network proxy isn’t pingable inside the pod. I just started a busybox POD using the following yaml file,

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

Then I logged into the pod using the following command,

kubectl exec -it busybox sh

Afterwards, I tried to ping the corporate network proxy. But I got a “ping: bad address ‘www-proxy.example.company.com’” response.

Note that I can ping the corporate network proxy on the node, and can connect to internet as well. But it doesn’t work inside POD.

4

Hi
You have to enter the address of corporate nameservers inside resolv.conf file so that name resolution start happening.

Also I try pinging the gateway,the same gateway which is within the host from within the pod.if it is happening then dns resolution is the issue which will be resolved by making entry in pods resolved.conf

5

I tried to get the same entry in resolved.conf on node to be added into the resolv.conf in POD, but it did not work either.

I believe the coredns is working, because when I tried to ping other service name in the cluster, the service name could be resolved as the correct clusterIP. But the clusterIP isn’t pingable either.

It seems that the node’s resolv.conf is used as upstream by the coredns, please see the following output,

-bash-4.2$ kc get configmap -n kube-system coredns -o yaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2018-12-18T20:58:09Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "86803"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 9fe8fdf0-0307-11e9-815c-0021f635b8e1
6

Note that everything is OK when previously I got K8S 1.9.7 installed. I only ran into this issue when I installed K8S 1.13.1.

7

Are you able to ping between pods running on 2 different hosts?

8

Currently I have only a one-node cluster. Just as I mentioned earlier, When I tried to ping another service name, then the service name could be resolved as the correct cluster IP.

But When I tried to call the REST API exposed by another service, it works,

curl http://example-service-name:6710/REST-API-PATH

9

This issue should have nothing to do with DNS, because it doesn’t work either when I ping another node IP (The target node isn’t a member of the K8S cluster) inside a POD.

It’s really frustrating. I guess it could be a compatibility issue between K8S and network add-on, such as flannel, Calico… There should a matrix to state clearly what version of each network add-on works with each K8S version.

10

It works after downgrading the flannel version from master to 0.10.0.

11

Just as a general note, service IPs are not pingable. They are essentially “fake” virtual IPs that are mappings to the exposed ports and protocol attached to the service. You will be able to use curl or other protocol specific tools to hit the port that is exposed but not ping them. My usual goto test is to pop in and use netcat to see if I can hit a remote service’s port :slight_smile:

12

@mrbobbytables

Thanks for the info. Yes, you are right that the service IPs are not pingable. I realized it a couple of days before.

Actually I summarized a couple of common “best practices” on installing kubernetes:

  1. If we install kubernetes per the official guide, then we will install the latest version. It isn’t good, we should always install a specific version instead of the latest version, otherwise, we may be hitting some potential issues introduced by the new version.

  2. Configure http(s)_proxy and no_proxy correctly.
    If we are behind a company proxy, then we should configure the http(s)_proxy and no_proxy correctly. The IP Address of the API server should be added into the no_proxy list, otherwise kubelet will try to connect to the API server via the proxy, accordingly it will definitely fail. The IP addresses of the nodes should be added into no_proxy.

Another point is that we should configure correct http(s)_proxy and no_proxy for docker engine as well.

  1. Restart machine when re-install K8S.
    We should always restart machine after “kubeadm reset”, especially after uninstalling a network add-on, i.e., flannel.