Kuberentes liveness cannot visit service by dns

General Discussions
1

Cluster information:

Kubernetes version: 1.10.12
Cloud being used: (put bare-metal if not on a public cloud)
Installation method:
Host OS: centos7.4
CNI and version: calico 3.1.1

I deploy es and fluentd in same namespace test , and wirte below config to ensure fluentd can visit es:

        livenessProbe:
          failureThreshold: 5
          httpGet:
            host: elasticsearch-logging
            path: /
            port: 9200
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

Not work, then i use whole dns name, still failed:

Readiness probe failed: Get http://elasticsearch-logging.test.svc.cluster.local:9200/: dial tcp: lookup elasticsearch-logging.test.svc.cluster.local: no such host

I remove liveness part and use curl in fluentd pod, that works:

root@fluentd-es-2dvmf:/# curl http://elasticsearch-logging:9200/
{
  "name" : "elasticsearch-logging-0",
  "cluster_name" : "skydiscovery-es-cluster",
  "cluster_uuid" : "fr3oSzpHT_qP9HQJ1WygnA",
  "version" : {
    "number" : "6.2.4",
    "build_hash" : "ccec39f",
    "build_date" : "2018-04-12T20:37:28.497551Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
1 Like
2

The host in the probe is probably the host http header to use (don’t have the reference documentation handy now). In other words, it connects to the pod IP but it uses the http “Host:” header you specify (in case it is needed to match a specific vhost).

When you do curl http://somehost you are using DNS to resolve which host it is. That will resolve via DNS, connect to that host and also add an http “Host:” header with value somehost.

So, in the first you connect to the pod IP and set a custom http header.

In the second case, you resolve a domain (can be a kubernetes service or whatever), connect to that IP address and then do an http request (and, by default, it always sets the host header to the domain unless specified otherwise).

Does it make sense?

3

I test with dns name and whole dns name, all failed.
What make me confused is that i do not find any docs about this.

4

When doing the curl, you are probably using a service. Check documentation for services :slight_smile:

For the probe, check the reference API documentation for that field in particular.

If I’m not wrong, the difference is what I explained in the past email. But please let me know if it isn’t like that :slight_smile:

5

refer to Comment， seems not related to header.

6

Oh, you are right. Checking the docs, it doesn’t refer to the http header host: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#httpgetaction-v1-core

So, my guess is that the kubelet (that is the component performing this probe, IIRC) does not use the kubernetes DNS server (at least in your case).

But please go ahead and try to see if it’s using them or not :slight_smile:

Btw, is this a theoretical question? Or did you have a real world problem with this? Please feel free to share the root issue :slight_smile:

Thanks,

Rodrigo