Readiness probe, headless service and DNS

General Discussions
1

Hi everyone,

Given a pod of a statefulset that is starting: do you know if there is there is causal relation between “readiness probe tells the pod is now ready” and “headless service name is resolvable inside the pod”?

The reason for that question is I have a mongodb replicaset complaining during startup:

getaddrinfo(“mymongo-mongodb-primary-0.mymongo-mongodb-headless.mongodb”) failed: Name or service not known.

Then, because the headless service name of the pod is not resolvable, the pod can’t determine that it is part of the MongoDB replica set and fails to initialize properly:

This node is not a member of the config

I’ve written a little C program that will wait until “mymongo-mongodb-primary-0.mymongo-mongodb-headless.mongodb” is resolvable (by calling getaddrinfo) before launching mongod, but that does not happen (I waited 1 minute max after the pod is started).

If I disable the readiness probe then the little program eventually manages to resolve “mymongo-mongodb-primary-0.mymongo-mongodb-headless.mongodb” (let’s say 10 seconds after the pod is started).

Is it a known behavior?
I can observe it on a 1.17.5 cluster (3 nodes, 2 masters) installed with Kubespray 1 month ago. I upgraded it to 1.18.3 (with Kubespray) but it is still the same.

Thanks for any insight!

Bruno.

2

Hi everyone,

Let me answer myself :slight_smile:
It looks like the answer is yes.

I’ve finally found this documentation page…:
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

… mentioning this note:

Because A or AAAA records are not created for Pod names, hostname is required for the Pod’s A or AAAA record to be created. A Pod with no hostname but with subdomain will only create the A or AAAA record for the headless service ( default-subdomain.my-namespace.svc.cluster-domain.example ), pointing to the Pod’s IP address. Also, Pod needs to become ready in order to have a record unless publishNotReadyAddresses=True is set on the Service.

1 Like