I have a local multi-node cluster created with kind under docker desktop for windows (2 worker nodes). I installed the latest version of kube prometheus operator. Everything seems to be working fine except grafana cannot connect to the prometheus service. This service is defined using ClusterIP type. Below is the kubectl describe service output. It has 2 pods running underneath, one on each worked. By default grafana is configured to connect to the service via dns name and the name resolves to the ClusterIP (10.96.104.255) below. This doesn’t work. Testing the data source from the grafana UI times out connecting to the service. However, if I configure the data source to connect to one of the two endpoints directly (10.244.1.6 or 10.244.2.9 below), it works just fine. The grafana pod is running in the same subnet (at 10.244.1.4) so that all seems to work fine. However, the grafana pod cannot connect to the prometheus-k8s ClusterID (10.96.104.255). The name resolution works. Just can’t connect to the service via ClusterIP.
I hope all that makes sense. Is there additional setup, routing, or any other configuration that needs to be added here to get thus working?
kubectl describe service prometheus-k8s -n monitoring
Name: prometheus-k8s
Namespace: monitoring
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.96.104.255
IPs: 10.96.104.255
Port: web 9090/TCP
TargetPort: web/TCP
Endpoints: 10.244.1.6:9090,10.244.2.9:9090
Port: reloader-web 8080/TCP
TargetPort: reloader-web/TCP
Endpoints: 10.244.1.6:8080,10.244.2.9:8080
Session Affinity: ClientIP
Events:
Cluster information:
Kubernetes version:v1.21.1
Cloud being used: Docker Desktop for Windows
Installation method: Multinode cluster created using kind
Host OS: Windows