I have a local multi-node cluster created with kind under docker desktop for windows (2 worker nodes). I installed the latest version of kube prometheus operator. Everything seems to be working fine except grafana cannot connect to the prometheus service. This service is defined using ClusterIP type. Below is the kubectl describe service output. It has 2 pods running underneath, one on each worked. By default grafana is configured to connect to the service via dns name and the name resolves to the ClusterIP (10.96.104.255) below. This doesn’t work. Testing the data source from the grafana UI times out connecting to the service. However, if I configure the data source to connect to one of the two endpoints directly (10.244.1.6 or 10.244.2.9 below), it works just fine. The grafana pod is running in the same subnet (at 10.244.1.4) so that all seems to work fine. However, the grafana pod cannot connect to the prometheus-k8s ClusterID (10.96.104.255). The name resolution works. Just can’t connect to the service via ClusterIP.
I hope all that makes sense. Is there additional setup, routing, or any other configuration that needs to be added here to get thus working?
kubectl describe service prometheus-k8s -n monitoring
IP Family Policy: SingleStack
IP Families: IPv4
Port: web 9090/TCP
Port: reloader-web 8080/TCP
Session Affinity: ClientIP
Cloud being used: Docker Desktop for Windows
Installation method: Multinode cluster created using kind
Host OS: Windows