GKE Metrics are gone

General Discussions
1

Hi All,

I do not know if this is the right place, so forgive me if not the palce. I have a 3 cluster in GKE, development, staging and production. They’re all configured to collect metrics and logs of the pods. Everything was fine util 10 days ago when the production cluster stopped to collect metrics data, here an images showing the fact:

image
image2970×658 51 KB

Since then I started another node in that node pool and the new node is collecting data as normal.

Any ideas on why that node has stopped collecting metrics data?

Any help or hint will be very appreciated.

thank you.

2

I forgot the node details:

image
image844×278 16.3 KB

3

probably the metrics pod on that node is either malfunctioning/failing or not sending data (my best guess anyway).

check all daemonset pods are running:

kubectl get ds -n kube-system workload-metrics

check all pods are running in that daemonset

4

Resolved,

it was the gke-metrics-agent DaemonSet running on the misbehaving node that was in error. Restarting it resolved the issue.

2 Likes