How to add Relabel kubelet cadvisor metrics?

General Discussions
1

Trying to relabel cadvisor metrics with custom labels. It is getting relabeled but the prometheus scrapper is not discovering __meta_kubernetes_pod_label_a_id and __meta_kubernetes_pod_label_b_id. Relabels are refecting in prometheus.env.yaml and kubelet servicemonitor though.
My end goal is to query one of the cadvisor metrics with custom labels.
eg: container_cpu_usage_seconds_total{a_id=“1234”, b_id=“3456”}

Spec:
      Attach Metadata:
        Node:  false
      Endpoints:
        Bearer Token File:  /var/run/secrets/kubernetes.io/serviceaccount/token
        Honor Labels:       true
        Honor Timestamps:   true
        Port:               https-metrics
        Relabelings:
          Action:  replace
          Source Labels:
            __meta_kubernetes_pod_label_a_id
          Target Label:  a_id
          Action:        replace
          Source Labels:
            __meta_kubernetes_pod_label_b_id
          Target Label:  b_id

part of prometheus.env.yaml

   - job_name: serviceMonitor/monitoring/kube-prometheus-stack-kubelet/1   
      honor_labels: true                                                    
      honor_timestamps: true                                                
      kubernetes_sd_configs:                                                
      - role: endpoints                                                     
        namespaces:                                                         
          names:                                                            
          - kube-system                                                     
        attach_metadata:                                                    
          node: false                                                       
      metrics_path: /metrics/cadvisor 
    
    ............................
    ............................
    
      - source_labels:                                                                                             
        - __meta_kubernetes_pod_label_a_id                                                                                                                       
        target_label: a_id                                                                                                                                        
        action: replace                                                                                                                                                    
      - source_labels:                                                                                                                                                     
        - __meta_kubernetes_pod_label_b_id                                                                                                              
        target_label: b_id                                                                                                                               
        action: replace
1 Like