[Question] cAdvisor implementation in kubelet

ryysud · December 6, 2022, 6:50am

Hi,

In kubelet, not all metrics of cAdvisor are exposed, but only those in the following option parameters are exposed by default.

cpu
memory
cpuLoad
diskIO
network
app
process
oom_event

github.com

kubernetes/kubernetes/blob/v1.25.4/pkg/kubelet/cadvisor/cadvisor_linux.go#L86-L95


      
          	includedMetrics := cadvisormetrics.MetricSet{
          		cadvisormetrics.CpuUsageMetrics:     struct{}{},
          		cadvisormetrics.MemoryUsageMetrics:  struct{}{},
          		cadvisormetrics.CpuLoadMetrics:      struct{}{},
          		cadvisormetrics.DiskIOMetrics:       struct{}{},
          		cadvisormetrics.NetworkUsageMetrics: struct{}{},
          		cadvisormetrics.AppMetrics:          struct{}{},
          		cadvisormetrics.ProcessMetrics:      struct{}{},
          		cadvisormetrics.OOMMetrics:          struct{}{},
          	}

The –enable_load_reader flag in cAdvisor is required to expose cpuLoad metrics, but is not configured in kubelet as below:

github.com

kubernetes/kubernetes/blob/v1.25.4/pkg/kubelet/cadvisor/cadvisor_linux.go#L63-L80


      
          func init() {
          	// Override cAdvisor flag defaults.
          	flagOverrides := map[string]string{
          		// Override the default cAdvisor housekeeping interval.
          		"housekeeping_interval": defaultHousekeepingInterval.String(),
          		// Disable event storage by default.
          		"event_storage_event_limit": "default=0",
          		"event_storage_age_limit":   "default=0",
          	}
          	for name, defaultValue := range flagOverrides {
          		if f := flag.Lookup(name); f != nil {
          			f.DefValue = defaultValue
          			f.Value.Set(defaultValue)
          		} else {
          			klog.ErrorS(nil, "Expected cAdvisor flag not found", "flag", name)
          		}
          	}
          }

Currently, those metrics are always 0. Is this the intended implementation?

ryysud · December 8, 2022, 3:05am

Is this implemented because we can pass --enable_load_reader (deprecated) via kubelet?

github.com

kubernetes/kubernetes/blob/v1.25.4/cmd/kubelet/app/options/globalflags_linux.go#L53


      
          	// e2e node tests rely on this
          	register(global, local, "housekeeping_interval")
          
          
	// These flags were implicit from cadvisor, and are mistakes that should be registered deprecated:
          	const deprecated = "This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed."
          
          
	registerDeprecated(global, local, "application_metrics_count_limit", deprecated)
          	registerDeprecated(global, local, "boot_id_file", deprecated)
          	registerDeprecated(global, local, "container_hints", deprecated)
          	registerDeprecated(global, local, "containerd", deprecated)
          	registerDeprecated(global, local, "enable_load_reader", deprecated)
          	registerDeprecated(global, local, "event_storage_age_limit", deprecated)
          	registerDeprecated(global, local, "event_storage_event_limit", deprecated)
          	registerDeprecated(global, local, "global_housekeeping_interval", deprecated)
          	registerDeprecated(global, local, "log_cadvisor_usage", deprecated)
          	registerDeprecated(global, local, "machine_id_file", deprecated)
          	registerDeprecated(global, local, "storage_driver_user", deprecated)
          	registerDeprecated(global, local, "storage_driver_password", deprecated)
          	registerDeprecated(global, local, "storage_driver_host", deprecated)
          	registerDeprecated(global, local, "storage_driver_db", deprecated)
          	registerDeprecated(global, local, "storage_driver_table", deprecated)

Topic		Replies	Views
How to change cadvisor metrics refresh rate? General Discussions	0	978	September 4, 2019
Is there a way to collect resource utilization metrics of pods deployed in a specific k8s namespace without using a clusterRole? General Discussions	0	350	May 3, 2022
Windows pod level metrics (cadvisor) and Prometheus Windows	1	2023	July 23, 2021
Data discrepancy in current memory usage metrics between kubectl top command and metrics exposed by kubelet(scraped by Prometheus) General Discussions prometheus	1	1903	November 9, 2023
What is kubelet server Pod Container endpoint used for? General Discussions development	0	550	February 7, 2019

[Question] cAdvisor implementation in kubelet

Related Topics