[Question] cAdvisor implementation in kubelet

ryysud · December 6, 2022, 6:50am

Hi,

In kubelet, not all metrics of cAdvisor are exposed, but only those in the following option parameters are exposed by default.

cpu
memory
cpuLoad
diskIO
network
app
process
oom_event

github.com

kubernetes/kubernetes/blob/v1.25.4/pkg/kubelet/cadvisor/cadvisor_linux.go#L86-L95


      
          	includedMetrics := cadvisormetrics.MetricSet{
          		cadvisormetrics.CpuUsageMetrics:     struct{}{},
          		cadvisormetrics.MemoryUsageMetrics:  struct{}{},
          		cadvisormetrics.CpuLoadMetrics:      struct{}{},
          		cadvisormetrics.DiskIOMetrics:       struct{}{},
          		cadvisormetrics.NetworkUsageMetrics: struct{}{},
          		cadvisormetrics.AppMetrics:          struct{}{},
          		cadvisormetrics.ProcessMetrics:      struct{}{},
          		cadvisormetrics.OOMMetrics:          struct{}{},
          	}

The –enable_load_reader flag in cAdvisor is required to expose cpuLoad metrics, but is not configured in kubelet as below:

github.com

kubernetes/kubernetes/blob/v1.25.4/pkg/kubelet/cadvisor/cadvisor_linux.go#L63-L80


      
          func init() {
          	// Override cAdvisor flag defaults.
          	flagOverrides := map[string]string{
          		// Override the default cAdvisor housekeeping interval.
          		"housekeeping_interval": defaultHousekeepingInterval.String(),
          		// Disable event storage by default.
          		"event_storage_event_limit": "default=0",
          		"event_storage_age_limit":   "default=0",
          	}
          	for name, defaultValue := range flagOverrides {
          		if f := flag.Lookup(name); f != nil {
          			f.DefValue = defaultValue
          			f.Value.Set(defaultValue)
          		} else {
          			klog.ErrorS(nil, "Expected cAdvisor flag not found", "flag", name)
          		}
          	}
          }

Currently, those metrics are always 0. Is this the intended implementation?

ryysud · December 8, 2022, 3:05am

Is this implemented because we can pass --enable_load_reader (deprecated) via kubelet?

github.com

kubernetes/kubernetes/blob/v1.25.4/cmd/kubelet/app/options/globalflags_linux.go#L53


      
          	// e2e node tests rely on this
          	register(global, local, "housekeeping_interval")
          
          
	// These flags were implicit from cadvisor, and are mistakes that should be registered deprecated:
          	const deprecated = "This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed."
          
          
	registerDeprecated(global, local, "application_metrics_count_limit", deprecated)
          	registerDeprecated(global, local, "boot_id_file", deprecated)
          	registerDeprecated(global, local, "container_hints", deprecated)
          	registerDeprecated(global, local, "containerd", deprecated)
          	registerDeprecated(global, local, "enable_load_reader", deprecated)
          	registerDeprecated(global, local, "event_storage_age_limit", deprecated)
          	registerDeprecated(global, local, "event_storage_event_limit", deprecated)
          	registerDeprecated(global, local, "global_housekeeping_interval", deprecated)
          	registerDeprecated(global, local, "log_cadvisor_usage", deprecated)
          	registerDeprecated(global, local, "machine_id_file", deprecated)
          	registerDeprecated(global, local, "storage_driver_user", deprecated)
          	registerDeprecated(global, local, "storage_driver_password", deprecated)
          	registerDeprecated(global, local, "storage_driver_host", deprecated)
          	registerDeprecated(global, local, "storage_driver_db", deprecated)
          	registerDeprecated(global, local, "storage_driver_table", deprecated)

Topic		Replies	Views
How to start a local k8s cluster with kubelet expose cadvisor metrics? General Discussions development	0	982	September 4, 2019
Unable to configure the cadvisor parameter in kublet General Discussions	2	975	September 18, 2019
Details of the bug in builtin cadvisor (0.37.0) in kubelet of 1.19.2 General Discussions	1	804	January 20, 2021
How to change cadvisor metrics refresh rate? General Discussions	0	1109	September 4, 2019
Is there a way to collect resource utilization metrics of pods deployed in a specific k8s namespace without using a clusterRole? General Discussions	0	466	May 3, 2022

[Question] cAdvisor implementation in kubelet

Related topics