We have a DaemonSet that requires different amount of resources depending on node capacity. In clusters with vastly different node sizes, this creates a problem - either on some nodes our Pods will not have enough resources, or they will have too much (hence wasting cluster resources), or both.
Is there a way around it? Ideally, without developing our own custom controller…
Thanks!
Alex
I don’t think there is a canned solution for you. I have “joked” about a node-scaled DaemonSetSet but never implemented it.
1 Like
Thanks, @thockin. I am curious… if someone wanted to do something like this, is there an elegant-enough way to do it without duplicating DaemonSetController? E.g., using the upcoming in-place Pod resize feature? In-Place Update of Pod Resources · Issue #1287 · kubernetes/enhancements · GitHub
You could do it with VPA but that seems harder than it needs to be. I envisioned an API which had a daemonset template plus a node label key plus a list-map of label-value to cpu and memory setting. The controller just creates a daemonset for each label key-value, based on the template, but varies by the resources.
Small nodes get small CPU, large nodes get large.
This would not scale to huge numbers of variations.
Another approach would be look at each node’s allocatable CPU and memory and assign a label automatically, so you don’t need to configure the possible values a priori.
It could be done entirely as a CRD, out of core .
1 Like
Is anyone able to make this work. We are in similar situation.