How are controllers selected by Kubernetes?

General Discussions
#1

As background I’m working on a SaaS product and I want to be able to offer up the ability to switch different applications in and out for one another (assuming backwards / forwards compatibility). As part of this, I’m evaluating different ways to keep different kubernetes clusters in sync with potentially heterogeneous deployment configurations. This is challenging for a few different reasons, but application and cluster lifecycle management is right at the forefront of my concerns right now. To accomplish this I’m rapidly coming to the conclusion that it’s necessary to have (at least?) 1 Operator to manage the lifecycle.

I’ve been through a decent chunk of the documentation but I have a few questions about how kubernetes works under the hood. In particular:

  • It sounds like Controllers are designed to work on a single Kind, does this mean writing a custom Operator implies defining a custom Kind?

  • How does kubernetes decide which Controller(s) are applied? Is there some order of precedence or some inclusion / exclusion criteria?

Can anyone offer up their $0.02 on this? It would be greatly appreciated!

Thank you

#2

It sounds like Controllers are designed to work on a single Kind, does this mean writing a custom Operator implies defining a custom Kind?

This is very common, and is how you can use the Kubernetes API machinery to serve your own API. But you can also write controllers for other Kinds. All it means to be “a controller” is that you consume the k8s API and react to it somehow. That could be to run a pod when it gets scheduled or to instantiate iptables rules or simply to log something in an external system. There are very few rules here.

How does kubernetes decide which Controller(s) are applied?

It doesn’t! Controllers are self-registered. Anyone can run a controller - in the cluster or from outside, as long as they have permissions to access the API. When an event happens (say a Pod is created), all connected watch clients will get an update. There’s not sequencing or interlock on those notifications - it’s all async.

For your problem statement, you could maybe use labels on a custom Kind to indicate which of your controllers you want to use. then each controller can watch for that label and only receive events when that subset changes.

#3

I really like idea of using labels. I think that would go a long way towards solving some of the problems I see potentially cropping up.

Thank you, this was very helpful, though I must say I’m a bit surprised! It sounds like the system could end up with cycles.

#4

Ideally all controllers are disjoint or at least driving to the same result. If they never “settle” then, yes, controllers can oscillate.

1 Like