We need a proper way of making Custom Controllers, it’s 2018 and we’re boilerplating / copy pasting stuff.
3 Likes
I managed to make the sample controller from scratch using kubebuilder. Imho, I think KubeBuilder is a step forward against copy-pasting or/and using code-generator. I like the Injector/arguments/GenericController scheme, still hard but nice when you figure it all out. It’s still not what I’d like to do when making Custom Controllers, but thumbs up for the person(s) who worked on it.
It seems you already found kubebuilder, which is the current prototype within SIG API Machinery for the “next step” in this area.
We’re working on better documentation for kubebuilder (as well as general controller concepts) to make “figuring it all out” much easier. If you can elaborate on the friction you encountered, I’ll make sure it gets to the people working on kubebuilder.
I’m also interested in your thoughts on “what [you’d] like to do when making custom controllers”. That is, ignoring what’s available now, what are some aspects of your ideal user experience? It’s still pretty early in our efforts to make controller frameworks, so feedback from users at this point will go a long way.
Lastly, I’ll give a shameless plug for another framework for writing custom controllers that focuses on simple use cases:
https://metacontroller.netlify.com/
In a nutshell, it sort of takes the webhook model from things like auth plugins and admission plugins and applies it to custom controllers. I’d be interested in hearing whether you think your use case for writing a controller would fit in this model.
2 Likes
@radu-munteanu Here, I found good articles which compares these methods under-the-hood-of-the-operator-sdk and under-the-hood-of-kubebuilder-framework. It might help you understand comprehensively.
1 Like
Thanks for all the links. I’ve read just a little bit on both metacontroller and operator before.
I had to start with something, and I had a list… I’ve started with the Sample Controller and the KubeBuilder, as I thought Maciej Szulik’s presentation at KubeCon was the most comprehensive, explaining a lot of does and don’ts when implementing a custom controller.
One point I wanted to make with this topic is that there is no out of the shelf library that everyone can use. Everyone is doing the custom controllers in some other way. I get it, open source, everyone embrace diversity and creativity and everyone can shoot themselves in the foot if they want or invent the next big thing, but what if more people would focus on one great library than having too many options and too little standardization. In the end, everyone has the same goal: making custom controllers easy.
Now, replying to @enisoc specifically on the friction part, I’ve been coding in Go for about three years, I’m probably not the best developer when it comes to coding, but what I did find in common to all the libraries used is the ease of use, I didn’t have to read several pages of documentation to understand how things are put together. Just by looking at functions and procedures I could easily start using those in a new project, with no boilerplate. For KubeBuilder, some things are not that easy to figure out (not v hard, but not easy), and here are some examples:
- there’s a main() for the cli built, and then a critical init() in the inject package; if someone looks at the main func, it doesn’t understand what does inject run, what InjectArgs are defined.
- you find the init, you find the controller, then you want to use the existing resources (like Deployments), and you look at the surface, look at the patterns you see generated, and you think “I might need a new Informer and a way to change the deployment, so an API client”…Where can I find an API client? You see that the arguments you added in init have only one clientset, your custom client set… Can I extend my current API client to work with deployments? Of course not, because a client is bound to one API group (hint: setConfigDefaults). Wait a minute, the InjectArgs contain some other InjectArgs from kubebuilder… and going down that path, you see that all the existing v1 resources are already there for you to use, so all you need is to do is use
arguments.KubernetesClientSet.AppsV1()andarguments.ControllerManager.GetInformerProvider(&appsv1.Deployment{}).(v1informers.DeploymentInformer).Lister()and you’re done (ok, maybe you need to add the Deployment informer in init). - you have two resources to look for (like in the sample controller - your Foo and the Deployment), in the sample-controller you’d manipulate the event handlers of each Informer so, eventually, in your syncHandler func, you know that you have a key for a Foo object. In KubeBuilder, when watching the Deployments, you make yourself a Predicate, and the Predicate has some functions that return a bool variable that describes if you want or not to process the object futher, adding its key to your queue and, eventually, into the Reconcile function. The Reconcile function has a Key paramenter, not an object (which makes sense, because the queue works with Keys). So, instead of having in Reconcile a key that you know for sure is from a Foo object, like for the syncHandler, you end up with a key that might be a Foo, might be a Deployment, might be any key for any of the resources you need to handle.You end up querying your custom resource lister, then the Deployment lister, to find out what kind of object you have. When you get a Deployment, you’ll then check if the owner is a Foo. So instead of working with only one type of key at this point, you end up double checking stuff that might have been nicer to process somewhere before. What if the predicate’s function would allow you to return another object, so I could drop the current object’s key, returning false, yet testing from start the new object? Probably these things are done the way they are for a reason. The issue here is that the Key itself offers too little information: name and namespace. I could have multiple resources with the same name and namespace.
How do I imagine a good user experience… I don’t have time left for a better picture… Bear with me… I’d see all informers, clients, reconciler, predicates, the link between resources and predicates, and, finally, the controller all created in one place:
- make informers/informerfactories based on resource type
- make api clients
- make a reconciler{listers, clients} (implements
Reconcile(key)) - create enqueueHandlers (some kind of Predicates)
- create a link between the resource type and enqueueHandlers (similar to gc.Watch() in KubeBuilder, but maybe not doing watch(), watch(), watch())
- create a new controller adding all in
- start the controller
If I look at all the points, I can see them put together in one file, in one screen. I could understand the whole flow, the whole picture just by looking at this:
informers (resource) -> enqueue (-> queue) -> reconcile (listers, clients)
I know K8s is a huge project with a lot of thinking put into it. This is just my own opinion on custom controllers. Take everything I’ve said with a grain of salt. I’m no expert and I don’t want to be the only one coming with feedback on custom controllers.
I hope others can share their thoughts on this matter so that, in the future, we’ll all end up with a better, easier to use platform.
2 Likes
Thanks for the detailed writeup. I work on kubebuilder so this sort of feedback is helpful.
One challenge is that a well written Controller uses a level based approach which is very different from your typically http server which uses an edge based approach. This seems to be where a lot of folks get tripped up.
If you haven’t already checkout http://book.kubebuilder.io/basics/what_is_a_controller.html which explains the more about this
-
We are working on a universal library for writing controllers that we hope to upstream into client-go and addresses some of your points around the inject package and is designed to look more like the go standard libs. See the PR for the interfaces and types here: https://github.com/kubernetes-sigs/kubebuilder/pull/187
-
You may want to take a closer look at the mapping functions between events and object keys. The canonical pattern is to take an event for one type (e.g. Pod) and map it to a key for another type (e.g. ReplicaSet) before enqueuing it, not after Reconciling it.
-
You a critical step in the informers enqueue pipeline is mapping an object to a key of the reconcile type: informers (resource) -> map-to-key-of-reconcile-type (func) -> enqueue -> reconcile. This is necessary to batch events for a object together and to properly multiplex events to a single object. If you are watching Pods for a ReplicaSet, and you get 100 Pod events for the same RepliaSet, you want to Reconcile it once, not 100 times. For this to be possible.
-
Not having the type in the reconcile key was intentional because handling multiple types in your Reconcile is usually an anti-pattern (with a few Exceptions such as autoscalers that work on any type). As a rule of thumb, if your Controller doesn’t need to watch on arbitrary types it doesn’t know about at compile time, you probably only want it to Reconcile 1 type per method. Other types of objects should be mapped before they are enqueue, not after.
3 Likes
@pwittrock Nice insights! 
Looking forward for v2 lib.
Thanks for good insights.
I have been looking at apiserver-builder recently and found that the controller parts generated by it are much easier to follow than those generated by Kubebuilder. Is this because the controller generation in apiserver-builder has not yet evolved to the point of supporting Predicates when creating event subscriptions and event mapping to batch events when dispatching for reconciliation?
Also, I was wondering if there is any plan to consolidate the controller generation bits from both of these libraries, instead of having two different approaches for controller generation? Then apiserver-builder can depend on this library (could be Kubebuilder v3 or something like ‘controller-generator-lib’).
@radu-munteanu here’s my attempt at comparing what I call 2nd-generation options: Kubebuilder, Operator-SDK and Metacontroller
https://admiralty.io/kubernetes-custom-resource-controller-and-operator-development-tools.html
My blog post might be outdated in just a few weeks 
but I’ll try to keep up 
3 Likes
December post (I forgot to send it 
)
End year review: I think the controllers business is still pretty bad, seven months since I started the topic. The KubeBuilder (K8s 1.12 release) from being over complicated, now it’s over simplified. I do like that actually the whole app is a manager that can have multiple controllers. I don’t like that I am somewhat encouraged to change only the Reconcile function, even though there is no built-in leader election yet (or so I’ve read: Leader election · Issue #230 · kubernetes-sigs/kubebuilder · GitHub). The sample controller hasn’t changed much, it still uses the same callback functions approach so much JS-like, not so much Go-like (that applies to KubeBuilder as well).
Now, another thing that bothers me is the fact that all these solutions are made in such a way that they dictate the architecture of the application. I think the controller logic should be more like a module of my app, from where I can get data and send back processed data through channels. It should be just one piece of a puzzle, not the whole puzzle.
Lately, I’ve been looking a bit on how to make the whole logic from scratch just using the api machinery and client go libs. I haven’t done much, for now, I ended up with some code that looks more or less like the sample-controller.
January update
I’ve been trying to update my KubeBuilder Sample Controller with latest code from K8s 1.13 release. Everything is almost the same as it was in 1.12, but I’ve got two errors at runtime. I’ve posted on Slack this issue on Thurday, unfortunately, I haven’t received any answers.
Hi, I have a few errors with kubebuilder 1.0.7, before I used 1.0.5 and I had no errors (edited)
when I apply a new resource I get this error"level": "error", "ts": 1548256859.920939, "logger": "kubebuilder.controller", "msg": "Reconciler error", "controller": "foo-controller", "request": "default/foo-sample-01", "error": "resource name may not be empty",the resource is updated but I get this error
if I update a resource, like increasing the number of replicas, the replicas are increased, but I still get this other error"level": "error", "ts": 1548259361.6205256, "logger": "kubebuilder.controller", "msg": "Reconciler error", "controller": "foo-controller", "request": "default/foo-sample-01", "error": "Operation cannot be fulfilled on deployments.apps \"foo-sample-01-deployment\": the object has been modified; please apply your changes to the latest version and try again",
1 Like