Custom controllers; where to cache meta data (ephemeral)

General Discussions
1

New to kubernetes and go here.

I’m writing a controller that will monitor a specific subset of pods for state changes and publish these changes to various external data sources; cassandra, coredns, redis, etc…

I’m curious what approaches are best for caching this data within kubernetes environment. I assume etcd would be ideal for this but are there mechanisms in client-go (or the kubeapi directly via rest) to do this or will my controller need all the plumbing to interface with etcd directly? Or is there a better option all together.

The cache can be ephemeral and only matters if the cluster is up and hasn’t gone down.

Thx for any help.

1 Like
2

Are you using any particular framework (e.g. kubebuilder)? Some do implement their own caching mechanism.

re:etcd. There is no direct plumbing to communicate with etcd through kubernetes. The only thing that communicates with etcd is the kube-apiserver. All other components communicate with the kube-apiserver. It’s sort of the keystone for everything.

3

Wasn’t away of any frameworks but definitely will check them out now. I’m sorta jumping blindly into the deepend here.

Thanks for the info.

1 Like
4

Out of pure interest: what kind of changes are you reacting to? Can you just “watch” for changes to the Pods, or do you interact with them directly somehow?

On-topic: you can store some information in Custom Resource Definitions (CRD). Ultimately, yes, they are stored in etcd, but are made available via the Kubernetes API. If you will update your data very often, however, perhaps choose a different route, like a shared memcached service or something.

5

Mainly DNS/Service Discovery updates. All our orchestration, dns, etc… is done at a layer above all our k8s clusters. We try and design and build our services to be control plane agnostic. Applications that subscribe to this service ignore cluster-level DNS.

But I’m totally new to all this. New to go. Somewhat new to K8s. Literally using a controller as my first Go “hello world” program while learning kubernetes internals on the fly. (much fun)