Streaming Kubernetes events to Kafka

General Discussions
1

Hello everyone,

Do you know of any native or non-native mechanism for Kubernetes that enables streaming resource events (pod/service/configmap added/updated/deleted) to Kafka or a webhook?

My major concern with Kubernetes watches is the limited scalability of event processing. I need to consume events e.g., for all pods deployed in the cluster, and with watches, as far as I experienced, one can only have a single thread receiving the events - the one that initiated the watch. It seems to be a bottleneck for large workloads.

At best, I expect events to be sent to an event broker and then consumed from the queue by a group of processing workers.

Looking for something similar to Webhook backend in Kubernetes Audit feature (Auditing - Kubernetes) but for resource events not resource requests/responses.

Thanks

2 Likes
2

Hi Bzurowski,

For now, one could combine the iris project that sends configured k8s events to a webhook (https://github.com/olegsu/iris) with kafka http bridge (https://github.com/strimzi/strimzi-kafka-bridge) to somewhat achieve the effect of streaming the k8s events to a kafka topic.

imho, a kafka-connect connector for k8s events would be an interesting tool as it would enable creation of flexible cluster management pipelines using minimal scripting amount.

1 Like