Identifying a leader using sidecar and changing the labels in service via sidecar

General Discussions
1

I am using apache atlas in an active/standby fashion. I want to direct the traffic to the active pod using kubernetes service (atlas-service in this case) itself. I’d prefer not to use HAProxy, LivenessProbe or ReadinessProbe
Is there a way to do this using an approach like- using a side-car keep on continuously checking for the leader pod (active in this case for atlas) and as soon as the first pod (active) gets terminated and the other pod becomes as a leader (passive → active), the sidecar recognises it and changes the label in the service (atlas-service) so that the traffic now gets directed to the newly elected active pod

2

It sounds like apache atlas should be designed as a StatefulSet. You might want to read up about running stateful applications in Kubernetes. It’s more involved than stateless apps, but not hard persay.

The documentation discusses running replicated stateful applications here, by way of example.

You don’t really need a headless service, all services support {pod_name}.{service_name} in DNS, but what is important is setting the “serviceName” of your statefulset if the name is not the same as the service.

As for tracking with sidecars, you can build your sidecars to use a lease to track the leader perhaps.

As for using a proxy, I have no real opinion on that. HAProxy works fine as does nginx for some things. I don’t think that matters too much. You’re likely going to have to code around some proxy service to get what you want, unless there’s a tool that already exists for apache atlas, similar to how we have vitess for mysql.

3

We already have atlas as a stateful set. And currently we use HAProxy for loadbalancing. What I want to do it use kubernetes service of atlas for the same by deploying a sidecar which tracks the leader and as the active pod dies and other one becomes a leader the sidecar must add a label to kubernetes service so that it starts directing to the newly appointed leader

4

Well services work on selectors. In theory I would assume this would work:

  • have the service select on a label called “leader: yes”
  • start your pods with “leader: no”
  • have a sidecar container with a service account that has permissions to update the pods in the namespace to update the leader status

I’ve never done this, but in my head it seems fine. I’m not sure what challenges you might run into or how reliable it would be.

If you do this, let me know how it goes, I’m definitely curious.

1 Like
5

But won’t this restart the pod when the label in a pod changes?

6

If you edit the label of a running pod, no.
If you edit the pod template in a deployment or statefulset, I suppose the replicas would be replaced with new posts that match the template.

edit: s/template/pod/