Strategy for Operator Work Claims

I’ve been mulling over this idea on how to have any number of replica operators to handle queued work without a leader and I got something working today.

The example in that gist creates a CRD, a service account with role binding, a configmap with code, and a deployment with 3 replicas.

Once you deploy it, you can spam create some TaskOwner resources like so:

cat <<EOF | kubectl apply -f -
apiVersion: future.ideas.protosam/v1beta1
kind: TaskOwner
    name: mytaskname-$(openssl rand -hex 12)

Then see that they get claimed:

$ kubectl get task-owners
NAME                                  OWNER                                 AGE
mytaskname-0b47c71573cabd75b9e2e7f8   leaderless-operator-b6b6c8d54-pb68d   8s
mytaskname-21e021284a93c35bd59502c1   leaderless-operator-b6b6c8d54-48kdp   5s
mytaskname-26d19152a995f546a4e0dc0f   leaderless-operator-b6b6c8d54-pb68d   14s
mytaskname-43dd3c1cf807befc2d748c1a   leaderless-operator-b6b6c8d54-pb68d   10s
mytaskname-46ac838d5aaee4a35c8c1295   leaderless-operator-b6b6c8d54-pb68d   7s
mytaskname-522f19d9ca224518103f3a55   leaderless-operator-b6b6c8d54-pb68d   5s
mytaskname-58f3cbdcde1864967125206d   leaderless-operator-b6b6c8d54-48kdp   6s
mytaskname-5b4ba1c8e863edbdac497e9d   leaderless-operator-b6b6c8d54-pb68d   9s
mytaskname-6c5d56982b83bd5572053b7b   leaderless-operator-b6b6c8d54-pb68d   15s
mytaskname-7b3a9bf11e5392b439e9ffbd   leaderless-operator-b6b6c8d54-7r9zf   12s
mytaskname-8a8f771f1d5641effac9ec83   leaderless-operator-b6b6c8d54-7r9zf   11s
mytaskname-a46870f82f0f526307b6caa5   leaderless-operator-b6b6c8d54-pb68d   13s
mytaskname-d1a3fef9f9316f94fabcc3b1   leaderless-operator-b6b6c8d54-7r9zf   8s
mytaskname-f41e81af918426c282d9b39b   leaderless-operator-b6b6c8d54-7r9zf   12s

Anyways, just wanted to share this as it makes scalable, leaderless, operators a possibility. As for reliability, I have not rigorously tested this yet to confirm my observations thus far. So if you happen to know of any caveats to this, the feedback would be great.