Trying to understand Lease Lock

General Discussions
1

I am trying the go through the Leader Election process manually.

First I started by creating a Lease.
kubectl apply -f my-lease.yaml

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  labels:
    apiserver.kubernetes.io/identity: my-le-test
    kubernetes.io/hostname: pod-lease
  name: my-lease
  namespace: default
spec:
  holderIdentity: pod-1
  leaseDurationSeconds: 40
  renewTime: "2024-10-29T23:57:36.059086Z"

If lease from pod-1 is still valid. I assume pod-2 should not be able to acquire a lease. So there should be some mechanism to prevent pod-2 to update the Lease. But then I run:
kubectl patch Lease my-lease -p '{"spec":{"holderIdentity":"pod-2","renewTime":"2024-10-29T23:57:37.059086Z"}}'

it just responded with
lease.coordination.k8s.io/my-lease patched
and the Lease has been updated.

So it seems my assumptions are wrong? How does the Lease Lock work in k8s?

Can I simulate the Lease Lock using kubectl?

2

Hi,
Please check client-go/tools/leaderelection/leaderelection.go at master · kubernetes/client-go · GitHub
There is some logic “under the hood”. Kubernetes api allows to update lease records without any validations.

3

Thanks I read the code but where does the Lock happens? If these logic all happens in the pod itself it would be inaccurate since if leader pod is down. pod-1 and pod-2 could simultaneously see that leader is down and try to claim it.

I’ve dig down to this leaselock.go file but this file does not seems to have any lock mechanism applied to it. ( it just calls ll.Client.Leases().Update() which I assume is identical to kubectl patch Lease ? )

4

Yes, pods might try to simultaneously put lock and become a leader.

What do you mean by this? What mechanism are you looking for?

Yes, this is similar to the patch command.

5

Let’s say pod-1 is down.
pod-2 sees it so pod-2 issued an update to the Lease and got a success.
in the meantime pod-3 also sees this and issued an update to the Lease. I assume this should be failing or should not happen since the code prevents it?

Where exactly in the code that prevents this? Since pod-2 and pod-3 runs the same code at the same time. Following the code, they will be getting the same responses from the api-server, that

  1. the lease has expired
  2. ll.Client.Leases().Update() returns no error

there should be a shared lock somewhere but I can’t find it.

6

I assume you are missing the point. pod-2 does not care whether pod-1 is down or not. pod-1 might get restarted and remain leader (if get back up real quick). pod-2 is going to check whether the lease record was updated or not. If the lease record has not been updated for --leader-elect-lease-duration after the last update, then pod-2 is going to try to apply lock.

7

Yes I understand this. But let’s say pod-1 is really down and won’t get back up.

pod-2 is going to do it’s due diligence ( check update time, etc ) then decided to acquire the lease.
pod-3 is doing the same thing at the same time right? So it will try to acquire the lease too.

  1. In the start, Lease holder is pod-1 but expired
  2. pod-2 acquires the lease (Lease holder: pod-2)
  3. pod-3 acquires the lease in like…1ms after (Lease holder: pod-3)
  4. pod-2 thinks it has the lease. Now it do the leader thing.
  5. pod-3 thinks it has the lease too. Now it do the leader thing.

Now we ended up with 2 pods thinking it’s the leader. Which should be prevented.

But how?

8

if pod-3 updates lease immediately after pod-2, in this case pod-2 is going to receive the client-go/tools/leaderelection/leaderelection.go at master · kubernetes/client-go · GitHub callback and stop being the leader, since its identity does not correspond to identity from the lease record.

9

So, for a brief moment, it cannot prevent two leader exists at the same time?

10

Yes, that is correct.

11

Nice! I can now evaluate my choices.

Thank you for the help!