KMS V2 Encryption Cache Issue

General Discussions
1

I deployed my KMS V2 encryption plugin in KIND Cluster. I wanted to test if the API server calls the Decrypt method. So after creating multiple secrets with different DEK seed, I restarted the Kube API Server, expecting the api server to call Decrypt method of my plugin. But the Kube API Server did not call the Decrypt method and I suspect its getting the decrypted DEK Seed from somewhere even after restart and gets cached again.
How do I force the Kube API server to call DEK Seed after restart ?

Solutions I tried: (Which did not work)

  • cachesize: -1 in EncryptionConfiguration
  • Disabled and Enabled EncryptionConfiguration.

Note: Same scenario works fine in Redhat OpenShift

Cluster information:

Kubernetes version: v1.33.1
Installation method: Kind Cluster
Host OS: Mac

2

Fixed this issue. We need to use
docker exec ${CONTAINER} crictl stopp ${POD_ID}
Instead of
kubectl delete pod
crictl stopp bypasses the kubernetes API and stops the container and gets the cache cleared.
On the other hand kubectl delete pod does a graceful shutting down and restarts the pod by preserving the DEK seed cache.

Another way we can clear the DEK Seed cache is by updating the api server manifest in the path /etc/kubernetes/manifests/kube-apiserver.yaml with some annotations. This will trigger a restart that clears the cache as we expect.