Encryption at Rest: pending question

3 weeks ago I opened an help request for encrypting at rest here: Where can I find the key_id returned by the EncryptRequest procedure call and the key_id returned by the Status ? · Issue #2 · kubernetes/kms · GitHub but I didn’t receive any answers

Cluster information:

Kubernetes version:

root@k8s-eu-1-master:~# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.2", GitCommit:"fc04e732bb3e7198d2fa44efa5457c7c6f8c0f5b", GitTreeState:"clean", BuildDate:"2023-02-22T13:39:03Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.2", GitCommit:"fc04e732bb3e7198d2fa44efa5457c7c6f8c0f5b", GitTreeState:"clean", BuildDate:"2023-02-22T13:32:22Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}

Cloud being used: (put bare-metal if not on a public cloud) : bare-metal Contabo Cloud
Installation method: Host OS: Ubuntu 22.04
CNI and version:

root@k8s-eu-1-master:/etc/cni/net.d# ls -lah
total 16K
drwx------ 2 root root 4.0K Mar 13 17:57 .
drwxr-xr-x 3 root root 4.0K Mar 12 20:11 ..
-rw-r--r-- 1 root root  666 Apr 12 08:51 10-calico.conflist

But :

root@k8s-eu-1-master:/etc/cni/net.d# kubectl calico -h
error: unknown command “calico” for “kubectl”

CRI and version:

 containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-04-12 08:49:12 CEST; 3 weeks 1 day ago
       Docs: https://containerd.io
    Process: 451 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 457 (containerd)
      Tasks: 102
     Memory: 108.0M
        CPU: 9h 22min 6.403s
     CGroup: /system.slice/containerd.service
             ├─457 /usr/bin/containerd
             ├─603 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 33e2e09ecaf2d6db85f3904a874aeba88a030a27690032412fc8b22437bc2504 -address /run/containerd/containerd.sock
             ├─604 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 841163836577533de7c856a3df2ea3c8846b69dbd2e6f707a19845f47bc9c1c0 -address /run/containerd/containerd.sock
             ├─610 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 393aef04fb4d1b189f52c655713fd63348a82b409555bdd4252026b3246f7167 -address /run/containerd/containerd.sock
             ├─633 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id a750c63ff1ff09d90d40eebbaf6656e553e189e6cd76bc121427fea026ae9354 -address /run/containerd/containerd.sock
             ├─929 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id e2cd3ab55059ea5eafddab5597ba198efba7186e4a62ff25b0108eeb1a998f44 -address /run/containerd/containerd.sock
             └─956 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 73b9de69c53f9799e417e15c08f59e2ae5a1366d98d1f62a60cdabc72e422756 -address /run/containerd/containerd.sock

Notice: journal has been rotated since unit was started, output may be incomplete.

  root@k8s-eu-1-master:~# kubectl get all
  NAME                                                 READY   STATUS    RESTARTS         AGE
  pod/arango-kube-arangodb-operator-844d65cdd5-pl4t2   1/1     Running   11 (7d20h ago)   26d
  pod/cluster-agnt-apt6zd1n-75b1c6                     1/1     Running   0                26d
  pod/cluster-agnt-mwchqsli-75b1c6                     1/1     Running   0                26d
  pod/cluster-agnt-uhbsq34m-75b1c6                     1/1     Running   0                26d
  pod/cluster-crdn-2avzgbi1-75b1c6                     1/1     Running   0                26d
  pod/cluster-crdn-ets3nigq-75b1c6                     1/1     Running   0                26d
  pod/cluster-crdn-n3r0x9by-75b1c6                     1/1     Running   0                26d
  pod/cluster-prmr-07wqascu-75b1c6                     1/1     Running   0                26d
  pod/cluster-prmr-kenmxcns-75b1c6                     1/1     Running   0                26d
  pod/cluster-prmr-nsgmx9lg-75b1c6                     1/1     Running   0                26d
  pod/local-storage-4hkdx                              1/1     Running   0                26d
  pod/local-storage-5xhmh                              1/1     Running   0                26d
  pod/local-storage-b654d                              1/1     Running   0                26d
  pod/local-storage-mvndq                              1/1     Running   0                26d
  pod/local-storage-rk85s                              1/1     Running   0                26d
  NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  service/arango-kube-arangodb-operator   ClusterIP     <none>        8528/TCP,8628/TCP,8728/TCP   26d
  service/cluster                         ClusterIP    <none>        8529/TCP                     26d
  service/cluster-agent-apt6zd1n          ClusterIP    <none>        8529/TCP                     26d
  service/cluster-agent-mwchqsli          ClusterIP   <none>        8529/TCP                     26d
  service/cluster-agent-uhbsq34m          ClusterIP    <none>        8529/TCP                     26d
  service/cluster-coordinator-2avzgbi1    ClusterIP     <none>        8529/TCP                     26d
  service/cluster-coordinator-ets3nigq    ClusterIP    <none>        8529/TCP                     26d
  service/cluster-coordinator-n3r0x9by    ClusterIP    <none>        8529/TCP                     26d
  service/cluster-dbserver-07wqascu       ClusterIP   <none>        8529/TCP                     26d
  service/cluster-dbserver-kenmxcns       ClusterIP     <none>        8529/TCP                     26d
  service/cluster-dbserver-nsgmx9lg       ClusterIP    <none>        8529/TCP                     26d
  service/cluster-ea                      NodePort     <none>        8529:32527/TCP               26d
  service/cluster-int                     ClusterIP   None             <none>        8529/TCP                     26d
  service/kubernetes                      ClusterIP        <none>        443/TCP                      52d
  service/local-storage                   ClusterIP     <none>        8929/TCP                     26d
  daemonset.apps/local-storage   5         5         5       5            5           <none>          26d
  NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/arango-kube-arangodb-operator   1/1     1            1           26d
  NAME                                                       DESIRED   CURRENT   READY   AGE
  replicaset.apps/arango-kube-arangodb-operator-844d65cdd5   1         1         1       26d

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

arangoDeployment.yaml :

apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
  name: "cluster"
  mode: Cluster
        storageClassName: my-local-ssd
        - ReadWriteOnce
            storage: 1Gi
        volumeMode: Filesystem
        storageClassName: my-local-ssd
        - ReadWriteOnce
            storage: 1Gi
        volumeMode: Filesystem

arangoLocalStorage.yaml :

  apiVersion: "storage.arangodb.com/v1alpha"
    kind: "ArangoLocalStorage"
      name: "local-storage"
        name: my-local-ssd
        isDefault: true
      - /mnt/big-ssd-disk

encryptionConfiguration.yaml :

# https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration
# https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/
# https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/#implementing-a-kms-plugin

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
  - resources:
      - secrets
      - configmaps
      - pandas.awesome.bears.example

      - kms:
            apiVersion: v2
            name: myKmsPlugin
            endpoint: unix:///tmp/socketfile.sock
            cachesize: 100
            timeout: 3s

In Using a KMS provider for data encryption | Kubernetes :

The API server considers the key_id returned from the Status procedure call to be authoritative. Thus, a change to this value signals to the API server that the remote KEK has changed, and data encrypted with the old KEK should be marked stale when a no-op write is performed (as described below). If an EncryptRequest procedure call returns a key_id that is different from Status, the response is thrown away and the plugin is considered unhealthy. Thus implementations must guarantee that the key_id returned from Status will be the same as the one returned by EncryptRequest. Furthermore, plugins must ensure that the key_id is stable and does not flip-flop between values (i.e. during a remote KEK rotation)

The StatusResponse struct kms/api.pb.go at master · kubernetes/kms · GitHub has a field KeyId , but the EncryptRequest struct has Uid field

Where can I find the key_id returned by the EncryptRequest procedure call, in order to compare this key_id with the one returned by the Status ?

By the way, with :

    func (s *Server) getStatus(ctx context.Context, req *kms.StatusRequest) (*kms.StatusResponse) {
            resp, _ := s.Status(ctx, req)
            return resp
    func (s *Server) Encrypt(ctx context.Context, req *kms.EncryptRequest) (*kms.EncryptResponse) {
            UId := req.Uid
            log.Println("UId-EncryptRequest: ", UId)
            statusReq := s.StatusRequest{
                    XXX_NoUnkeyedLiteral: {},
                    XXX_unrecognized: []byte,
                    XXX_sizecache: 10,
            statusKId, _ := s.Status(ctx, statusReq)
            if (statusKId.KeyId != UId ) {
                    log.Printf("Watch out!")
                    return nil 
            resp := s.Encrypt(ctx, req)
            log.Printf("Encrypt-resp: ", resp)
            return resp

I get this other error s.StatusRequest is not a type :

    root@k8s-eu-1-master:~/kms/apis/v2/api# go run ./server/main.go 
    # command-line-arguments
    server/main.go:58:24: s.StatusRequest is not a type
    server/main.go:59:39: missing type in composite literal

but here the StatusRequest struct type is defined : kms/api.pb.go at a38ec9832062d14d72f0f4cee8d6904a8be5fbee · kubernetes/kms · GitHub

Where can I find the key_id returned by the EncryptRequest procedure call and the key_id returned by the Status ?
How to correctly setup the *Encryption at Rest* ?

I’d email the sig auth list with your question. That repo is what’s known as a staging repo, essentially a directory that’s kept in sync with a subdirectory in the main kubernetes repo, and it isn’t necessarily looked at for triage purposes as most issues are opened up directly against kubernetes/kubernetes.

If most issues are opened up directly against GitHub - kubernetes/kubernetes: Production-Grade Container Scheduling and Management , would it be more effective in open an issue there?

Clicking on “Support Request” leads to this forum :https://discuss.kubernetes.io/

So… where should I land off to be heard and get hints and help from my issue with Encryption at Rest ?

It’s not related to an actual issue, and is considered more of a support request (often closed in kubernetes/kubernetes and directed here), you could also join the sig auth mailing slack channel and ask there.

Clicking on “Support Request” leads to this forum :https://discuss.kubernetes.io/

How to connect to the sig auth mailing slack channel ?

All their communication outlets are listed on their sig auth page linked above.

These are the Kubernetes Channels I see inDoK.community in Slack:

sig-auth is on the Kubernetes slack - you can join here: slack.k8s.io

1 Like

Thank you. Just joined the channel

Hopefully this will help

1 Like

I described the issue I’m experiencing here: Slack in the #sig-auth channel

1 Like