How to secure Kubernetes secrets on bare metal?

General Discussions
,
1

We are planning to run some of our software in a Kubernetes cluster running on bare metal (i.e. not using a cloud provider). We need to provide some secret data to our microservices (e.g. API keys, private keys), and want to ensure these secrets are stored and retrieved as securely as possible.

The Kubernetes Secrets documentation provides various recommendations:

  • Enable encryption at rest

    • The documentation for this states:

      Caution: Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption. Please use kms provider for additional security.

      …

      Encrypting Secrets with a locally managed key [i.e. not kms] protects against an etcd compromise, but it fails to protect against a host compromise. Since the encryption keys are stored on the host in the EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption keys.

  • Use an external secret store provider, e.g. Vault provider

    • This seems to be an alternative to using native Kubernetes secrets, and outsources secret management to an external component.

My thought process is:

  1. Using the kms provider seems to require us to use a Cloud provider - which isn’t an option - or develop our own - and I assume “rolling our own” isn’t recommended for this sort of thing. So we have to use an external secret store provider, of which Vault is the only supported provider.

  2. Vault requires the Pod to authenticate using its service account, which boils down to a JWT minted using a private key stored on a control plane node.

  3. This private key is no more protected than a static secret defined in an EncryptionConfig file, assuming our control plane is separate from our worker nodes.

  4. So, my question:

    What do we gain from using an external secret store, over using a static secret in an EncryptionConfig file?

    Is it just security theatre? Or is there a flaw in the reasoning above?

    Alternatively, is there a better way to manage Kubernetes secrets when running on bare metal?

2

There are a couple of differences. The first difference is in the availability of the key. Having the encryption key stored on the same drive as the encrypted data means that if the physical server was stolen then the person who took it would have both the data and the key to decrypt the data – not particularly secure. If you use a good KMS then the key is stored elsewhere.

Even if you self-host the KMS (such as your own Vault server) generally they will have encrypted storage which must be unsealed either using an external KMS of some kind or else using a set of “unseal keys” which have to be used to unlock it each time the service restarts, so the encryption key is stored in memory but never unencrypted on disk.

The other difference is that with a KMS it can do automatic key rotation; when using aesgcm for example you need to rotate the key after 200K writes which it should do automatically when using a key server. If you don’t rotate the key then it can lead to security vulnerabilities, though the specifics are beyond my expertise =]