How to disable token based authentication and use certificates instead (hardening k8s)

General Discussions microk8s
1

Hello everybody,

We getting close to going to production with our new clusters based on Microk8s.
Currently we’re hardening the clusters according to the CIS benchmark: https://github.com/aquasecurity/kube-bench

Right now we’re looking at disabling token based authentication, which is enabled by default in Microk8s.

When I remove the kube-apiserver argument:
--token-auth-file=${SNAP_DATA}/credentials/known_tokens.csv
and optionally delete the known_tokens.csv file then token based access no longer works.
So far so good.

I could also create new users that use certificates according to this documentation:
https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#normal-user
although I expect to use AD-integration via oauth2-proxy instead for normal users.

So, my questions are these:

  • When disabling tokens, what about the existing users defined in known_tokens.csv?
  • Do I try to update the admin user so it authenticates via certificate instead?
    • If yes, how?
    • If no, should the clusterrole be deleted? I guess it can’t be used anymore if so.
    • Should I use the more commonly used cluster-admin for a new admin user?
  • and what about the remaining entries in the file. Will proxy, controller-manager, kubelet and scheduler still work or should their authentication be updated to use certificates?
    • Kubelet at least must use certificates via the setting the kubelet arguments: –tls-cert-file and –tls-private-key-file

Btw: I’ve created this issue to have the default installation be more CIS compliant by default:
https://github.com/ubuntu/microk8s/issues/2517

Thanks in advance for any insight!
Best regards
Jesper Berggren

2

Well, in the lack of responses, here’s our solution (if anyone else is interested):

Setup TLS bootstrapping for kubelet as described in: TLS bootstrapping | Kubernetes
This creates a bootstrap that’ll generate client certificates for kubelet.
Remember to set the following arguments in kubelet config:

--rotate-certificates=true
--rotate-server-certificates=true
--tls-cert-file=${SNAP_DATA}/certs/kubelet-server-current.pem
--tls-private-key-file=${SNAP_DATA}/certs/kubelet-server-current.pem

and set this in kube-apiserver:
--kubelet-certificate-authority=${SNAP_DATA}/certs/ca.crt

Since bootstrapping of certificates is not (yet?) possible for scheduler, kube-controller and proxy then manually create certificates for them including the admin user:

cd /var/snap/microk8s/current/certs

openssl genrsa -out admin.key 2048
openssl req -new -key admin.key -subj "/CN=admin/O=system:masters" -out admin.csr
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -CAcreateserial  -out admin.crt -days 1000

openssl genrsa -out kube-controller-manager.key 2048
openssl req -new -key kube-controller-manager.key -subj "/CN=system:kube-controller-manager" -out kube-controller-manager.csr
openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out kube-controller-manager.crt -days 1000

openssl genrsa -out kube-proxy.key 2048
openssl req -new -key kube-proxy.key -subj "/CN=system:kube-proxy" -out kube-proxy.csr
openssl x509 -req -in kube-proxy.csr -CA ca.crt -CAkey ca.key -CAcreateserial  -out kube-proxy.crt -days 1000

openssl genrsa -out kube-scheduler.key 2048
openssl req -new -key kube-scheduler.key -subj "/CN=system:kube-scheduler" -out kube-scheduler.csr
openssl x509 -req -in kube-scheduler.csr -CA ca.crt -CAkey ca.key -CAcreateserial  -out kube-scheduler.crt -days 1000

Replace tokens in config files (client.config, controller.config, proxy.config and scheduler.config):
cd /var/snap/microk8s/current/credentials
Replace last line containing token with (example for admin, which also could be done in ~/.kube/config):

users:
- name: admin
  user:
    client-certificate: /var/snap/microk8s/current/certs/admin.crt
    client-key: /var/snap/microk8s/current/certs/admin.key

Disable token based access:
vi /var/snap/microk8s/current/args/kube-apiserver
Delete or comment out this line:
--token-auth-file=${SNAP_DATA}/credentials/known_tokens.csv

Restart cluster and you’re done.

Hope that helps somebody :slight_smile:

Best regards
Jesper Berggren

1 Like
3

Thank you so much! You helped me a lot!