How to enable auto CSR approval for kubelet server certificates?

General Discussions
1

Greetings!

With a fresh 1.13.3 cluster stood up via Kubernetes The Hard Way plus the additional configuration for TLS bootstrapping, I’m struggling to get auto approval of kubelet server certificates. Client certs are bootstrapping fine and kubelet -> apiserver communication is working smoothly, but apiserver -> kubelet communication is the problem at hand.

Server certificates are being requested, but they are stuck pending manual intervention, and I’ve not been able to divine the RBAC incantation needed to auto-approve server CSRs the same way that client CSRs are.

Here are the CSRs (after having just reinstantiated the cluster):

NAME                                                   AGE   REQUESTOR                      CONDITION
csr-m7rjs                                              4s    system:node:k8s-lab3-worker1   Pending
node-csr-aTpBsagYzYaZYJM6iGMN5AvqzVXATDj1BrmZs_dZCJA   5s    system:bootstrap:ec5591        Approved,Issued

At this point obviously apiserver -> kubelet communication (via kubectl exec or logs) fails. If I manually approve the certificate, things work as expected.

The fact that both client and server CSRs are being issued both leads me to believe the kubelet is configured properly (plus the fact that manually approving makes it go).

The main thing that triggers my spidey sense is the fact that when the apiserver starts up for the first time, I see:

Feb  6 00:14:13 k8s-lab3-controller1[3495]: I0206 00:14:13.697030    3495 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:nodeclient
Feb  6 00:14:13 k8s-lab3-controller1[3495]: I0206 00:14:13.706561    3495 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

The cluster roles for client cert signing are auto-created by the apiserver. But certificatesigningrequests:selfnodeserver is not auto-created. Does this suggest auto approval for server certs isn’t actually implemented or supported?

Well, I’ve created it manually:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]

And then I bind the role to the system:nodes group:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: auto-approve-server-renewals-for-nodes
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
  apiGroup: rbac.authorization.k8s.io

And just to be sure, system:nodes is one of the groups associated with the server CSR:

$ kubectl get csr csr-m7rjs -o template --template {{.spec.groups}}
[system:nodes system:authenticated]

I’ve tried several hours worth of black belt levels of Copying and Pasting from Stack Overflow (with most of the advice really applying to older versions of Kubernetes) to no avail. I’m hoping the brain trust here can spot what I’m doing wrong.

In case it’s relevant, here’s how I’m starting the apiserver (and again this is v1.13.3 so I’m ):

/usr/local/bin/kube-apiserver \
  --advertise-address=172.24.22.168 \
  --allow-privileged=true \
  --anonymous-auth=false \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=10 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --client-ca-file=/etc/kubernetes/pki/ca.crt \
  --enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,AlwaysPullImages,DenyEscalatingExec,SecurityContextDeny,EventRateLimit \
  --admission-control-config-file=/etc/kubernetes/admissionconfig.yaml \
  --enable-bootstrap-token-auth=true \
  --enable-swagger-ui=true \
  --etcd-cafile=/etc/kubernetes/pki/ca.crt \
  --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
  --etcd-servers=https://172.24.22.168:2379 \
  --event-ttl=1h \
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \
  --feature-gates=RotateKubeletServerCertificate=true \
  --insecure-port=0 \
  --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt \
  --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
  --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
  --kubelet-https=true \
  --profiling=false \
  --repair-malformed-updates=false \
  --runtime-config=api/all \
  --service-account-lookup=true \
  --service-account-key-file=/etc/kubernetes/pki/sa.crt  \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
  --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver.key \
  --v=2

(Given RotateKubeletServerCertificate is true by default as of 1.12 I suppose the --feature-gates argument is redundant but I happened to have left it in.)

Many thanks for any help you might be able to offer.

2

It turns out auto approval for server CSRs was removed.

So much for that.