Microk8s apiserver-kicker is refreshing front-proxy-client.cert daily, restarting the node despite cert being valid for a year

General Discussions microk8s
1

This started happening awhile ago but I’ve only noticed recently. Benefits of it being a homelab with low personal usage. Below is an excerpt of logs taken from journalctl -n 1000 -u snap.microk8s.daemon-apiserver-kicker

Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[7373]: Signature ok
Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[7373]: subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[7373]: Getting CA Private Key
Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[7409]: Signature ok
Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[7409]: subject=CN = front-proxy-client
Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[7409]: Getting CA Private Key
Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Restarting the cluster-agent
Dec 28 11:04:22 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Reconfiguring the kube-apiserver
Dec 28 11:04:22 corvus sudo[8302]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/s>
Dec 28 11:04:22 corvus sudo[8302]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 28 11:04:22 corvus sudo[8302]: pam_unix(sudo:session): session closed for user root
Dec 28 11:04:22 corvus sudo[8308]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/s>
Dec 28 11:04:22 corvus sudo[8308]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 28 11:04:22 corvus sudo[8308]: pam_unix(sudo:session): session closed for user root
Dec 28 11:04:24 corvus sudo[8726]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/s>
Dec 28 11:04:24 corvus sudo[8726]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 28 11:04:24 corvus sudo[8726]: pam_unix(sudo:session): session closed for user root
Dec 28 11:05:15 corvus microk8s.daemon-apiserver-kicker[32935]: Signature ok
Dec 28 11:05:15 corvus microk8s.daemon-apiserver-kicker[32935]: subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Dec 28 11:05:15 corvus microk8s.daemon-apiserver-kicker[32935]: Getting CA Private Key
Dec 28 11:05:15 corvus microk8s.daemon-apiserver-kicker[32946]: Signature ok
Dec 28 11:05:15 corvus microk8s.daemon-apiserver-kicker[32946]: subject=CN = front-proxy-client
Dec 28 11:05:15 corvus microk8s.daemon-apiserver-kicker[32946]: Getting CA Private Key
Dec 28 11:05:15 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Restarting the cluster-agent
Dec 28 11:05:16 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Reconfiguring the kube-apiserver
Dec 28 11:05:16 corvus sudo[33025]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/>
Dec 28 11:05:16 corvus sudo[33025]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 28 11:05:16 corvus sudo[33025]: pam_unix(sudo:session): session closed for user root
Dec 28 11:05:16 corvus sudo[33031]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/>
Dec 28 11:05:16 corvus sudo[33031]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 28 11:05:16 corvus sudo[33031]: pam_unix(sudo:session): session closed for user root
Dec 28 11:05:27 corvus sudo[35046]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/>
Dec 28 11:05:27 corvus sudo[35046]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 28 11:05:27 corvus sudo[35046]: pam_unix(sudo:session): session closed for user root
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1838624]: Signature ok
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1838624]: subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1838624]: Getting CA Private Key
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1838635]: Signature ok
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1838635]: subject=CN = front-proxy-client
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1838635]: Getting CA Private Key
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Restarting the cluster-agent
Dec 29 10:56:46 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Reconfiguring the kube-apiserver
Dec 29 10:56:46 corvus sudo[1838713]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin>
Dec 29 10:56:46 corvus sudo[1838713]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 29 10:56:46 corvus sudo[1838713]: pam_unix(sudo:session): session closed for user root
Dec 29 10:56:46 corvus sudo[1838719]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin>
Dec 29 10:56:46 corvus sudo[1838719]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 29 10:56:46 corvus sudo[1838719]: pam_unix(sudo:session): session closed for user root
Dec 29 10:56:58 corvus sudo[1841392]: root : PWD=/var/snap/microk8s/6364 ; USER=root ; ENV=PATH=/snap/microk8s/6364/usr/bin:/snap/microk8s/6364/bin:/snap/microk8s/6364/usr/sbin:/snap/microk8s/6364/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin>
Dec 29 10:56:58 corvus sudo[1841392]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Dec 29 10:56:58 corvus sudo[1841392]: pam_unix(sudo:session): session closed for user root
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[3166435]: Signature ok
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[3166435]: subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[3166435]: Getting CA Private Key
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[3166446]: Signature ok
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[3166446]: subject=CN = front-proxy-client
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[3166446]: Getting CA Private Key
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Restarting the cluster-agent
Dec 30 10:48:10 corvus microk8s.daemon-apiserver-kicker[1047]: cert change detected. Reconfiguring the kube-apiserver

Not sure what to do, any advice is welcome.

2

Certificate reissues occur when there is a network change on the host system. This way k8s will present valid certificates to all available interfaces even if they change. If you do not want this behavior you can touch /var/snap/microk8s/current/var/lock/no-cert-reissue. Have a look at microk8s/microk8s-resources/wrappers/apiservice-kicker at master · canonical/microk8s · GitHub

3

Thank you kjackal, I ran into How to disable apiserver-kicker? (And why shouldn't I?) · Issue #2790 · canonical/microk8s · GitHub so for now I’ve added --advertise-address 0.0.0.0 to the kube-apiserver args. From your link it seems this will also do what I need in the sense of stopping the kicker. It’s just odd it seems to restart at (roughly) the same time every day. I don’t think there would be anything running on the host that would change the network, it’s a server wired directly to a router with a static IP address assigned.