Hi, I am looking for a cause for a phenomenon related to apiserver-loopback-client certificate expiration.
Kubernetes: v1.18.x
Cloud in use: bare metal
When deploying a workload through deployment, I get the following error.
Warning ReplicaSetCreateError 28m (x16 over 30m) deployment-controller Failed to create new replica set "prj-rtmhrenlwo-9542-container-68988c865f": Get https://[::1]:6443/api/v1/namespaces/ctc-arc-aiportal/resourcequotas: x509: certificate has expired or is not yet valid
At this point, checking the kube-apiserver log issues the following log.
E0311 08:07:29.218349 1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get",URL:"https://[::1]:6443/api/v1/namespaces/ctc-arc-aiportal/resourcequotas", Err:x509.CertificateInvalidError{Cert:(*x509.Certificate)(0xc012792100), Reason:1, Detail:""}}
I0311 08:07:30.898667 1 log.go:172] http: TLS handshake error from [::1]:41412: remote error: tls: bad certificate
I0311 08:07:30.903607 1 log.go:172] http: TLS handshake error from [::1]:41414: remote error: tls: bad certificate
E0311 08:07:30.903659 1 controller.go:223] unable to sync kubernetes service: Post https://[::1]:6443/api/v1/namespaces: x509: certificate has expired or is not yet valid
apiserver LoopbackClient Server cert expired after 1 year · Issue #86552 · kubernetes/kubernetes · GitHub via issue
We found that the apiserver-loopback-client certificate has a fixed expiration of 1 year.
$ curl --resolve apiserver-loopback-client:6443:172.23.192.117 -k -v https://apiserver-loopback-client:6443/healthz
* Server certificate:
* subject: CN=apiserver-loopback-client@1657082084
* start date: Jul 06 03:34:43 2022 GMT
* expire date: Jul 06 03:34:43 2023 GMT
* common name: apiserver-loopback-client@1657082084
* issuer: CN=apiserver-loopback-client-ca@1657082083
But the question is
The error occurred when the last start time of the container in kube-apiserver was already 9 months old.
Based on the above situation and inferring from the error log, we believe that the error occurred while checking resourcequotas with apiserver-loopback-client in the ReplicaSetCreate event.
From this, I can infer that there have been no events to use apiserver-loopback-client in the 9 months since the certificate expired.
Is what I’m expecting a possible scenario?
What are some of the events that would utilize apiserver-loopback-client?