MicroK8s API Server TLS Certificate Failure - "failed to find any PEM data in certificate input"

General Discussions microk8s
1

Problem Description:
Our MicroK8s API server crashes immediately upon startup with TLS certificate validation errors. The kubelet service cannot establish secure connections, resulting in complete cluster failure.

Error Message:

E0709 06:12:03.421939 1141940 server.go:180] “Failed to listen and serve” err=“tls: failed to find any PEM data in certificate input” systemd[1]: snap.microk8s.daemon-kubelite.service: Main process exited, code=exited, status=1/FAILURE

Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg microk8s.daemon-kubelite[1141940]: I0709 06:12:03.409669 1141940 kubelet.go:816] “Not starting ClusterTrustBundle informer because we are in static kubelet mode” Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg microk8s.daemon-kubelite[1141940]: I0709 06:12:03.410378 1141940 server.go:1272] “Started kubelet” Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg microk8s.daemon-kubelite[1141940]: I0709 06:12:03.412876 1141940 fs_resource_analyzer.go:67] “Starting FS ResourceAnalyzer” Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg microk8s.daemon-kubelite[1141940]: I0709 06:12:03.419683 1141940 server.go:163] “Starting to listen” address=“0.0.0.0” port=10250 Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg microk8s.daemon-kubelite[1141940]: I0709 06:12:03.421045 1141940 server.go:444] “Adding debug handlers to kubelet server” Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg microk8s.daemon-kubelite[1141940]: E0709 06:12:03.421939 1141940 server.go:180] “Failed to listen and serve” err=“tls: failed to find any PEM data in certificate input” Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg systemd[1]: snap.microk8s.daemon-kubelite.service: Main process exited, code=exited, status=1/FAILURE Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg systemd[1]: snap.microk8s.daemon-kubelite.service: Failed with result ‘exit-code’. Jul 09 06:12:03 tmds-develop-svcgw.services.nonprod.c0.rhbbank.com.sg systemd[1]: snap.microk8s.daemon-kubelite.service: Consumed 3.047s CPU time.

Comprehensive Troubleshooting Completed:

  1. Certificate Refresh (Failed):

    microk8s refresh-certs --cert ca.crt
microk8s refresh-certs --cert server.crt
microk8s refresh-certs --cert front-proxy-client.crt

  2. Service Restart (Failed):
    microk8s stop

    microk8s start

  3. CNI Reconfiguration (Failed):
    kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml

  4. HA Cluster Reset (Failed):
    microk8s disable ha-cluster

    microk8s enable ha-cluster

**Aditional Detailes:**

OS: Rocky Linux 9.5

version of microk8s: v1.30.11 revision 7966

Runs on AWS EC2, instance type: c5.2xlarge

Questions:

  1. Certificate Corruption: What could cause MicroK8s to lose PEM certificate data, and how can we recover?

  2. Certificate Storage: Where does MicroK8s store TLS certificates, and how can we verify their integrity?

  3. Manual Recovery: Is there a way to manually regenerate or restore the TLS certificates?

  4. Diagnostic Commands: What additional diagnostic commands can help identify the exact certificate issue?

Thank you!