Healthceck-client.crt: no such file or directory trying to backup etcd

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.20
Cloud being used: VMWare/CentOS
Installation method: kubeadm init
Host OS: CentOS 8

I created a job to backup etcd. It mounts the /etc/kubernetes/pki, but getting an error that it cannot find the files even though they exist. Here is the yaml:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: backup
namespace: kube-system
spec:

activeDeadlineSeconds: 100

schedule: “*/1 * * * *”
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
# Same image as in /etc/kubernetes/manifests/etcd.yaml
image: k8s.gcr.io/etcd:3.4.13-0
env:
- name: ETCDCTL_API
value: “3”
command: ["/bin/sh"]
args: ["-c", “etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-snapshot.db”]
volumeMounts:
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
readOnly: true
- mountPath: /backup
name: backup
restartPolicy: OnFailure
hostNetwork: true
volumes:
- name: etcd-certs
hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
- name: backup
hostPath:
path: /data/etcd_backup
type: DirectoryOrCreate

File is in expected location:

$ ls -l /etc/kubernetes/pki/etcd
total 32
-rw------- 1 root root 1058 Dec 30 17:57 ca.crt
-rw------- 1 root root 1675 Dec 30 17:57 ca.key
-rw-r–r-- 1 root root 1139 Jan 15 08:34 healthcheck-client.crt
-rw------- 1 root root 1675 Jan 15 08:34 healthcheck-client.key
-rw-r–r-- 1 root root 1237 Jan 15 08:34 peer.crt
-rw------- 1 root root 1675 Jan 15 08:34 peer.key
-rw-r–r-- 1 root root 1237 Jan 15 08:34 server.crt
-rw------- 1 root root 1679 Jan 15 08:34 server.key