FailedMount. MountVolume.SetUp failed for volume "cert"

General Discussions
1

Good day community,

Hope someone could help me to resolve the issue that I’ve faced :blush:

I have created a K3S cluster using Rancher and on top of it installed cert-manager (it’s a requirement for actions-runner-controller).
Then, as the second step I installed actions-runner-controller using Helm in the namespace ‘actions-runner-system’.

However, I see that pod is stuck:

$ k --kubeconfig .kube\config.yaml get pods -n actions-runner-system

NAME                                         READY   STATUS              RESTARTS   AGE
actions-runner-controller-5b8c8c8c8d-8cp9p   0/2     ContainerCreating   0          14m

I checked that pod:

$ k --kubeconfig .kube\config.yaml describe pod actions-runner-controller-5455b9f4d6-2nvbt -n actions-runner-system

Name:             actions-runner-controller-5455b9f4d6-2nvbt
Namespace:        actions-runner-system
Priority:         0
Service Account:  actions-runner-controller
Node:             host1.saitama.net/10.158.146.19
Start Time:       Wed, 15 Mar 2023 13:24:35 +0100
Labels:           app.kubernetes.io/instance=actions-runner-controller
                  app.kubernetes.io/name=actions-runner-controller
                  pod-template-hash=5455b9f4d6
Annotations:      cattle.io/timestamp: 2023-03-15T12:24:35Z
Status:           Pending
IP:
IPs:              <none>
Controlled By:    ReplicaSet/actions-runner-controller-5455b9f4d6
Containers:
  manager:
    Container ID:
    Image:         summerwind/actions-runner-controller:v0.27.0
    Image ID:
    Port:          9443/TCP
    Host Port:     0/TCP
    Command:
      /manager
    Args:
      --metrics-addr=127.0.0.1:8080
      --enable-leader-election
      --port=9443
      --sync-period=1m
      --default-scale-down-delay=10m
      --docker-image=docker:dind
      --runner-image=summerwind/actions-runner:latest
      --log-format=text
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      GITHUB_TOKEN:                <set to the key 'github_token' in secret 'controller-manager'>                Optional: true
      GITHUB_APP_ID:               <set to the key 'github_app_id' in secret 'controller-manager'>               Optional: true
      GITHUB_APP_INSTALLATION_ID:  <set to the key 'github_app_installation_id' in secret 'controller-manager'>  Optional: true
      GITHUB_APP_PRIVATE_KEY:      <set to the key 'github_app_private_key' in secret 'controller-manager'>      Optional: true
      GITHUB_BASICAUTH_PASSWORD:   <set to the key 'github_basicauth_password' in secret 'controller-manager'>   Optional: true
    Mounts:
      /etc/actions-runner-controller from secret (ro)
      /tmp from tmp (rw)
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qdk54 (ro)
  kube-rbac-proxy:
    Container ID:
    Image:         quay.io/brancz/kube-rbac-proxy:v0.13.1
    Image ID:
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qdk54 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  controller-manager
    Optional:    false
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  actions-runner-controller-serving-cert
    Optional:    false
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-qdk54:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    3m24s                default-scheduler  Successfully assigned actions-runner-system/actions-runner-controller-5455b9f4d6-2nvbt to host1.saitama.net
  Warning  FailedMount  81s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[secret tmp cert kube-api-access-qdk54]: timed out waiting for the condition
  Warning  FailedMount  76s (x9 over 3m24s)  kubelet            MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found

There are some Warning messages, but I can’t get what’s wrong. Is this issue related to the ‘cert-manager’ or it’s a problem with the ‘actions-runner-controller’ itself? Or maybe there is some issue in the K3S (checked its status: active(running))?

I have tested everything in minikube on my local pc and there everything works fine. However, it’s worth mentioning that the K3S cluster is running in an isolated environment, but probably that shouldn’t be the case.

Cluster information:

Kubernetes version: v1.24.7+k3s1
Installation method: Rancher
Host OS: RHEL8


$ helm --kubeconfig .kube\config.yaml list -n actions-runner-system

NAME                            NAMESPACE               REVISION        UPDATED                                 STATUS  CHART                                   APP VERSION
actions-runner-controller       actions-runner-system   1               2023-03-15 10:28:11.3575084 +0100 CET   failed  actions-runner-controller-0.22.0        0.27.0

Thanks in advance,

2

I am currently having the exact same issue, working with minikube also… were you able to fix the error ?

3

Just leaving this in case someone stumbles across it.
I was having the same issue trying to run CloudNativePG operator on K3s. As mentioned here, I had to enable the NamespaceLifecycle in my K3s server. Might apply to minikube too.