Unable to attach or mount volumes: unmounted volumes=[code-storage]

General Discussions
1

I set up my volume and volume claims. I have my namespace and in there a deployment with php-fpm, nginx and mysql for local testing on Minikube

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: smt-local
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: laravel
          image: smart48/smt-laravel:latest
          ports:
            - containerPort: 9000
          resources:
            requests:
              cpu: 250m
            limits:
              cpu: 500m
          volumeMounts:
          - name: code-storage
            mountPath: /data/smtapp
          # examples
          # - name: orientdb-config
          #   mountPath: /data/orientdb/config
          # - name: orientdb-databases
          #   mountPath: /data/orientdb/databases 
          # - name: orientdb-backup
          #   mountPath: /data/orientdb/backup
        - name: nginx
          image: smart48/smt-nginx:latest
          ports:
            - containerPort: 9376
          volumeMounts:
          # example
          - name: nginx-config
            mountPath: /data/nginx/config
        - name: laravel-horizon
          image: smart48/smt-laravel-horizon:latest
          # not sure if we need the port and volume here
          # ports:
          #   - containerPort: 9377
          # volumeMounts:
          #   - name: code
          #     mountPath: /code
          command: ["/usr/local/bin/php", "artisan", "horizon"]
          lifecycle:
            preStop:
              exec:
                command: ["/usr/local/bin/php", "artisan", "horizon:terminate"]
        - name: mysql
          image: smart48/smt-mysql:latest
          imagePullPolicy: Never
          env:
            - name: mysql_ROOT_PASSWORD
              value: .sweetpwd.
            - name: mysql_DATABASE
              value: my_db
            - name: mysql_USER
              value: db_user
            - name: mysql_PASSWORD
              value: .mypwd
          args: ["--default-authentication-plugin=mysql_native_password"]
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: code-storage
          persistentVolumeClaim:
            claimName: code-pv-claim
        - name: nginx-config
          persistentVolumeClaim:
            claimName: code-pv-claim
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim

However the data directories (empty atm and nothing added to /data) cannot be mounted:

kubectl describe pod app-67fc69b8d8-nkr6d -n smt-local 
Name:           app-67fc69b8d8-nkr6d
Namespace:      smt-local
Priority:       0
Node:           minikube/192.168.64.5
Start Time:     Fri, 04 Dec 2020 13:54:56 +0700
Labels:         app=web
                pod-template-hash=67fc69b8d8
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/app-67fc69b8d8
Containers:
  laravel:
    Container ID:   
    Image:          smart48/smt-laravel:latest
    Image ID:       
    Port:           9000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:  500m
    Requests:
      cpu:        250m
    Environment:  <none>
    Mounts:
      /data/smtapp from code-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cp4nw (ro)
  nginx:
    Container ID:   
    Image:          smart48/smt-nginx:latest
    Image ID:       
    Port:           9376/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data/nginx/config from nginx-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cp4nw (ro)
  laravel-horizon:
    Container ID:  
    Image:         smart48/smt-laravel-horizon:latest
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/php
      artisan
      horizon
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cp4nw (ro)
  mysql:
    Container ID:  
    Image:         smart48/smt-mysql:latest
    Image ID:      
    Port:          3306/TCP
    Host Port:     0/TCP
    Args:
      --default-authentication-plugin=mysql_native_password
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      mysql_ROOT_PASSWORD:  .sweetpwd.
      mysql_DATABASE:       my_db
      mysql_USER:           db_user
      mysql_PASSWORD:       .mypwd
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cp4nw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  code-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  code-pv-claim
    ReadOnly:   false
  nginx-config:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  code-pv-claim
    ReadOnly:   false
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-cp4nw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-cp4nw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age    From               Message
  ----     ------       ----   ----               -------
  Normal   Scheduled    2m35s  default-scheduler  Successfully assigned smt-local/app-67fc69b8d8-nkr6d to minikube
  Warning  FailedMount  31s    kubelet            Unable to attach or mount volumes: unmounted volumes=[code-storage], unattached volumes=[code-storage default-token-cp4nw nginx-config mysql-persistent-storage]: timed out waiting for the condition

Here code pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: code-pv
  namespace: smt-local
spec:
  accessModes:
      - ReadWriteOnce
  capacity:
    storage: 20Gi
  hostPath:
    # minikube is configured to persist files stored under /data/ and a few 
    # other directories such as /var/lib/minikube in the _vm_
    path: /data

and code pvc

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: code-pv-claim
  namespace: smt-local
  labels:
    type: code
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Why would I get

Warning  FailedMount  31s    kubelet            Unable to attach or mount volumes: unmounted volumes=[code-storage], unattached volumes=[code-storage default-token-cp4nw nginx-config mysql-persistent-storage]: timed out waiting for the condition

And a little later these events:

vents:
  Type     Reason       Age                     From     Message
  ----     ------       ----                    ----     -------
  Warning  FailedMount  9m54s (x42 over 3h8m)   kubelet  Unable to attach or mount volumes: unmounted volumes=[nginx-config], unattached volumes=[code-storage default-token-cp4nw nginx-config mysql-persistent-storage]: timed out waiting for the condition
  Warning  FailedMount  5m20s (x22 over 3h31m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[nginx-config], unattached volumes=[default-token-cp4nw nginx-config mysql-persistent-storage code-storage]: timed out waiting for the condition
  Warning  FailedMount  50s (x15 over 166m)     kubelet  Unable to attach or mount volumes: unmounted volumes=[nginx-config], unattached volumes=[mysql-persistent-storage code-storage default-token-cp4nw nginx-config]: timed out waiting for the condition
2

When I check the code volume claim I see that the app deployments are mounted

kubectl describe pvc code-pv-claim  -n smt-local 
Name:          code-pv-claim
Namespace:     smt-local
StorageClass:  standard
Status:        Bound
Volume:        pvc-46b0a443-883a-4e97-af27-beeef943bcdb
Labels:        type=code
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    app-67fc69b8d8-r2zr7
               app-67fc69b8d8-r2zr7
               app-67fc69b8d8-wpdt2
               app-67fc69b8d8-wpdt2
               app-87494ff59-5zrg2
               app-87494ff59-5zrg2
               app-87494ff59-s62x7
               app-87494ff59-s62x7
Events:        <none>

And when I check the pod and its replications:

➜  smt-deploy git:(main) ✗ kubectl get pods -n smt-local                   
NAME                   READY   STATUS              RESTARTS   AGE
app-67fc69b8d8-r2zr7   0/4     ContainerCreating   0          3h52m
app-67fc69b8d8-wpdt2   0/4     ContainerCreating   0          3h52m
app-87494ff59-5zrg2    0/4     ContainerCreating   0          170m
app-87494ff59-s62x7    0/4     ContainerCreating   0          170m

I see all there as well. So if these pods mounted the code pvc that is a good thing… so why the error with the VolumeMounts?

Well the persistent volume code-pvc seems quiet as in not claimed:

kubectl get pv -n smt-local                    
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                      STORAGECLASS   REASON   AGE
code-pv                                    20Gi       RWO            Retain           Available                                                      4h3m
pvc-46b0a443-883a-4e97-af27-beeef943bcdb   10Gi       RWO            Delete           Bound       smt-local/code-pv-claim    standard                4h3m
pvc-515fa8d5-0855-44da-967d-8ca3459a2a71   2Gi        RWO            Delete           Bound       smt-local/redis-pv-claim   standard                4h3m
pvc-8c4612c6-c98f-425f-acb6-8fe652442dd9   2Gi        RWO            Delete           Bound       smt-local/mysql-pv-claim   standard                4h3m

Whereas the claims are made to other pvcs.

And description of the code-pv also shows nothing is happening :

kubectl describe pv code-pv   -n smt-local
Name:            code-pv
Labels:          <none>
Annotations:     <none>
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        20Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /data
    HostPathType:  
Events:            <none>

So perhaps My Persistent Volume claims need to focus on the Vm’s HostPath as well?

3

I decided to mount the data path this time. Did not realize this was necessary but it of course is in a virtualbox / hypervisor. So ran a

minikube start --mount-string="$HOME/code/smt-data:/data"

This however does not seem to change the issues with pod having unattached volumes:

  Warning  FailedMount  12s    kubelet            Unable to attach or mount volumes: unmounted volumes=[nginx-config], unattached volumes=[mysql-persistent-storage code-storage default-token-cp4nw nginx-config]: timed out waiting for the condition

So I must be missing something still why the volumes cannot be mounted.

created
➜  smt-deploy git:(main) kubectl get pv                          
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                      STORAGECLASS   REASON   AGE
code-pv                                    20Gi       RWO            Retain           Available                                                      9s
pvc-02e0570b-b569-4c84-8294-aa4999377743   2Gi        RWO            Delete           Bound       smt-local/redis-pv-claim   standard                5m8s
pvc-1e67c1d5-36b0-4ff4-8fd5-c2fde1b3ee33   10Gi       RWO            Delete           Bound       smt-local/code-pv-claim    standard                5m17s
pvc-3e60de71-3a69-4d7d-a962-2ffd8d3b2f10   2Gi        RWO            Delete           Bound       smt-local/mysql-pv-claim   standard                5m11s
pvc-46b0a443-883a-4e97-af27-beeef943bcdb   10Gi       RWO            Delete           Released    smt-local/code-pv-claim    standard                19h
pvc-515fa8d5-0855-44da-967d-8ca3459a2a71   2Gi        RWO            Delete           Released    smt-local/redis-pv-claim   standard                19h
pvc-8c4612c6-c98f-425f-acb6-8fe652442dd9   2Gi        RWO            Delete           Released    smt-local/mysql-pv-claim   standard                19h
4

I created all the directories inside the hosts directory as referred to in deployment but that does not seem to matter either. I still see

  Warning  FailedMount  25s    kubelet            Unable to attach or mount volumes: unmounted volumes=[nginx-config], unattached volumes=[code-storage default-token-bhtvz nginx-config mysql-persistent-storage]: timed out waiting for the condition

Why timed-out or waiting for what condition? How can I find out about this issue?

5

Changed the claim and added Volume name

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: code-pv-claim
  namespace: smt-local
  labels:
    type: code
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ""
  volumeName: pv0001
  resources:
    requests:
      storage: 5Gi

Now I see

21s         Warning   FailedScheduling    pod/app-69dd7cd86f-4cqjq               0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
6

Once I had given Nginx its own volume and or volume claim I could both continue with a hostPath setup and a non hostPath standard storage class setup. Will probably continue with a non hostPath setup as that saves me some work on separate volumes and generating dirs in the VM. But we shall see. Other errors to work on now.