MountVolume.NewMounter initialization failed for volume "gcxi-share-pv-local" : path "/opt/app/gencx/gencxshared/gcxi/shared/" does not exist

General Discussions
1

Hello
I am very new to K8s and am trying to deploy a solution for a reporting app. Seems the application deployment succeeds till it tries to create pods and gets stuck at Containercreation.

When i look at pod logs i see its trying to mount volume but fails saying path does not exist.
here are the logs

[ ~]$ kubectl describe pod gcxi-0 -n gcxi

Name: gcxi-0

Namespace: gcxi

Priority: 0

Node: slcgensysl13tv/1.2.4.4

Start Time: Mon, 01 Mar 2021 13:57:55 -0700

Labels: aadpodidbinding=gcxi-identity

            app.kubernetes.io/instance=gcxi-9.0.016.03

            app.kubernetes.io/managed-by=Helm

            app.kubernetes.io/name=gcxi

            app.kubernetes.io/part-of=gcxi

            app.kubernetes.io/version=9.0.016.03

            controller-revision-hash=gcxi-7d889766b8

            gcxi/app=gcxi-worker

            gcxi/deployment-code=gcxi

            gcxi/global-code=gcxi

            gcxi/role=gcxi-worker

            helm.sh/chart=gcxi-9.0.016.03

            service=gcxi

            servicename=gcxi

            statefulset.kubernetes.io/pod-name=gcxi-0

Annotations: checksum/config: ebc136cae0d0be91522079384c1f4c1e98981954280d43c54f134cd602a1ee66

Status: Pending

IP:

IPs:

Controlled By: StatefulSet/gcxi

Containers:

gcxi:

Container ID:

Image:         gcxi:9.0.016.03

Image ID:

Ports:         8180/TCP, 34952/TCP, 8080/TCP

Host Ports:    0/TCP, 0/TCP, 0/TCP

Command:

  /genesys/gcxi/mstr_start.sh

State:          Waiting

  Reason:       ContainerCreating

Ready:          False

Restart Count:  0

Limits:

  memory:  64Gi

Requests:

  cpu:      1

  memory:   8Gi

Readiness:  tcp-socket :web delay=300s timeout=5s period=10s #success=1 #failure=2

Environment Variables from:

  gcxi-config      ConfigMap  Optional: false

  gcxi-config-ext  ConfigMap  Optional: false

  gcxi-config-pg   ConfigMap  Optional: false

Environment:

  POD_NAME:   gcxi-0 (v1:metadata.name)

  POD_MOUNT:   (v1:metadata.labels['gcxi/deployment-code'])

Mounts:

  /genesys/gcxi_shared from gcxi-share (rw)

  /mnt/log from gcxi-log (rw)

  /opt/app/gcti/gcxi/var from gcxi-var (ro)

  /var/run/secrets/kubernetes.io/serviceaccount from default-token-kz462 (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

gcxi-log:

Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

ClaimName:  gcxi-log-pvc

ReadOnly:   false

gcxi-share:

Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

ClaimName:  gcxi-share-pvc

ReadOnly:   false

gcxi-var:

Type:                Projected (a volume that contains injected data from multiple sources)

SecretName:          gcxi-secret

SecretOptionalName:  0xc0007a5088

SecretName:          gcxi-secret-ext

SecretOptionalName:  0xc0007a5089

SecretName:          gcxi-secret-pg

SecretOptionalName:  0xc0007a508a

default-token-kz462:

Type:        Secret (a volume populated by a Secret)

SecretName:  default-token-kz462

Optional:    false

QoS Class: Burstable

Node-Selectors:

Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s

             node.kubernetes.io/unreachable:NoExecute for 300s

Events:

Type Reason Age From Message

Normal Scheduled default-scheduler Successfully assigned gcxi/gcxi-0 to slcgensysl13tv

Warning FailedMount 20m (x11 over 26m) kubelet, slcgensysl13tv MountVolume.NewMounter initialization failed for volume “gcxi-share-pv-local” : path “/opt/app/gencx/gencxshared/gcxi/shared/” does not exist

Warning FailedMount 11m (x5 over 24m) kubelet, slcgensysl13tv Unable to attach or mount volumes: unmounted volumes=[gcxi-log gcxi-share], unattached volumes=[gcxi-log gcxi-share gcxi-var default-token-kz462]: timed out waiting for the condition

Warning FailedMount 6m29s (x2 over 22m) kubelet, slcgensysl13tv Unable to attach or mount volumes: unmounted volumes=[gcxi-log gcxi-share], unattached volumes=[gcxi-var default-token-kz462 gcxi-log gcxi-share]: timed out waiting for the condition

Warning FailedMount 8s (x21 over 26m) kubelet, slcgensysl13tv MountVolume.NewMounter initialization failed for volume “gcxi-log-pv-local” : path “/opt/app/gcti/gcxi/logs” does not exist

not sure how to proceed, please advise

Regards
Abhishek

2

This was last March … I am encountering exactly the same problem right now. The path name exists in my Ubuntu 20 instance, but kubernetes says it’s not there. You must have found a fix. Would be grateful if you replied with the fix you found. Thanks.

3

Hopefully @abhishek8682 figured this out, 5 months is quite a while.

@ctapang what’s the storage class being used? Also what is the CSI?

4

Thanks protosam. Storage class is “PersistentVolume” and the path refers to a folder in the machine where kubectl is being invoked (not the host machine). The host is Windows but the persistent volume resides in the Windows System Linux 2 (WSL2) Ubuntu 20 instance so I think there is no CSI (at least I did not have to download any CSI driver).

5

I mean’t the StorageClass that the PersistentVolume is using, but I think I understand. You want use hostPath to access some stuff (check out the example there btw, it’s easiest to just use hostPath in the pod spec).

Are Docker and WSL2 even in the same VM? The path would have to exist in the VM that Docker is running in.