Volume attachment fails - CSI Cinder - Openstack

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.19.7
Cloud being used: (put bare-metal if not on a public cloud) Openstack - On Prem
Installation method: Kubespray
Host OS: Ubuntu bionic
CNI and version: Quay
CRI and version: docker://19.3.14

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

2

$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
10m Normal WaitForFirstConsumer persistentvolumeclaim/csi-pvc waiting for first consumer to be created before binding
10m Normal ExternalProvisioning persistentvolumeclaim/csi-pvc waiting for a volume to be created, either by external provisioner “cinder.csi.openstack.org” or manually created by system administrator
10m Normal Provisioning persistentvolumeclaim/csi-pvc External provisioner is provisioning volume for claim “default/csi-pvc”
10m Normal ProvisioningSucceeded persistentvolumeclaim/csi-pvc Successfully provisioned volume pvc-c54ef457-b6f2-4a3c-b396-4f24cd3060a0
10m Normal Scheduled pod/nginx-csi Successfully assigned default/nginx-csi to srr-k8s-node-1
16s Warning FailedAttachVolume pod/nginx-csi AttachVolume.Attach failed for volume “pvc-c54ef457-b6f2-4a3c-b396-4f24cd3060a0” : rpc error: code = NotFound desc = ControllerPublishVolume Instance not found
8m30s Warning FailedMount pod/nginx-csi Unable to attach or mount volumes: unmounted volumes=[csi-data], unattached volumes=[default-token-dzztj csi-data]: timed out waiting for the condition
105s Warning FailedMount pod/nginx-csi Unable to attach or mount volumes: unmounted volumes=[csi-data], unattached volumes=[csi-data default-token-dzztj]: timed out waiting for the condition

ensemble@mano-launchpad:~$ cat pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc
spec:
accessModes:

  • ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
    storageClassName: standard

$ cat nginx.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-csi
spec:
hostAliases:

  • ip: “10.1.23.90”
    hostnames:
    • “controller”
      containers:
    • image: nginx
      imagePullPolicy: IfNotPresent
      name: nginx
      ports:
      • containerPort: 80
        protocol: TCP
        volumeMounts:
      • mountPath: /var/lib/www/html
        name: csi-data
        volumes:
    • name: csi-data
      persistentVolumeClaim:
      claimName: csi-pvc
      readOnly: false

$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard (default) cinder.csi.openstack.org Delete WaitForFirstConsumer false 34m
$

$ kubectl logs -f csi-cinder-controllerplugin-868c8bc55f-h9l6n -n kube-system
Defaulted container “csi-attacher” out of: csi-attacher, csi-provisioner, csi-snapshotter, csi-resizer, cinder-csi-plugin
I0621 04:19:28.889157 1 main.go:91] Version: v2.2.0-0-g97411fa7
I0621 04:19:28.890752 1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I0621 04:19:30.720456 1 common.go:111] Probing CSI driver for readiness
W0621 04:19:30.727413 1 metrics.go:142] metrics endpoint will not be started because metrics-address was not specified.
I0621 04:19:30.728822 1 controller.go:121] Starting CSI attacher
E0621 04:31:45.076082 1 reflector.go:320] k8s.io/client-go/informers/factory.go:135: Failed to watch *v1beta1.CSINode: Get https://10.233.0.1:443/apis/storage.k8s.io/v1beta1/csinodes?allowWatchBookmarks=true&resourceVersion=1296&timeout=5m19s&timeoutSeconds=319&watch=true: dial tcp 10.233.0.1:443: connect: connection refused