Overview
I’m trying to install the VCD named-disk CSI driver (cloud-director-named-disk-csi-driver) on MicroK8s. Everything deploys fine and seems to be running, but when trying to deploy a pod with a PVC, the PV/PVC is created, a named-disk is created in VCD, but it fails to attach it to the node/pod with error:
CSINode ubuntu24 does not contain driver named-disk.csi.cloud-director.vmware.com
Environment:
- OS: Ubuntu 24.04
- MicroK8s: v1.32.2 & 1.27.16
- VCD named-disk CSI driver: v1.6.0
- VCD version: 10.5.1.23400185
Installation of CSI driver
git clone GitHub - vmware/cloud-director-named-disk-csi-driver: Container Storage Interface (CSI) driver for VMware Cloud Director
git checkout tags/1.6.0
Edit vcloud-basic-auth.yaml & vcloud-csi-config.yaml with my VCD details
kubectl apply -f manifests/vcloud-basic-auth.yaml
kubectl apply -f manifests/vcloud-csi-config.yaml
kubectl apply -f manifests/csi-driver.yaml
kubectl apply -f manifests/csi-controller.yaml
kubectl apply -f manifests/csi-node.yaml
Resources
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: vcd-sk-sc
provisioner: named-disk.csi.cloud-director.vmware.com
parameters:
storageProfile: "SK-DC01 System Disks"
filesystem: "ext4"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: vcd-sk-sc
---
apiVersion: v1
kind: Pod
metadata:
name: nginx-pod
spec:
containers:
- name: nginx
image: nginx:latest
volumeMounts:
- mountPath: /usr/share/nginx/html
name: nginx-storage
volumes:
- name: nginx-storage
persistentVolumeClaim:
claimName: nginx-pvc
Logs & Troubleshooting
CSI socket is created:
root@ubuntu24:~# ls -la /var/lib/kubelet/csi-plugins/named-disk.csi.cloud-director.vmware.com/
total 8
drwxr-xr-x 2 root root 4096 Mar 19 08:12 .
drwxr-xr-x 3 root root 4096 Mar 18 17:05 ..
srwxr-xr-x 1 root root 0 Mar 19 08:12 csi.sock
CSI Driver resource is created:
% kubectl get csidrivers
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
named-disk.csi.cloud-director.vmware.com true false false <unset> false Persistent 28s
But no drivers seems to have been registered:
% kubectl get csinode ubuntu24 -o yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-volume,kubernetes.io/vsphere-volume
creationTimestamp: "2025-03-11T19:45:18Z"
name: ubuntu24
ownerReferences:
- apiVersion: v1
kind: Node
name: ubuntu24
uid: 17116251-6f52-43f8-b211-3af19be09529
resourceVersion: "99"
uid: 94165cb5-b16a-429a-b07a-984b27fc6c8a
spec:
drivers: null
node-driver-registrar logs:
% kubectl -n kube-system logs csi-vcd-nodeplugin-snmhj -c node-driver-registrar
I0319 07:12:07.813474 1 main.go:113] Version: v2.2.0
I0319 07:12:07.813864 1 main.go:137] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0319 07:12:07.813892 1 connection.go:153] Connecting to unix:///csi/csi.sock
I0319 07:12:09.778832 1 main.go:144] Calling CSI driver to discover driver name
I0319 07:12:09.778846 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0319 07:12:09.778849 1 connection.go:183] GRPC request: {}
I0319 07:12:09.780683 1 connection.go:185] GRPC response: {"name":"named-disk.csi.cloud-director.vmware.com","vendor_version":"1.6.0"}
I0319 07:12:09.780736 1 connection.go:186] GRPC error: <nil>
I0319 07:12:09.780742 1 main.go:154] CSI driver name: "named-disk.csi.cloud-director.vmware.com"
I0319 07:12:09.780819 1 node_register.go:52] Starting Registration Server at: /registration/named-disk.csi.cloud-director.vmware.com-reg.sock
I0319 07:12:09.780986 1 node_register.go:61] Registration Server started at: /registration/named-disk.csi.cloud-director.vmware.com-reg.sock
I0319 07:12:09.781067 1 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: ""
vcd-csi-driver logs:
kubectl logs -n kube-system -l app=csi-vcd-nodeplugin --tail=100 -c vcd-csi-plugin
I0319 15:58:41.001003 1 driver.go:236] Listening for connections on address: "//csi/csi.sock"
I0319 15:58:41.148448 1 driver.go:210] GRPC call: [/csi.v1.Identity/GetPluginInfo]: [&csi.GetPluginInfoRequest{XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}]
I0319 15:58:41.148479 1 identity.go:38] GetPluginInfo: called with args [{XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
I0319 15:58:41.447907 1 driver.go:210] GRPC call: [/csi.v1.Identity/Probe]: [&csi.ProbeRequest{XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}]
I0319 15:58:41.448044 1 identity.go:33] Probe: called with args [{XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
I0319 15:58:41.448961 1 driver.go:210] GRPC call: [/csi.v1.Identity/GetPluginInfo]: [&csi.GetPluginInfoRequest{XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}]
I0319 15:58:41.449121 1 identity.go:38] GetPluginInfo: called with args [{XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
I0319 15:58:41.449715 1 driver.go:210] GRPC call: [/csi.v1.Identity/GetPluginCapabilities]: [&csi.GetPluginCapabilitiesRequest{XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}]
I0319 15:58:41.449731 1 identity.go:48] GetPluginCapabilities: called with args [{XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]
I0319 15:58:41.452572 1 driver.go:210] GRPC call: [/csi.v1.Controller/ControllerGetCapabilities]: [&csi.ControllerGetCapabilitiesRequest{XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}]
I0319 15:58:41.452588 1 controller.go:369] ControllerGetCapabilities: called with args [csi.ControllerGetCapabilitiesRequest{XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}]
I0319 15:58:41.452591 1 controller.go:371] Returning controller capabilities [[]*csi.ControllerServiceCapability{(*csi.ControllerServiceCapability)(0xc000329e60), (*csi.ControllerServiceCapability)(0xc000329ec0), (*csi.ControllerServiceCapability)(0xc000329f20), (*csi.ControllerServiceCapability)(0xc000329f80)}]
Kubelite logs:
Mar 20 05:26:13 ubuntu24 microk8s.daemon-kubelite[1967]: W0320 05:26:13.234683 1967 logging.go:55] [core] [Channel #4567 SubChannel #4568]grpc: addrConn.createTransport failed to connect to {Addr: "/var/snap/microk8s/common/var/lib/kubelet/plugins_registry/named-disk.csi.cloud-director.vmware.com-reg.sock", ServerName: "%2Fvar%2Fsnap%2Fmicrok8s%2Fcommon%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fnamed-disk.csi.cloud-director.vmware.com-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/snap/microk8s/common/var/lib/kubelet/plugins_registry/named-disk.csi.cloud-director.vmware.com-reg.sock: connect: invalid argument"
Mar 20 05:26:14 ubuntu24 microk8s.daemon-kubelite[1967]: I0320 05:26:14.362865 1967 reconciler.go:360] "attacherDetacher.AttachVolume started" volumeName="kubernetes.io/csi/named-disk.csi.cloud-director.vmware.com^pvc-e9bf8ff0-40fb-476d-ba95-b84ff1d50cf7" nodeName="ubuntu24" scheduledPods=["vcd-test/nginx-pod"]
Mar 20 05:26:14 ubuntu24 microk8s.daemon-kubelite[1967]: E0320 05:26:14.869055 1967 csi_attacher.go:645] kubernetes.io/csi: attachment for pvc-e9bf8ff0-40fb-476d-ba95-b84ff1d50cf7 failed: CSINode ubuntu24 does not contain driver named-disk.csi.cloud-director.vmware.com
Seems to me the CSI driver starts properly, creates it’s sockets, kubelite detects its presence but is unable to comunicate properly with it so it doesn’t register the driver on the node.
Any suggestions how to continue the troubleshooting?