Issue: Hybrid K8S cluster cannot use vsphere storage plugin, NO VM found error

General Discussions
1

So I created a cluster using kubespray with vsphere as a storage plugin, it worked fine since my cluster is deployed on an ESXi hypervisor. Then I added new nodes(VMs) to the cluster, the new nodes do not run on vsphere, I deleted the storage plugin part in the “kubelet.env” file on those nodes so the kubelet can start normally.

The problems started to show when I used a storage class that is based on vsphere provider. When I create a PVC I have an error message saying that no “NO VM FOUND”

When I looked into the logs I found out that the vsphere plugin is trying to locate the new vms using theirs UUI, and since those VMs are not running on the vsphere cluster he cannot find them. and he fire an error and stop the pvc creation. even when I created the a vmdk manually and attached it to a PV the problem persiste.

is there a way to create a PVC using the vsphere storage class. with this setup ?

2

I have researched setting up k8s on our VMWare cluster here, but have not actually setup a k8s cluster yet.

Based on my reading, vsphere CSI (container storage interface) stores volumes on a pre-selected datastore. i.e. in the kubespray docs at https://github.com/kubernetes-sigs/kubespray/blob/master/docs/vsphere.md that would be vsphere_datastore

The vsphere csi driver, installed on each node, maps vmfs ‘disks’ to pods on the respective node, as k8s volumes. This can only work for nodes that are running as VMs on VMWare hosts that have access to the vsphere_datastore

In your case, it sounds like you’ve added a node that’s not running as a VM on the VMWare cluster that holds vsphere_datastore, therefore the vsphere CSI driver is a) not running on that node (you had to remove it from the kubelet config), and b) there’s no way for the node to reach the VMFS datastore because it’s outside the VMWare cluster.

You could work around this issue by using a different storage provider, such as NFS, Rook or OpenEBS

Check out https://landscape.cncf.io/category=cloud-native-storage&format=card-mode&grouping=category for a list of storage providers

You could continue to use vsphere storage, but you’d need to label those pods such that they’d only get scheduled on nodes running in vsphere.