Spdk unable to scan NVMe PCI device (vfio) in the non-privileged mode

General Discussions
1

Cluster information:

Kubernetes version: v1.17.1
Cloud being used: bare-metal
Installation method: Manual
Host OS: 18.04
CNI and version: flannel
CRI and version: docker 19.03.12

vfio is enabled on the k8s cluster and so far pods are running in a privileged mode and now we would want to run pod in non-privileged mode but spdk unable to scan NVMe PCI device (vfio) in the non-privileged mode

nvme.c: 609:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
bdev_nvme.c:1766:spdk_bdev_nvme_create: *ERROR*: No controller was found with provided trid (traddr: 0000:88:00.0)

The vfio device seems to be present though in the non-privileged mode

$ kubectl exec -it pod-0 /bin/bash
$ ls -ld /dev/vfio/91
crw------- 1 root root 241, 10 Jul 16 14:44 /dev/vfio/91
$ readlink /sys/bus/pci/devices/0000\:88\:00.0/iommu_group
../../../../../../kernel/iommu_groups/91


{
    "apiVersion": "v1",
    "kind": "Pod",
    "spec": {
        "hostIPC": true,
        "hostNetwork": true,
        "containers": [
            {
                "image": "local-repo:5000/pa/project/podgenv:latest",
                "command": [
                    "red_agent.py"
                ],
                "securityContext": {
                    "privileged": false,
                    "runAsUser": 10073,
                    "capabilities": {
                        "add": [
                            "IPC_LOCK",
                            "SYS_PTRACE",
                            "SYS_NICE"
                        ]
                    }
                },
                "resources": {
                    "limits": {
                        "hugepages-1Gi": "4Gi",
                        "memory": "4Gi"
                    },
                    "requests": {
                        "memory": "4Gi"
                    }
                },
                "volumeMounts": [
                    {
                        "mountPath": "/project",
                        "name": "uservol0"
                    },
                    {
                        "mountPath": "/home/pa",
                        "name": "uservol1"
                    },
                    {
                        "mountPath": "/dev/hugepages",
                        "name": "hugepages"
                    },
                    {
                        "mountPath": "/dev/shm",
                        "name": "shm"
                    },
                    {
                        "mountPath": "/dev/sys",
                        "name": "sys"
                    },
                    {
                        "mountPath": "/dev/vfio/91",
                        "name": "vfio91"
                    }
                ],
                "name": "pod-0"
            }
        ],
        "volumes": [
            {
                "hostPath": {
                    "path": "/home/pa/project",
                    "type": "Directory"
                },
                "name": "uservol0"
            },
            {
                "hostPath": {
                    "path": "/home/pa",
                    "type": "Directory"
                },
                "name": "uservol1"
            },
            {
                "hostPath": {
                    "path": "/dev/hugepages",
                    "type": "Directory"
                },
                "name": "hugepages"
            },
            {
                "emptyDir": {
                    "medium": "Memory"
                },
                "name": "shm"
            },
            {
                "hostPath": {
                    "path": "/sys",
                    "type": "Directory"
                },
                "name": "sys"
            },
            {
                "hostPath": {
                    "path": "/dev/vfio/91",
                    "type": "CharDevice"
                },
                "name": "vfio91"
            }
        ],
        "hostname": "pod-0",
        "affinity": {
            "nodeAffinity": {
                "requiredDuringSchedulingIgnoredDuringExecution": {
                    "nodeSelectorTerms": [
                        {
                            "matchExpressions": [
                                {
                                    "key": "kubernetes.io/hostname",
                                    "operator": "In",
                                    "values": [
                                        "sn001"
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    },
    "metadata": {
        "name": "pod-0"
    }
}

any hints on running the pod with spdk PCI device on vfio on a non-privileged mode