Issue with using the block device inside a pod

General Discussions
1

Hi Team,

Components:
Kubernetes, Ceph

Problem statement:
We are trying to integrate Ceph with kubernetes.
We are unable to utilize the block volume mode in the pod.

Description:
OS: Almalinux 8.8
Ceph version: 18.2
Kubernetes version: 1.28.2

We have deployed a single node kubernetes cluster and a 3 node ceph cluster.
We are trying to integrate kubernetes with ceph using csi-rbd plugin.
We have used the following link to do the same:
https://docs.ceph.com/en/reef/rbd/rbd-kubernetes/

We are strugling with the usage of the pvc we created in the block mode. Kindly refer the “raw-block-pvc-4.yaml”.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc-4
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 500Gi
  storageClassName: csi-rbd-sc

We created a pod using the file “raw-block-pod-4.yaml”.

---
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-raw-block-volume-2
spec:
  containers:
    - name: fc-container
      image: quay.io/cephcsi/cephcsi:canary
      command: ["/bin/sh", "-c"]
      args: ["tail -f /dev/null"]
      volumeDevices:
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc-4

The pod was created successfully. The device path we used was /dev/xvda.
We are unable to use this path inside the pod. Its not gettting shown in the lsblk of the pod and its not even shown in the df -kh of the pod.
When we try to mount /dev/xvda on /tmp but we get permission denied error.
We get a similar error when we try to make partiotion of the disk. The partition gets created but partiotion table is not updated.
At the same time we are able to format the disk using the command mkfs.ext4 /dev/xvda.
Kindly refer the output:

"
[root@kube-cluster-1 ceph_provisioner]# k exec -it pod-with-raw-block-volume-2 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.4# hostname
pod-with-raw-block-volume-2
sh-4.4# df -kh
Filesystem                  Size  Used Avail Use% Mounted on
overlay                      70G   14G   57G  19% /
tmpfs                        64M     0   64M   0% /dev
tmpfs                        32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/almalinux-root   70G   14G   57G  19% /etc/hosts
shm                          64M     0   64M   0% /dev/shm
tmpfs                        63G   12K   63G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                        32G     0   32G   0% /proc/acpi
tmpfs                        32G     0   32G   0% /proc/scsi
tmpfs                        32G     0   32G   0% /sys/firmware
sh-4.4# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0     1G  0 loop
loop1    7:1    0   500G  0 loop
sda      8:0    0 446.6G  0 disk
|-sda1   8:1        200M  0 part
|-sda2   8:2          1M  0 part
|-sda3   8:3          1G  0 part
`-sda4   8:4    0 445.4G  0 part
sdb      8:16   0 893.3G  0 disk
rbd0   252:0    0     1G  0 disk
rbd1   252:16   0   500G  0 disk
rbd2   252:32   0   100G  0 disk
sh-4.4# ls /dev
core  fd  full  mqueue  null  ptmx  pts  random  shm  stderr  stdin  stdout  termination-log  tty  urandom  xvda  zero
sh-4.4#
"

Although we are able to see a volume corresponding to the PV.
But we are not able to use the block device.

Could anyone help me out.

Thanks and Regards,
Kushagra Gupta