Expanding Persistent Volume Claim storage capactiy in GKE

General Discussions
1

Cluster information:

Kubernetes version: 1.21.11-gke.1900
Cloud being used: (put bare-metal if not on a public cloud) GCP
Installation method:
Host OS:
CNI and version:
CRI and version:

1

I increased the capacity of my elasticsearch persistent volume claim(PVC) but was unable to create the pod with that new volume.

I followed the following steps -

  • Set the allowVolumeExpansion field to true in their StorageClass object(s)
  • Scaled ElasticSearch Operator Deployment to 0 Replicas.
  • Deleted the statefulset Object without deleting the pods usingkubectl delete sts <statefulset-name> --cascade=orphan

Before deleting I saved the yaml of the statefulset using

kubectl get sts <statefulset-name> -o yaml
  • Increased the storage in capacity in the yaml file of PVC.
  • Recreated the StatefulSet with the new storage request by the yaml file I saved using

kubectl apply -f file-name.yml

  • Scaled back the operator deployment to 1

But, when I recreated the stateful set, the CrashLoopBackOff error is being shown every-time.

Following are some logs -

  • readiness probe failed
  • Likely root cause: java.io.IOException: No space left on device
  • using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdb)]], net usable_space [0b], net total_space [975.8mb], types [ext4]

What I can infer from these logs is that storage capacity of the PV has not increased. But the yaml of the same is showing changes. Can anyone guide me here, what I am missing here ?