Hi,
This question is about StatefulSet and Local Persistent Volume (Kubernetes 1.14: Local Persistent Volumes GA | Kubernetes)
If we deploy a StatefulSet with the pods using local persistent volumes, when the Kube node hosting a persistent volume is gone, the corresponding pod become un-schedulable. My question is how can a operator reliably detect this problem?
I can’t find any documentation taking about this. Can operator receives a notification or something?
What I observed is when a node hosting a PV is deleted, the corresponding pod stuck in pending stage. One way I can think of to detect this problem is to find the PVC for the pod, then find the PV for the PVC, and then find the node the PV is on, then query to see if the node is there.
But the problem is, inspecting PV and node requires cluster level privilege, which ideally should not be given to an operator that is only supposed to manage namespace level resources.
Plus, I am not sure that (following Pod->PVC->PV->Node sequence) captures all possible situations that physical storage becomes inaccessible.
What is the proper way to detect the situation? Once the situation is detected, it is pretty easy to fix.
Thank you very much!