A problem with local persistent volumes keeps trying unmount and fails

General Discussions
1

Hello!

We share local-storage filesystem folder-backed PVs and PVCs among many pods. Reconciler keeps trying to unmount, then nestedoperations.go report an error that volume is still mounted and reports an error (GetDeviceMountRefs check failed for volume, … mount path is still mounted by other referneces).

Log fragment
journalctl -u kubelet.service

I1209 11:54:52.357529 1482 reconciler.go:294] operationExecutor.UnmountDevice started for volume “XXXXXXXX” on node “YYYYYYYY”
E1209 11:54:52.363848 1482 nestedpendingoperations.go:267
] Operation for ““kubernetes.io/local-volume/292d521b-e0fd-11e9-ab70-002590fcbd40-XXXXXXXXXXX”” failed. No retries permitted until 2019-12-09 11:56:54.363763246 +0300 MSK m=+7584457.089545278 (durationBeforeRetry 2m2s).
Error: "GetDeviceMountRefs check failed for volume “XXXXXXXXXXXXXX”

Cluster information:

Kubernetes version: v1.13
Cloud being used: bare-metal
Installation method: kubespray
Host OS: Ubuntu 16.04LTS
CNI and version: calico v3.1.3
CRI and version: docker 18.06.1-ce

YAML for PV

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2019-03-13T11:46:30Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: XXXXXXXXXXXXXXX
  resourceVersion: "3024931"
  selfLink: /api/v1/persistentvolumes/XXXXXXXXXXXXXXXX
  uid: a4cff352-4585-11e9-8e90-002590fcbd40
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: XXXXXXXXXXXXXXX-claim
    namespace: YYYYYYYY
    resourceVersion: "3024824"
    uid: 21d86053-45a1-11e9-8e90-002590fcbd40
  local:
    path: /local/path/on/server
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - hostname
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  volumeMode: Filesystem
status:
  phase: Bound

describe for the PVC bound to the volume

Name:          XXXXXXXXXXXX-claim
Namespace:     namespace
StorageClass:  local-storage
Status:        Bound
Volume:        XXXXXXXXX
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      50Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Events:        <none>
Mounted By:    XXXXXXX1-6f9dfb6f89-tqm9s
               XXXXXXX2-5445bb8f87-zj4cn
               XXXXXXX3-7bf446cc5d-s9jv5