NFS Persistent Volume Times Out - timeout expired waiting for volumes to attach or mount for pod

General Discussions
1

Hi, I have recently added a new node into our cluster. Build was identical in approach to existing nodes. I have a Persistent Volume setup to nfs mount a directory into the pod.
On launching the pod I see this error: Unable to mount volumes for pod “myapp_mynamespace”: timeout expired waiting for volumes to attach or mount for pod “mynamespace”/“myapp-xxxxx”. list of unmounted volumes=[nfs-logs].

Firewalls & connectivity have been checked. From the node in question (node2) I can make the same nfs mount but ONLY if I specify nfs version 3. (mount -v -t nfs -o vers=3 kubelog:/data/logs /tmp/mnt)

Cluster information:

Kubernetes version:
api1 v1.11.5 10.0.0.210 docker://1.13.1
node1 v1.11.5 10.0.0.211 docker://1.13.1
node2 v1.11.5 10.0.0.29 docker://1.13.1

Cloud being used: bare-metal
Installation method: Manual
Host OS: CentOS Linux 7 (Core)
CNI and version: kubernetes-cni.x86_64 0.6.0-0
CRI and version: cri-tools.x86_64 1.13.0-0

apiVersion: v1
kind: PersistentVolume
metadata:
pv.kubernetes.io/bound-by-controller: “yes”
finalizers:

  • kubernetes.io/pv-protection
    name: nfs-logs
    spec:
    accessModes:
  • ReadWriteMany
    capacity:
    storage: 50Gi
    claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: nfs-logs
    namespace: mynamespace
    nfs:
    path: /data/logs
    server: kubelog
    persistentVolumeReclaimPolicy: Retain