SupplementalGroups not working for NFS volume

General Discussions
1

I’m doing some experimenting with security context options, but I’m hitting a bump when it comes to file permissions. I am adding NFS volumes to my container, and setting the permissions through init containers, as fsGroup seems to not affect NFS volumes (see https://github.com/kubernetes/examples/issues/260). When defining supplemental groups, the user gets added to the groups, but permissions are only matched for the main group (as defined by runAsGroup). I’ve tried various combinations of user and group settings, but it only works if the user or main group matches. Am I misunderstanding how supplemental groups work, or is this incorrect behaviour?

Container output

$ groups
groups: cannot find name for group ID 2000
2000 groups: cannot find name for group ID 3000
3000
$ whoami
whoami: cannot find name for user ID 999
$ ls -alh /mnt
total 8.0K
drwxr-xr-x 1 root root 4.0K Apr 17 11:37 .
drwxr-xr-x 1 root root 4.0K Apr 17 11:37 ..
drwxrwxr-x 2 1000 3000 205 Apr 17 09:46 perm1
drwxrwxr-x 3 root 2000 84 Apr 17 11:35 perm2
$ touch /mnt/perm1/foo.txt
touch: cannot touch '/mnt/perm1/foo.txt': Permission denied
$ touch /mnt/perm2/foo.txt
$ ls -alh /mnt/perm2/foo.txt
-rw-r--r-- 1 999 2000 0 Apr 17 11:51 /mnt/perm2/foo.txt

Deployment .yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: permission-test
  labels:
    app: permission-test
    group: test
    origin: test
spec:
  template:
    metadata:
      labels:
        app: permission-test
        group: test
        origin: test
    spec:
      securityContext:
        runAsUser: 999
        runAsGroup: 2000
        fsGroup: 2000 # Does not seem to affect NFS volumes at all
        supplementalGroups:
        - 2000
        - 3000
      containers:
        - name: permission-test
          image: debian
          imagePullPolicy: IfNotPresent
          volumeMounts:
          - name: permission-test-volume
            mountPath: /mnt/perm1/
          - name: permission-test-volume-2
            mountPath: /mnt/perm2/
          command: [ 'sleep' ]
          args: [ 'infinity' ]
      volumes:
      - name: permission-test-volume
        persistentVolumeClaim:
          claimName: permission-test-pv-claim
      - name: permission-test-volume-2
        persistentVolumeClaim:
          claimName: permission-test-pv-claim-2
      initContainers:
      - name: nfs-fixer
        image: alpine
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: permission-test-volume-2
          mountPath: /mnt/perm2
        - name: permission-test-volume
          mountPath: /mnt/perm1
        command:
        - sh
        - -c
        args:
        - chmod 0775 /mnt/perm2; chgrp 2000 /mnt/perm2
          ;chmod 0775 /mnt/perm1; chown -hR 1000:3000 /mnt/perm1
  replicas: 1
  selector:
    matchLabels:
      app: permission-test

Cluster information:

Kubernetes version: 1.16.3
Cloud being used: bare-metal
Installation method: kubeadm