Write permissions on volume mount with security context fsgroup option

General Discussions
1

I’m trying to run a tomcat container in K8S with a non-root user, to do so I set User ‘tomcat’ with the appropriate permission in Docker Image. I have a startup script that creates a directory in /opt/var/logs (during container startup) and also starts tomcat service.

#steps in Dockerfile
#adding tomcat user and group and permission to /opt directory
addgroup tomcat -g 1001 && \
adduser -D -u 1001 -G tomcat tomcat && \
chown -R tomcat:tomcat /opt

#switch user
User tomcat

The pod runs fine in K8S when deployed using deployment without any volume mapped.

But I get a permission denied error (permission denied: creating directory /opt/var/logs/docker/) from the startup script, which fails to create a directory when I map the deployment with the persistent volume claim, even though I set the fsgroup as explained here, Configure a Security Context for a Pod or Container | Kubernetes.

I have a persistent volume of type hostPath.

The deployment definition is as below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ms-tomcat
  namespace: ms-ns
  labels:
    app: tomcat
spec:
  selector:
    matchLabels:
      app: tomcat
  template:
    metadata:
      labels:
        app: tomcat
    spec:
      securityContext:
          fsGroup: 1001
          runAsUser: 1001
          runAsGroup: 1001
      containers:
      - name: tomcat
        image: docker-registry.test.com/tomcat:1.2
        volumeMounts:
        - name: logging-volume
          mountPath: /opt/var/logs/docker
      imagePullSecrets:
      - name: test
      volumes:
      - name: logging-volume
        persistentVolumeClaim:
          claimName: nonroot-test-pvc

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name:  nonroot-test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: local-node-sc
  volumeName: nonroot-test-pv

The only solution I found so far is to run initcontianer with root and provide the permission to the directory from mapped volume but I have got more than 100 services on K8S and adding init containers would slow down everything.

initContainers:
      - name: volume-mount-hack
        image: busybox
        command: ["sh", "-c", "chown -R 501:501 /opt"]
        volumeMounts:
        - name: logging-volume
          mountPath: /opt/var/logs/docker

I also tried setting up mount option in storage class as well as in persistent volume but did not help, ibm cloud - Kubernetes Persistent Volume Claim mounted with wrong gid - Stack Overflow

Any help or suggestion is really appreciated.

1 Like
2

You’re troubleshooting file permissions. Remove your init container and try out the following. If you could provide this output it might also be useful in helping you.

Get a shell into the container.

kubectl exec -it POD_NAME -- sh

Confirm your uid and gid.

/ $ id
uid=1001 gid=1001 groups=1001

Test writing to the directory.

/ $ echo test > /opt/var/logs/docker/test.txt 
/ $ cat /opt/var/logs/docker/test.txt
test

Walk the directory and confirm permissions along the directory.

/ $ ls -lah /opt /opt/var /opt/var/logs /opt/var/logs/docker/
/opt:
total 12K    
drwxr-xr-x    1 root     root        4.0K Jun 30 14:06 .
drwxr-xr-x    1 root     root        4.0K Jun 30 14:06 ..
drwxr-xr-x    3 root     root        4.0K Jun 30 14:06 var

/opt/var:
total 12K    
drwxr-xr-x    3 root     root        4.0K Jun 30 14:06 .
drwxr-xr-x    1 root     root        4.0K Jun 30 14:06 ..
drwxr-xr-x    3 root     root        4.0K Jun 30 14:06 logs

/opt/var/logs:
total 12K    
drwxr-xr-x    3 root     root        4.0K Jun 30 14:06 .
drwxr-xr-x    3 root     root        4.0K Jun 30 14:06 ..
drwxrwsr-x    3 root     1001        4.0K Jun 30 14:07 docker

/opt/var/logs/docker/:
total 28K    
drwxrwsr-x    3 root     1001        4.0K Jun 30 14:07 .
drwxr-xr-x    3 root     root        4.0K Jun 30 14:06 ..
drwxrws---    2 root     1001       16.0K Jun 30 13:58 lost+found
-rw-r--r--    1 1001     1001           5 Jun 30 14:14 test.txt
3

@protosam - So the problem is pod never starts as the start up script fails to create a directory (for service) under /opt/var/logs/docker/<service_name>.

Error: mkdir: cannot create directory '/opt/var/logs/docker/local-service/': Permission denied

So I already tested with another image whose start up script does not create a folder. And when I run a pod from the image, the volume gets mapped

image
image591×530 9.2 KB

4

Set spec.containers.command so you can debug it with something like [ "sh", "-c", "sleep 84000s" ].

5

@protosam Found out that the PV of type hostpath does not support security context, and another type i.e. NFS is having the same issue as hostPath.
with type ‘emptyDir’ , it worked, but this type of volume lasts for the life of the pod, so not a reliable option.
with type ‘local’ again has the limitation, not recommended for production use as the volume is tightly bound to k8s host, if the host runs with an issue, the pod accessing that volume becomes unstable.

Any other type of volume you can recommend which will flexible as hostPath.

1 Like
6

If you’re developing in docker, I would say don’t worry about it. You should expect your dev environment to be ephemeral.

For live systems, cloud providers typically provide a CSI to use and you would use that. However, after I’ve found that smaller companies can have CSIs that are less fleshed out than others… and I also really want to use that on-board storage on the budget hosts I have been toying around with.

My favorite storage solution has been longhorn. It’s just simple.
There’s also rook, but I don’t like it’s operator pattern of configuring multiple storage clusters. I just want one big ceph cluster that I can use.

7

@mansing_shinde have you sort it out this issue? If you can share the solution. thx