Many of my workflows use pod iam roles. As documented here, I must include fsGroup in order for non-root containers to read the generated identity token. The problem with this is when I additionally include pvc’s that point to cifs pv’s, the volumes fail to mount because they time out. Seemingly this is because Kubernetes (Kubelet?) tries to chown all of the files on the volume, which takes too much time and causes the timeout. Questions…
Why doesnt Kubernetes try to chown all of the files when hostPath is used instead of a pvc? All of the workflows were fine until I made the switch to use pvcs from hostPath, and now the timeout issue happens.
Why does this problem happen on cifs pvcs but not nfs pvcs? I have noticed that nfs pvcs continue to mount just fine and the fsGroup seemingly doesn’t take effect as I don’t see the group id change on any of the files. However, the cifs pvcs can no longer be mounted seemingly due to the timeout issue. If it matters, I am using the native nfs pv lego and this cifs flexVolume plugin that has worked great up until now: https://github.com/fstab/cifs/blob/master/cifs
Can someone elaborate on or point to the block of code that shows how Kubernetes determines when to chown all of the files on a volume when fsGroup is included? Does it always try to chown all of the files when it’s included?
I suspect hostPath assumes a shared local directory so changing permissions could break things. A pvc is theoretically dedicated. But that’s just a guess.
The only suggestion I have is here:
fsGroupChangePolicy = OnRootMismatch
It’s an alpha feature.
I’ve never used it but may avoid it always trying to set permissions.
Thanks for sharing Stephen. I remember seeing that feature, but I think it’s for 1.18. We are currently on EKS 1.15 and hopefully soon to be EKS 1.16 (it is actually a huge hassle to jump from 1.15 to 1.16 because many of the valid resource apis changed). So, I don’t think we will get to 1.18 for many moons even when AWS offers it.
Until then I need to come up with some sort of scalable workaround or abandon pod iam roles altogether. I did come up with a way to have an init container change the permissions of the token, but it’s not really scalable and will only work in one off projects.