How to access EFS from kubernetes sa in AWS?

General Discussions
1

I deployed an EFS in AWS and a test pod on EKS from this document: Amazon EFS CSI driver.

EFS CSI Controller pods in the kube-system

kube-system    efs-csi-controller-5bb76d96d8-b7qhk        3/3     Running   0          26s
kube-system    efs-csi-controller-5bb76d96d8-hcgvc        3/3     Running   0          26s

After deployed a sample application from the doc, when confirm efs-csi-controller sa pod logs, it seems they didn’t work well.

Pod 1

$ kubectl logs efs-csi-controller-5bb76d96d8-b7qhk \
>     -n kube-system \
>     -c csi-provisioner \
>     --tail 10
W1030 08:15:59.073406       1 feature_gate.go:235] Setting GA feature gate Topology=true. It will be removed in a future release.
I1030 08:15:59.073485       1 feature_gate.go:243] feature gates: &{map[Topology:true]}
I1030 08:15:59.073500       1 csi-provisioner.go:132] Version: v2.1.1-0-g353098c90
I1030 08:15:59.073520       1 csi-provisioner.go:155] Building kube configs for running in cluster...
I1030 08:15:59.087072       1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I1030 08:15:59.087512       1 common.go:111] Probing CSI driver for readiness
I1030 08:15:59.090672       1 csi-provisioner.go:202] Detected CSI driver efs.csi.aws.com
I1030 08:15:59.091694       1 csi-provisioner.go:244] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I1030 08:15:59.091997       1 controller.go:756] Using saving PVs to API server in background
I1030 08:15:59.092834       1 leaderelection.go:243] attempting to acquire leader lease kube-system/efs-csi-aws-com...

Pod 2

$ kubectl logs efs-csi-controller-5bb76d96d8-hcgvc \
>     -n kube-system \
>     -c csi-provisioner \
>     --tail 10
I1030 08:16:32.628759       1 controller.go:1099] Final error received, removing PVC 111111a-d6fb-440a-9bb1-132901jfas from claims in progress
W1030 08:16:32.628783       1 controller.go:958] Retrying syncing claim "111111a-d6fb-440a-9bb1-132901jfas", failure 5
E1030 08:16:32.628798       1 controller.go:981] error syncing claim "111111a-d6fb-440a-9bb1-132901jfas": failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
I1030 08:16:32.628845       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs-claim", UID:"111111a-d6fb-440a-9bb1-132901jfas", APIVersion:"v1", ResourceVersion:"1724705", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
I1030 08:17:04.628997       1 controller.go:1332] provision "default/efs-claim" class "efs-sc": started
I1030 08:17:04.629193       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs-claim", UID:"111111a-d6fb-440a-9bb1-132901jfas", APIVersion:"v1", ResourceVersion:"1724705", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/efs-claim"
I1030 08:17:04.687957       1 controller.go:1099] Final error received, removing PVC 111111a-d6fb-440a-9bb1-132901jfas from claims in progress
W1030 08:17:04.687977       1 controller.go:958] Retrying syncing claim "111111a-d6fb-440a-9bb1-132901jfas", failure 6
E1030 08:17:04.688001       1 controller.go:981] error syncing claim "111111a-d6fb-440a-9bb1-132901jfas": failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
I1030 08:17:04.688044       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs-claim", UID:"111111a-d6fb-440a-9bb1-132901jfas", APIVersion:"v1", ResourceVersion:"1724705", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied

From events, I can see

$ kubectl get events
27m         Warning   FailedScheduling         pod/efs-app                                    skip schedule deleting pod: default/efs-app
7m38s       Warning   FailedScheduling         pod/efs-app                                    0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
7m24s       Warning   FailedScheduling         pod/efs-app                                    0/2 nodes are available: 2 persistentvolumeclaim "efs-claim" is being deleted.
7m24s       Warning   FailedScheduling         pod/efs-app                                    skip schedule deleting pod: default/efs-app
17s         Warning   FailedScheduling         pod/efs-app                                    0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
27m         Normal    ExternalProvisioning     persistentvolumeclaim/efs-claim                waiting for a volume to be created, either by external provisioner "efs.csi.aws.com" or manually created by system administrator
10m         Normal    ExternalProvisioning     persistentvolumeclaim/efs-claim                waiting for a volume to be created, either by external provisioner "efs.csi.aws.com" or manually created by system administrator
11m         Normal    Provisioning             persistentvolumeclaim/efs-claim                External provisioner is provisioning volume for claim "default/efs-claim"
11m         Warning   ProvisioningFailed       persistentvolumeclaim/efs-claim                failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
7m47s       Normal    Provisioning             persistentvolumeclaim/efs-claim                External provisioner is provisioning volume for claim "default/efs-claim"
7m47s       Warning   ProvisioningFailed       persistentvolumeclaim/efs-claim                failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
74s         Normal    ExternalProvisioning     persistentvolumeclaim/efs-claim                waiting for a volume to be created, either by external provisioner "efs.csi.aws.com" or manually created by system administrator
2m56s       Normal    Provisioning             persistentvolumeclaim/efs-claim                External provisioner is provisioning volume for claim "default/efs-claim"
2m56s       Warning   ProvisioningFailed       persistentvolumeclaim/efs-claim                failed to provision volume with StorageClass "efs-sc": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied

Service account was created by

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: efs-csi-controller-sa
  namespace: kube-system
  labels:
    app.kubernetes.io/name: aws-efs-csi-driver
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/AmazonEKS_EFS_CSI_Driver_Policy

The AmazonEKS_EFS_CSI_Driver_Policy is the json from https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/v1.3.2/docs/iam-policy-example.json

2 Likes
2

Same issue. Don’t understand why the rpc error is “Unauthenticated”

1 Like
3

Hi. Having the same issue here with that AWS tutorial. How did you solved it?

Y tried also by removing the Policy’s conditions

        "Condition": {
            "StringLike": {
                "aws:RequestTag/efs.csi.aws.com/cluster": "true"

and

       "Condition": {
            "StringEquals": {
                "aws:ResourceTag/efs.csi.aws.com/cluster": "true"

But still the rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied

Thanks

4

The problem here is the iam policy, you gotta use the latest one from the aws-efs-csi-driver github master branch or at least the version of it that matches your efs csi controller. Not sure when they became outta sync, so anyone else who runs into this issue shouldn’t use the iam policy from the aws documentation, which I think is pointing to v1.2.0, instead use this: https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.json