Trying to setup an EKS cluster with EBS CSI Driver Add On. When I install my helm chart in the cluster, csi driver fails to create VolumeAttachment. PV (created via terraform) was Bound to PVC but AttachVolume.Attach failed.
Here are some of the logs in EBS CSI Driver controller pod
ebs-plugin E0314 08:11:17.540346 1 batcher.go:161] "execute: error executing batch" err="error listing AWS instances: operation error EC2: DescribeInstances, get identity: get credentials: request canceled, context deadline exceeded"
ebs-plugin E0314 08:16:48.055711 1 batcher.go:161] "execute: error executing batch" err="error listing AWS instances: operation error EC2: DescribeInstances, get identity: get credentials: request canceled, context deadline exceeded"
csi-attacher I0314 07:57:31.394845 1 csi_handler.go:243] "Error processing" VolumeAttachment="csi-id" err="failed to detach: rpc error: code = Internal desc = Could not detach volume \"vol-id\" from node \"i-id\": error listing AWS instances: operation error EC2: DescribeInstances, get identity: get credentials: request canceled, context deadline exceeded"
I created another EKS with the exact same configuration in a separate VPC, and everything works fine there. What could be the problem?
Cluster information:
Kubernetes version: 1.30
Cloud being used: AWS (EKS)
Installation method: Terraform with AWS EKS module
Host OS: Amazon Linux 2023.6.20250218
CNI and version: 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni-init:v1.19.0-eksbuild.1
CRI and version: containerd://1.7.25