How to create aws-ebs-csi-driver with eks_blueprints_addons by Terraform?

General Discussions
1

I created AWS EBS CSI Driver addon with eks_blueprints_addons by Terraform:

    module "eks_cluster" {
      source  = "terraform-aws-modules/eks/aws"
      version = "~> 20.17"
      ...
    }

    module "eks_blueprints_addons" {
      source  = "aws-ia/eks-blueprints-addons/aws"
      version = "~> 1.1"

      cluster_name      = module.eks_cluster.cluster_name
      cluster_endpoint  = module.eks_cluster.cluster_endpoint
      cluster_version   = module.eks_cluster.cluster_version
      oidc_provider_arn = module.eks_cluster.oidc_provider_arn

      eks_addons = {
        aws-ebs-csi-driver = {
          most_recent = true
        }
      }
    }

When I deploy an app with PVC, I got this error:

Warning ProvisioningFailed 113s (x2 over 4m53s) ebs.csi.aws.com_ebs-csi-controller-c4bc5f559-k6fqp_5a4ed8cc-0085-4875-8070-87fceda36abf (combined from similar events): failed to provision volume with StorageClass "standard": rpc error: code = Internal desc = Could not create volume "pvc-f8f034de-5f7b-4ca6-bb0c-0c3e4be8026d": could not create volume in EC2: operation error EC2: CreateVolume, https response error StatusCode: 403, RequestID: 0d5d6201-4115-4cd9-bc99-201331b97450, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::111111111111:assumed-role/default-eks-node-group-2024071204291580710000000e/i-075c478df65c9bd58 is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:ap-northeast-1:111111111111:volume/* because no identity-based policy allows the ec2:CreateVolume action

Even I use this way got the same result:

    module "ebs_csi_driver_irsa" {
      source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
      version = "~> 5.20"

      role_name_prefix = "ebs-csi-driver-"

      attach_ebs_csi_policy = true

      oidc_providers = {
        main = {
          provider_arn               = module.eks_cluster.oidc_provider_arn
          namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
        }
      }

      tags = var.tags
    }

    module "eks_blueprints_addons" {
      source  = "aws-ia/eks-blueprints-addons/aws"
      version = "~> 1.1"

      cluster_name      = module.eks_cluster.cluster_name
      cluster_endpoint  = module.eks_cluster.cluster_endpoint
      cluster_version   = module.eks_cluster.cluster_version
      oidc_provider_arn = module.eks_cluster.oidc_provider_arn

      eks_addons = {
        aws-ebs-csi-driver = {
          most_recent              = true
          service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
        }
      }
    }

But if I do with eksctl, it works:

    eksctl create addon --name aws-ebs-csi-driver --cluster eks-test --service-account-role-arn arn:aws:iam::11111111:role/AmazonEKS_EBS_CSI_DriverRole --force

    eksctl create iamserviceaccount \
        --name ebs-csi-controller-sa \
        --namespace kube-system \
        --cluster eks-test \
        --role-name AmazonEKS_EBS_CSI_DriverRole \
        --role-only \
        --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
        --approve
2

From this article, it seems need to create an AmazonEKS_EBS_CSI_DriverRole and set to service_account_role_arn manually.

module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.12"

  cluster_name      = var.eks_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = var.eks_cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  eks_addons = {
      kube-proxy = {
        addon_version     = "v1.24.17-eksbuild.4"
        resolve_conflicts = "OVERWRITE"
      }
      coredns = {
        addon_version     = "v1.9.3-eksbuild.10"
        resolve_conflicts = "OVERWRITE"
      }
      aws-ebs-csi-driver = {
        addon_version            = "v1.26.0-eksbuild.1"
        resolve_conflicts        = "OVERWRITE"
        service_account_role_arn = "arn:aws:iam::${aws-account-id}:role/AmazonEKS_EBS_CSI_DriverRole"
      }
      snapshot-controller = {
        addon_version     = "v6.3.2-eksbuild.1"
        resolve_conflicts = "OVERWRITE"
      }
      vpc-cni = {
        addon_version = "v1.15.5-eksbuild.1"
        preserve      = true
        # terraform not happy with PRESERVE
        resolve_conflicts        = "NONE"
        service_account_role_arn = "arn:aws:iam::${aws-accounts-id}:role/AmazonEKSVPCCNIRole"
        configuration_values = jsonencode({
          env = {
            AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG = "true"
            ENI_CONFIG_LABEL_DEF               = "failure-domain.beta.kubernetes.io/zone"
          }
        })
      }
  }
  ...
}

resource "kubernetes_storage_class" "this" {
  metadata {
    name = "gp3"
    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }
  allow_volume_expansion = true
  storage_provisioner    = "ebs.csi.aws.com"
  reclaim_policy         = "Delete"
  volume_binding_mode    = "WaitForFirstConsumer"
  parameters = {
    type = "gp3"
  }
}

But why this way doesn’t work?

    module "ebs_csi_driver_irsa" {
      source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
      version = "~> 5.20"

      role_name_prefix = "ebs-csi-driver-"

      attach_ebs_csi_policy = true

      oidc_providers = {
        main = {
          provider_arn               = module.eks_cluster.oidc_provider_arn
          namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
        }
      }

      tags = var.tags
    }

    module "eks_blueprints_addons" {
      source  = "aws-ia/eks-blueprints-addons/aws"
      version = "~> 1.1"

      cluster_name      = module.eks_cluster.cluster_name
      cluster_endpoint  = module.eks_cluster.cluster_endpoint
      cluster_version   = module.eks_cluster.cluster_version
      oidc_provider_arn = module.eks_cluster.oidc_provider_arn

      eks_addons = {
        aws-ebs-csi-driver = {
          most_recent              = true
          service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
        }
      }
    }
3

The second time I created the full stack, it works with this way:

    module "ebs_csi_driver_irsa" {
      source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
      version = "~> 5.20"

      role_name_prefix = "ebs-csi-driver-"

      attach_ebs_csi_policy = true

      oidc_providers = {
        main = {
          provider_arn               = module.eks_cluster.oidc_provider_arn
          namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
        }
      }

      tags = var.tags
    }

    module "eks_blueprints_addons" {
      source  = "aws-ia/eks-blueprints-addons/aws"
      version = "~> 1.1"

      cluster_name      = module.eks_cluster.cluster_name
      cluster_endpoint  = module.eks_cluster.cluster_endpoint
      cluster_version   = module.eks_cluster.cluster_version
      oidc_provider_arn = module.eks_cluster.oidc_provider_arn

      eks_addons = {
        aws-ebs-csi-driver = {
          most_recent              = true
          service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
        }
      }
    }

The reason was I didn’t delete the old storageclass I created without an EBS CSI Driver IRSA.

The example is correct: