AWS Load Balancer Controller Install FAILED - Unable to Connect to Kubernetes API Server(Self-Managed Cluster, Not EKS)

General Discussions
1
  • Kubernetes Version: v1.31.1
  • AWS Load Balancer Controller Helm Chart: aws-load-balancer-controller-1.11.0 (App version v2.11.0)
  • CNI Plugin: Calico (unsure of VXLAN mode)
  • Container Runtime: containerd
  • Instance Metadata Configuration (IMDSv2):
    • HttpTokens: required
    • HttpPutResponseHopLimit: 2
  • Pod CIDR: 10.233.64.0/18
  • Security Group Rules: All opened to 0.0.0.0/0 for both ingress and egress.

Problem:
The AWS Load Balancer Controller Pod fails to start properly. Logs show the following error:

{"level":"error","ts":"2025-01-06T01:09:33Z","logger":"setup","msg":"unable to create controller","controller":"Ingress","error":"Get \"https://10.233.0.1:443/apis/networking.k8s.io/v1\": dial tcp 10.233.0.1:443: i/o timeout"}

Additionally, when I inspect the ServiceAccount associated with the controller, it has no tokens or mountable secrets:

kubectl describe serviceaccount aws-load-balancer-controller -n kube-system
Name:                aws-load-balancer-controller
Namespace:           kube-system
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::002137xxxxxx:role/AWS_xxxxx_Role
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Debugging Steps Taken:

  1. IMDSv2 Configuration:
  • Verified that HttpPutResponseHopLimit is set to 2.
  • Confirmed the metadata options using the AWS CLI.
  1. Firewall/Security Groups:
  • Opened all traffic to 0.0.0.0/0 for both ingress and egress.
  1. Helm and Kubernetes Versions:
  • Helm version: v3.16.3
  • Kubernetes client and server versions: v1.31.1
  1. IAM Role:
  • Confirmed that the correct IAM Role is associated with the ServiceAccount and verified its trust relationship.

Installation Steps:
The AWS Load Balancer Controller was installed by following the official installation guide. The steps included:

  1. Tagging Subnets: Ensured that the subnets were properly tagged for auto-discovery.
  2. Configuring VPC CNI: Installed amazon-vpc-cni-k8s to allow Pods to acquire IPs from the VPC subnets.
  3. Setting Up IMDSv2:
aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2 --http-tokens required --region <region> --instance-id <instance-id>
  1. Configuring IAM Role: Attached the required IAM policies to the Kubernetes nodes.
  2. Opening Ports: Verified that port 9443 was open in the security group rules.
  3. Installing the Controller via Helm:
helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set region=ap-northeast-1 \
  --set vpcId=vpc-xxxxxxxxxxxxxxx \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

Questions:

  1. What could be causing the controller to time out when connecting to the Kubernetes API Server?
  2. Are there additional configurations required for the AWS Load Balancer Controller in a self-managed Kubernetes cluster (not EKS)?
  3. Could Calico’s VXLAN or BGP configuration (if applicable) be causing the issue? How can I verify whether VXLAN is being used?