Microk8s EC2 and ECR

General Discussions microk8s
1

Hi all, I have recently discovered microk8s and I think it is wonderful for development and testing purposes, great work!

I have created an EC2 instance in AWS, with Ubuntu 18.04, and installed microk8s. I have also created a repository in ECR and uploaded an image to it. Then created a role to allow the EC2 instance to access the repository; the role works fine because I can do aws ecr get-login --region region --no-include-email.

Now I launch a pod using the image in the ECR repository, but the image pull fails.

How can I configure microk8s to access the ECR repository?

2

/cc @kjackal

3

OK, finally got it. The first tests were using --cloud-provider=aws, but this is not appliable for microk8s (I think that parameter is meant to be used with full k8s clusters). After more research, these are the steps to make Microk8s authenticate to AWS ECR, in the most elegant way. The first thing to do is to create a role, and associate the policy AmazonEC2ContainerRegistryReadOnly with the role. Then, associate the role with the EC2 instance. Test this works running this command in the instance:

aws ecr get-login --no-include-email --region xxxx

After that, create a job yaml definition (ecr-cred-refresh.yml):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-cred-helper
spec:
  concurrencyPolicy: Allow
  schedule: 0 */6 * * *
  failedJobsHistoryLimit: 1
  successfulJobsHistoryLimit: 3
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - command:
            - /bin/sh
            - -c
            - |-
              NAMESPACE=awx
              SERVICE_ACCOUNT=awx
              ACCOUNT=$(aws sts get-caller-identity --query 'Account' --output text)
              REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | python -c "import json,sys; print(json.loads(sys.stdin.read())['region'])")
              SECRET_NAME=${REGION}-ecr-registry
              EMAIL=anymail.doesnt.matter@email.com
              TOKEN=$(aws ecr get-login --region ${REGION} --registry-ids ${ACCOUNT} | cut -d' ' -f6)
              echo "ENV variables setup done."
              kubectl -n ${NAMESPACE} delete secret --ignore-not-found $SECRET_NAME
              kubectl -n ${NAMESPACE} create secret docker-registry $SECRET_NAME \
              --docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
              --docker-username=AWS \
              --docker-password="${TOKEN}" \
              --docker-email="${EMAIL}"
              echo "Secret created by name $SECRET_NAME"
              kubectl -n ${NAMESPACE} patch serviceaccount ${SERVICE_ACCOUNT} -p '{"imagePullSecrets":[{"name":"'$SECRET_NAME'"}]}'
              echo "All done."
            image: odaniait/aws-kubectl:latest
            imagePullPolicy: IfNotPresent
            name: ecr-cred-helper
            resources: {}
            securityContext:
              capabilities: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: Default
          hostNetwork: true
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30

And apply it:

kubectl -n awx apply -f ecr-cred-refresh.yml

Change the variables NAMESPACE, SERVICE_ACCOUNT, ACCOUNT and REGION if needed. This will refresh the token, create a secret and associate it with a service account. Then, launch the pod using that service account. You could also modify the script to associate the secret with the default service account.

To force the token refresh, run these commands:

JOB_NAME="manual-$(date --utc +%Y%m%d-%H%M%S)"
kubectl -n awx create job --from=cronjob/ecr-cred-helper ${JOB_NAME}
kubectl -n awx wait --for=condition=complete job.batch/${JOB_NAME}
kubectl -n awx logs job.batch/${JOB_NAME}

Reference: How to configure and use AWS ECR with kubernetes & Rancher2.0

1 Like