Hi all, I have recently discovered microk8s and I think it is wonderful for development and testing purposes, great work!
I have created an EC2 instance in AWS, with Ubuntu 18.04, and installed microk8s. I have also created a repository in ECR and uploaded an image to it. Then created a role to allow the EC2 instance to access the repository; the role works fine because I can do aws ecr get-login --region region --no-include-email
.
Now I launch a pod using the image in the ECR repository, but the image pull fails.
How can I configure microk8s to access the ECR repository?
OK, finally got it. The first tests were using --cloud-provider=aws
, but this is not appliable for microk8s (I think that parameter is meant to be used with full k8s clusters). After more research, these are the steps to make Microk8s authenticate to AWS ECR, in the most elegant way. The first thing to do is to create a role, and associate the policy AmazonEC2ContainerRegistryReadOnly
with the role. Then, associate the role with the EC2 instance. Test this works running this command in the instance:
aws ecr get-login --no-include-email --region xxxx
After that, create a job yaml definition (ecr-cred-refresh.yml
):
apiVersion: batch/v1
kind: CronJob
metadata:
name: ecr-cred-helper
spec:
concurrencyPolicy: Allow
schedule: 0 */6 * * *
failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 3
suspend: false
jobTemplate:
spec:
template:
spec:
containers:
- command:
- /bin/sh
- -c
- |-
NAMESPACE=awx
SERVICE_ACCOUNT=awx
ACCOUNT=$(aws sts get-caller-identity --query 'Account' --output text)
REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | python -c "import json,sys; print(json.loads(sys.stdin.read())['region'])")
SECRET_NAME=${REGION}-ecr-registry
EMAIL=anymail.doesnt.matter@email.com
TOKEN=$(aws ecr get-login --region ${REGION} --registry-ids ${ACCOUNT} | cut -d' ' -f6)
echo "ENV variables setup done."
kubectl -n ${NAMESPACE} delete secret --ignore-not-found $SECRET_NAME
kubectl -n ${NAMESPACE} create secret docker-registry $SECRET_NAME \
--docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--docker-email="${EMAIL}"
echo "Secret created by name $SECRET_NAME"
kubectl -n ${NAMESPACE} patch serviceaccount ${SERVICE_ACCOUNT} -p '{"imagePullSecrets":[{"name":"'$SECRET_NAME'"}]}'
echo "All done."
image: odaniait/aws-kubectl:latest
imagePullPolicy: IfNotPresent
name: ecr-cred-helper
resources: {}
securityContext:
capabilities: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: Default
hostNetwork: true
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
And apply it:
kubectl -n awx apply -f ecr-cred-refresh.yml
Change the variables NAMESPACE, SERVICE_ACCOUNT, ACCOUNT and REGION if needed. This will refresh the token, create a secret and associate it with a service account. Then, launch the pod using that service account. You could also modify the script to associate the secret with the default service account.
To force the token refresh, run these commands:
JOB_NAME="manual-$(date --utc +%Y%m%d-%H%M%S)"
kubectl -n awx create job --from=cronjob/ecr-cred-helper ${JOB_NAME}
kubectl -n awx wait --for=condition=complete job.batch/${JOB_NAME}
kubectl -n awx logs job.batch/${JOB_NAME}
Reference: How to configure and use AWS ECR with kubernetes & Rancher2.0
1 Like