Always get "exceeded its progress deadline" when trying to push a pod to Kubernetes kluster

I have tried to deploy my application to a Kubernetes kluster via DevOps Azure. Judging by the log, it seems everything is working as expected except the last step, to actually get the container to the kluster.

The reason for this can be seen below in the logs, primarily error: deployment "forsta-integration-service" exceeded its progress deadline and ##[error]Rollout status check failed.. The reason for exceeded its progress deadline I really believe is because I for some reason do not preform createSecret, but I have followed this tutorial and he do not set the key anywhere else and have a hard time finding information on how to do it properly.

I have tripple checked so that all the names are correct so I am very confused. If someone know what the problem may be I would really appreciate it!

Cluster information:

Kubernetes version: v1.26.0
Cloud being used: Azure
Installation method: Azure Pipeline
Host OS: Ubuntu

azure-pipeline.yml

trigger:
  - master

resources:
  - repo: self

variables:
  imageRepo: forsta-integration-service
  tag: '$(Build.BuildId)'

stages:
  - stage: Build
    displayName: Build image
    jobs:
      - job: Build
        displayName: Build
        pool:
          vmImage: ubuntu-latest
        steps:
          - task: Docker@2
            displayName: Build and Push image
            inputs:
              containerRegistry: 'container-registry-dev'
              repository: '$(imageRepo)'
              command: 'buildAndPush'
              Dockerfile: '$(Build.SourcesDirectory)/Dockerfile'
              tags: |
                $(tag)
          - task: PublishPipelineArtifact@1
            inputs:
              targetPath: '$(Pipeline.Workspace)/s/kubernetes'
              artifact: 'manifests'
              publishLocation: 'pipeline'
  - stage: Deploy
    displayName: Deploy to Dev
    dependsOn: Build
    variables:
      acrsecret: samberadev
      acrdevurl: 'samberadev.azurecr.io'
      replicaNo: 3
    jobs:
      - deployment: Deploy
        displayName: Deploy to AKS
        environment: 'k8sdev.default'
        pool:
          vmImage: ubuntu-latest
        strategy:
          runOnce:
            deploy:
              steps:
                - task: DownloadPipelineArtifact@2
                  inputs:
                    buildType: 'current'
                    artifactName: 'manifests'
                    targetPath: '$(Pipeline.Workspace)/manifests'
                - task: KubernetesManifest@0
                  inputs:
                    action: 'createSecret'
                    namespace: 'default'
                    secretType: 'dockerRegistry'
                    secretName: '$(acrsecret)'
                    dockerRegistryEndpoint: 'container-registry-dev'
                - task: KubernetesManifest@0
                  inputs:
                    action: 'deploy'
                    kubernetesServiceConnection: 'K8S_forsta_integration_service_dev-default'
                    namespace: 'default'
                    manifests: |
                      $(Pipeline.Workspace)/manifests/deployment.yml
                      $(Pipeline.Workspace)/manifests/service.yml
                    containers: '$(acrdevurl)/$(imageRepo):$(tag)'
                    imagePullSecrets: '$(acrsecret)'

deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: forsta-integration-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: forsta-integration-service
  template:
    metadata:
      labels:
        app: forsta-integration-service
    spec:
      containers:
        - name: forsta-integration-service
          image: "sambera.azurecr.io/forsta-integration-service"
          ports:
            - containerPort: 80

service.yml

apiVersion: v1
kind: Service
metadata:
  name: forsta-integration-service
spec:
  type: LoadBalancer
  ports:
    - port: 80
  selector:
    app: forsta-integration-service

Error Log

2022-12-26T12:29:11.6683000Z ##[section]Starting: KubernetesManifest2
2022-12-26T12:29:11.6693156Z ==============================================================================
2022-12-26T12:29:11.6693857Z Task         : Deploy to Kubernetes
2022-12-26T12:29:11.6694391Z Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
2022-12-26T12:29:11.6694875Z Version      : 0.212.0
2022-12-26T12:29:11.6695182Z Author       : Microsoft Corporation
2022-12-26T12:29:11.6695552Z Help         : https://aka.ms/azpipes-k8s-manifest-tsg
2022-12-26T12:29:11.6695981Z ==============================================================================
2022-12-26T12:29:12.0218958Z ==============================================================================
2022-12-26T12:29:12.0220037Z 			Kubectl Client Version: v1.26.0
2022-12-26T12:29:12.0220560Z 			Kubectl Server Version: v1.23.12
2022-12-26T12:29:12.0221486Z ==============================================================================
2022-12-26T12:29:12.0316747Z [command]/usr/local/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_forsta-integration-service_1672057752029,/home/vsts/work/_temp/Service_forsta-integration-service_1672057752029 --namespace default
2022-12-26T12:29:12.2735388Z deployment.apps/forsta-integration-service configured
2022-12-26T12:29:12.2736891Z service/forsta-integration-service unchanged
2022-12-26T12:29:12.2843963Z [command]/usr/local/bin/kubectl rollout status Deployment/forsta-integration-service --timeout 0s --namespace default
2022-12-26T12:39:13.0510558Z Waiting for deployment "forsta-integration-service" rollout to finish: 1 old replicas are pending termination...
2022-12-26T12:39:13.0523544Z error: deployment "forsta-integration-service" exceeded its progress deadline
2022-12-26T12:39:13.0555762Z ##[error]Error: error: deployment "forsta-integration-service" exceeded its progress deadline
2022-12-26T12:39:13.0571784Z [command]/usr/local/bin/kubectl describe Deployment forsta-integration-service --namespace default
2022-12-26T12:39:13.3271355Z Name:                   forsta-integration-service
2022-12-26T12:39:13.3272807Z Namespace:              default
2022-12-26T12:39:13.3273843Z CreationTimestamp:      Wed, 21 Dec 2022 14:48:58 +0000
2022-12-26T12:39:13.3274539Z Labels:                 <none>
2022-12-26T12:39:13.3275220Z Annotations:            deployment.kubernetes.io/revision: 5
2022-12-26T12:39:13.3276053Z Selector:               app=forsta-integration-service
2022-12-26T12:39:13.3276801Z Replicas:               1 desired | 1 updated | 2 total | 0 available | 2 unavailable
2022-12-26T12:39:13.3277510Z StrategyType:           RollingUpdate
2022-12-26T12:39:13.3278141Z MinReadySeconds:        0
2022-12-26T12:39:13.3278805Z RollingUpdateStrategy:  25% max unavailable, 25% max surge
2022-12-26T12:39:13.3279476Z Pod Template:
2022-12-26T12:39:13.3280155Z   Labels:  app=forsta-integration-service
2022-12-26T12:39:13.3280781Z   Containers:
2022-12-26T12:39:13.3281421Z    forsta-integration-service:
2022-12-26T12:39:13.3282207Z     Image:        sambera.azurecr.io/forsta-integration-service
2022-12-26T12:39:13.3282865Z     Port:         80/TCP
2022-12-26T12:39:13.3283448Z     Host Port:    0/TCP
2022-12-26T12:39:13.3284019Z     Environment:  <none>
2022-12-26T12:39:13.3284585Z     Mounts:       <none>
2022-12-26T12:39:13.3285160Z   Volumes:        <none>
2022-12-26T12:39:13.3285712Z Conditions:
2022-12-26T12:39:13.3286276Z   Type           Status  Reason
2022-12-26T12:39:13.3286913Z   ----           ------  ------
2022-12-26T12:39:13.3287574Z   Available      False   MinimumReplicasUnavailable
2022-12-26T12:39:13.3288266Z   Progressing    False   ProgressDeadlineExceeded
2022-12-26T12:39:13.3289135Z OldReplicaSets:  forsta-integration-service-7bf55cf6c5 (1/1 replicas created)
2022-12-26T12:39:13.3290058Z NewReplicaSet:   forsta-integration-service-6f449fb895 (1/1 replicas created)
2022-12-26T12:39:13.3291129Z Events:
2022-12-26T12:39:13.3291737Z   Type    Reason             Age   From                   Message
2022-12-26T12:39:13.3294355Z   ----    ------             ----  ----                   -------
2022-12-26T12:39:13.3295184Z   Normal  ScalingReplicaSet  53m   deployment-controller  Scaled down replica set forsta-integration-service-7964659bfc to 0
2022-12-26T12:39:13.3296113Z   Normal  ScalingReplicaSet  53m   deployment-controller  Scaled up replica set forsta-integration-service-7bf55cf6c5 to 1
2022-12-26T12:39:13.3297036Z   Normal  ScalingReplicaSet  10m   deployment-controller  Scaled down replica set forsta-integration-service-69c7b79768 to 0
2022-12-26T12:39:13.3297965Z   Normal  ScalingReplicaSet  10m   deployment-controller  Scaled up replica set forsta-integration-service-6f449fb895 to 1
2022-12-26T12:39:13.3298900Z For more information, go to https://dev.azure.com/kantarsifo/forsta-integration-service/_environments/1/providers/kubernetes/1
2022-12-26T12:39:13.3308248Z [command]/usr/local/bin/kubectl get service/forsta-integration-service -o json --namespace default
2022-12-26T12:39:13.4203802Z {
2022-12-26T12:39:13.4221878Z     "apiVersion": "v1",
2022-12-26T12:39:13.4280108Z     "kind": "Service",
2022-12-26T12:39:13.4280809Z     "metadata": {
2022-12-26T12:39:13.4281272Z         "annotations": {
2022-12-26T12:39:13.4282754Z             "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"forsta-integration-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":80}],\"selector\":{\"app\":\"forsta-integration-service\"},\"type\":\"LoadBalancer\"}}\n"
2022-12-26T12:39:13.4283691Z         },
2022-12-26T12:39:13.4284269Z         "creationTimestamp": "2022-12-21T14:34:38Z",
2022-12-26T12:39:13.4284826Z         "finalizers": [
2022-12-26T12:39:13.4285402Z             "service.kubernetes.io/load-balancer-cleanup"
2022-12-26T12:39:13.4285901Z         ],
2022-12-26T12:39:13.4286447Z         "name": "forsta-integration-service",
2022-12-26T12:39:13.4286946Z         "namespace": "default",
2022-12-26T12:39:13.4287435Z         "resourceVersion": "720755",
2022-12-26T12:39:13.4288035Z         "uid": "b5e8f5dc-8b2a-40a2-9747-6e207b25a6aa"
2022-12-26T12:39:13.4288532Z     },
2022-12-26T12:39:13.4288937Z     "spec": {
2022-12-26T12:39:13.4289414Z         "allocateLoadBalancerNodePorts": true,
2022-12-26T12:39:13.4289940Z         "clusterIP": "10.0.195.208",
2022-12-26T12:39:13.4290408Z         "clusterIPs": [
2022-12-26T12:39:13.4290853Z             "10.0.195.208"
2022-12-26T12:39:13.4291272Z         ],
2022-12-26T12:39:13.4291750Z         "externalTrafficPolicy": "Cluster",
2022-12-26T12:39:13.4292267Z         "internalTrafficPolicy": "Cluster",
2022-12-26T12:39:13.4292743Z         "ipFamilies": [
2022-12-26T12:39:13.4293167Z             "IPv4"
2022-12-26T12:39:13.4293886Z         ],
2022-12-26T12:39:13.4294343Z         "ipFamilyPolicy": "SingleStack",
2022-12-26T12:39:13.4294817Z         "ports": [
2022-12-26T12:39:13.4295221Z             {
2022-12-26T12:39:13.4295669Z                 "nodePort": 31683,
2022-12-26T12:39:13.4296130Z                 "port": 80,
2022-12-26T12:39:13.4296577Z                 "protocol": "TCP",
2022-12-26T12:39:13.4297045Z                 "targetPort": 80
2022-12-26T12:39:13.4297471Z             }
2022-12-26T12:39:13.4297861Z         ],
2022-12-26T12:39:13.4298273Z         "selector": {
2022-12-26T12:39:13.4298835Z             "app": "forsta-integration-service"
2022-12-26T12:39:13.4299296Z         },
2022-12-26T12:39:13.4299734Z         "sessionAffinity": "None",
2022-12-26T12:39:13.4300219Z         "type": "LoadBalancer"
2022-12-26T12:39:13.4300662Z     },
2022-12-26T12:39:13.4301065Z     "status": {
2022-12-26T12:39:13.4301496Z         "loadBalancer": {
2022-12-26T12:39:13.4301930Z             "ingress": [
2022-12-26T12:39:13.4302356Z                 {
2022-12-26T12:39:13.4302795Z                     "ip": "20.105.66.233"
2022-12-26T12:39:13.4303436Z                 }
2022-12-26T12:39:13.4304048Z             ]
2022-12-26T12:39:13.4304373Z         }
2022-12-26T12:39:13.4304916Z     }
2022-12-26T12:39:13.4305186Z }
2022-12-26T12:39:13.4306138Z service forsta-integration-service external IP is 20.105.66.233
2022-12-26T12:39:13.4310088Z ##[error]Rollout status check failed.
2022-12-26T12:39:13.4323352Z ##[section]Finishing: KubernetesManifest2

From the error message “deployment “forsta-integration-service” exceeded its progress deadline” it appears to me that the deployment is taking too long to complete, and the Kubernetes system is marking it as failed. This is usually caused by a problem with the resources allocated to the deployment, such as not having enough CPU or memory resources available.

Based on your logs, I think you might be missing a step to create the secret object in your Kubernetes cluster that’s required to access the container registry.

In the pipeline you have the step - task: KubernetesManifest@0 with the action: 'createSecret' but this task doesn’t contains the neccessary inputs like secretName, acrSecret etc. You need to create the Secret in the Kubernetes cluster that holds the credentials to access the container registry, then reference that Secret in the deployment definition. You also need to configure the pipeline’s service connection with the necessary access and permissions to create and modify Kubernetes resources in your cluster.

You should include the following inputs with the task of createSecret :

        - task: KubernetesManifest@0
          inputs:
            action: 'createSecret'
            namespace: 'default'
            secretType: 'dockerRegistry'
            secretName: '$(acrsecret)'
            dockerRegistryEndpoint: 'container-registry-dev'

Can you try if that works and let me know?