Init side car containers termination sequence

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

as per the K8S documentation which is quoted below , the init side car container (ie restart policy is Always) will be shutting down the last if termination grace period is not over which is 30 seconds . But what I’m seeing is it is getting terminated as same as rest all containers , do you know the reason for it please.

Upon Pod termination, the kubelet postpones terminating sidecar containers until the main application container has fully stopped. The sidecar containers are then shut down in the opposite order of their appearance in the Pod specification. This approach ensures that the sidecars remain operational, supporting other containers within the Pod, until their service is no longer required

Expected : init side car will be terminated at the last after all main containers are gets terminated.

image
image819×449 10.4 KB

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-prd-dpd
  labels:
    name: myapp-prd-dpd
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      restartPolicy: Always
      terminationGracePeriodSeconds: 60
      initContainers: 
        - name: init-myservice 
          image: busybox 
          restartPolicy: Always # here restart policy is always , we can't set on failure or never
          command: ['sh', '-c', 'tail -F /var/tmp/logs.txt']
          volumeMounts:
          - name: app-config
            mountPath: /var/tmp
      containers:
        - name: nginx-container
          image: nginx:latest
          ports:
          - containerPort: 80
          volumeMounts:
          - name: app-config 
            mountPath: /var/tmp
          command: ['sh', '-c', 'while true; do echo "logging" >> /var/tmp/logs.txt; sleep 1; done']

        - name: second-container
          image: busybox:latest
          command: [ "sh", "-c", "while true; do echo 'Secondary container running'; sleep 3600; done" ]
      volumes: 
        - name: app-config 
          emptyDir: {}

Cluster information:

Kubernetes version:
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.32.0
Cloud being used: (put bare-metal if not on a public cloud)
Installation method:
Host OS:
CNI and version:
CRI and version:

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

2

Please see the Kubernetes events and you can see it is init side car container is getting terminated first.

Name:             myapp-prd-dpd-6566cf5b84-mxqmt.181b231edc759fdf
Namespace:        default
Labels:           <none>
Annotations:      <none>
API Version:      v1
Count:            1
Event Time:       <nil>
First Timestamp:  2025-01-16T09:44:56Z
Involved Object:
  API Version:       v1
  Field Path:        spec.containers{second-container}
  Kind:              Pod
  Name:              myapp-prd-dpd-6566cf5b84-mxqmt
  Namespace:         default
  Resource Version:  3986
  UID:               1a35d3dc-0ce2-47da-8956-cb9ae7a44a76
Kind:                Event
Last Timestamp:      2025-01-16T09:44:56Z
Message:             Started container second-container
Metadata:
  Creation Timestamp:  2025-01-16T09:44:56Z
  Resource Version:    4138
  UID:                 21bc353c-13ca-4953-a99d-114d6694d394
Reason:                Started
Reporting Component:   kubelet
Reporting Instance:    node01
Source:
  Component:  kubelet
  Host:       node01
Type:         Normal
Events:       <none>

Name:             myapp-prd-dpd-6566cf5b84-mxqmt.181b2404323d09d7
Namespace:        default
Labels:           <none>
Annotations:      <none>
API Version:      v1
Count:            1
Event Time:       <nil>
First Timestamp:  2025-01-16T10:01:21Z
Involved Object:
  API Version:       v1
  Field Path:        spec.initContainers{init-myservice}
  Kind:              Pod
  Name:              myapp-prd-dpd-6566cf5b84-mxqmt
  Namespace:         default
  Resource Version:  3986
  UID:               1a35d3dc-0ce2-47da-8956-cb9ae7a44a76
Kind:                Event
Last Timestamp:      2025-01-16T10:01:21Z
Message:             Stopping container init-myservice
Metadata:
  Creation Timestamp:  2025-01-16T10:01:21Z
  Resource Version:    5673
  UID:                 25b15cb4-5189-4a3e-bd5c-fa3e2ef08db0
Reason:                Killing
Reporting Component:   kubelet
Reporting Instance:    node01
Source:
  Component:  kubelet
  Host:       node01
Type:         Normal
Events:       <none>

Name:             myapp-prd-dpd-6566cf5b84-mxqmt.181b2404323d3065
Namespace:        default
Labels:           <none>
Annotations:      <none>
API Version:      v1
Count:            1
Event Time:       <nil>
First Timestamp:  2025-01-16T10:01:21Z
Involved Object:
  API Version:       v1
  Field Path:        spec.containers{second-container}
  Kind:              Pod
  Name:              myapp-prd-dpd-6566cf5b84-mxqmt
  Namespace:         default
  Resource Version:  3986
  UID:               1a35d3dc-0ce2-47da-8956-cb9ae7a44a76
Kind:                Event
Last Timestamp:      2025-01-16T10:01:21Z
Message:             Stopping container second-container
Metadata:
  Creation Timestamp:  2025-01-16T10:01:21Z
  Resource Version:    5674
  UID:                 3143749b-84c7-4e60-b8e6-1ae910c1c1ad
Reason:                Killing
Reporting Component:   kubelet
Reporting Instance:    node01
Source:
  Component:  kubelet
  Host:       node01
Type:         Normal
Events:       <none>

Name:             myapp-prd-dpd-6566cf5b84-mxqmt.181b2404323dd47b
Namespace:        default
Labels:           <none>
Annotations:      <none>
API Version:      v1
Count:            1
Event Time:       <nil>
First Timestamp:  2025-01-16T10:01:21Z
Involved Object:
  API Version:       v1
  Field Path:        spec.containers{nginx-container}
  Kind:              Pod
  Name:              myapp-prd-dpd-6566cf5b84-mxqmt
  Namespace:         default
  Resource Version:  3986
  UID:               1a35d3dc-0ce2-47da-8956-cb9ae7a44a76
Kind:                Event
Last Timestamp:      2025-01-16T10:01:21Z
Message:             Stopping container nginx-container
Metadata:
  Creation Timestamp:  2025-01-16T10:01:21Z
  Resource Version:    5675
  UID:                 91479118-6b9a-46a0-8b02-753e84b5f501
Reason:                Killing
Reporting Component:   kubelet
Reporting Instance:    node01
Source:
  Component:  kubelet
  Host:       node01
Type:         Normal
Events:       <none>
3

Init Containers: Init containers are designed to run before the main application containers start. They complete their execution and exit. Once exited, they are no longer running when the main containers are running. The lifecycle of init containers is separate from the lifecycle of the main containers.
Restart Policy: The restart policy of init containers is implicitly OnFailure and cannot be set to Always. In your YAML, specifying restartPolicy: Always for the init container is ignored by Kubernetes.

Once an init container finishes its task, it does not keep running to support the main containers.

Sidecar Containers: For functionality that runs alongside the main application containers (e.g., logging or monitoring), use sidecar containers, not init containers. Sidecars remain running for the lifecycle of the Pod and can be shut down after the main containers.

Fix:

Move the init-myservice container from initContainers to containers in your Pod spec.

spec:
    terminationGracePeriodSeconds: 60
    containers:
      - name: init-myservice
        image: busybox
        command: ['sh', '-c', 'tail -F /var/tmp/logs.txt']
        volumeMounts:
        - name: app-config
          mountPath: /var/tmp
      - name: nginx-container
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: app-config
          mountPath: /var/tmp
        command: ['sh', '-c', 'while true; do echo "logging" >> /var/tmp/logs.txt; sleep 1; done']
      - name: second-container
        image: busybox:latest
        command: [ "sh", "-c", "while true; do echo 'Secondary container running'; sleep 3600; done" ]