Is there a way to keep Succeeded pod for longer even after the node is recycled

General Discussions
1

Cluster information:

Kubernetes version: 1.29, 1.30
Cloud being used: AWS - EKS

Summary:
Is there a way to keep the pod associated with a job longer once the pod succeeds?

Details:
We are using Spinnaker RunJobs, which creates a Kubernetes Job. Spinnaker continues to poll for the existence of the job, and once the job is completed. Spinnaker polls for the log via command
kubectl -n <NAMESPACE> logs job/<JOB_NAME>
This works fine if the pod corresponding to the job exists on the Kubernetes cluster.
But when the pod is deleted, the above command returns

error: timed out waiting for the condition

Though spinnaker RunJobs works most of the time.
But we patch our Kubernetes cluster every week, and in some cases, as soon as the pod associated with the Job succeeds,
Node recycling occurs, and the pod is no longer available; the Spinnaker pipeline fails with the same error: timed out waiting for the condition.

Steps to reproduce

  1. Create a hello-world job and apply
    1.kubectl apply -f job.yaml
  2. kubectl logs job/hello-world
    1.Logs would be shown
  3. kubectl get pod and find the hello-world job pod.
    1.Get the POD_NAME
  4. Delete the pod - kubectl delete pod <POD_NAME>
    1.pod is deleted
  5. Run kubectl logs job/hello-world
    1.Kubernetes would return
    2.error: timed out waiting for the condition
1 Like