I’m trying to submit high volumes of one-off commands using Kubernetes. For this, I initially tried using k8s Jobs which seems to be the advertised feature for this use-case.
When I submit my tasks as 100 different k8s Jobs (with completions=1), it takes ~30s to finish. All jobs run the ‘true’ command (I can share the complete JobSpec, if needed).
But when I run submit the same tasks (true commands) directly as k8s pods, it’s much faster. Submitting 100 pods and waiting for them to finish takes just ~10s. 3x faster than Jobs!
My understanding is that Jobs also create Pods underneath (and do some more extra stuff). But, I’m unable to understand the huge performance difference in both the cases. What exactly is causing the delay in case of Jobs? Is there any configuration option I can tweak to improve the performance of k8s Jobs?
Cluster information:
Kubernetes version: v1.29.2
Cloud being used: on VM
Installation method: kind
Host OS: RedHat Enterprise 8.10
The test-cluster was created using kind. It has one control-place, and one worker. I’ve tried with bigger clusters with multiple nodes as well, but got similar results.