Pod counts in jobs/deployments

General Discussions
1

Hi folks,

This conundrum has been doing my head in for days, it seems simple but isn’t and I’m trying not to have to deploy a whole workflow platform to deal with it.

All I want to do is run a set of containers which run a batch task, but the pods need to download a chunk of data unique to their environment. Statefulsets are perfect because the incremental hostname makes it easy to figure out which number of the batch it is, except for some reason it was deemed unacceptable to have statefulset pods complete, so instead it ends up in a cycle of doom…

Jobs, eh? How did that spec come to be without the ability to track hostname, or a variable of the “I’m pod 3 of 5” ilk?

Anyone got any bright ideas?

2

Without understanding your use-case completely, two options come to mind.

  1. Utilize a long-running pod that acts as a job controller that allocates the data to each job worker
  2. Leverage Metacontroller’s IndexedJob example
    https://github.com/GoogleCloudPlatform/metacontroller/tree/master/examples/indexedjob