I have a use case where every pod would require it’s own cache volume,
but when the pod is deleted, I do not want to delete this cache volume, I want to make use of all the files that have been downloaded in them.
I want to be able to autoscale my pods.
Furthermore, the caches are interchangeable (but volume to pod interchangeability is not a requirement by any means).
The cache volumes contain very similar data (depending on what request a pod received, it will put stuff in that cache, e.g. if every pod received the exact same requests, the contents would be the same, if required data is not in cache, pod will download it into cache)
These caches have a small amount of data, but it is constantly changing (and slightly growing).
The pods cannot share a RWX cache.
I have been thinking about this for many weeks and cannot figure it out. Maybe i need to drop the idea of autoscaling and just have a fixed amount of pods each with their cache volume…
What would you do?
e.g. Would it be possible to have a pool of cache volumes (equal to max replicas set on hpa), that pods can grab a cache from when they are spawned?
e.g. #2 Should I just use a statefulset and forget about autoscaling?