Ideas? Autoscaling with cache volumes that should not be deleted


#1

Hi,

I have a use case where every pod would require it’s own cache volume,
but when the pod is deleted, I do not want to delete this cache volume, I want to make use of all the files that have been downloaded in them.
I want to be able to autoscale my pods.
Furthermore, the caches are interchangeable (but volume to pod interchangeability is not a requirement by any means).
The cache volumes contain very similar data (depending on what request a pod received, it will put stuff in that cache, e.g. if every pod received the exact same requests, the contents would be the same, if required data is not in cache, pod will download it into cache)
These caches have a small amount of data, but it is constantly changing (and slightly growing).

The pods cannot share a RWX cache.

I have been thinking about this for many weeks and cannot figure it out. Maybe i need to drop the idea of autoscaling and just have a fixed amount of pods each with their cache volume…

What would you do?

e.g. Would it be possible to have a pool of cache volumes (equal to max replicas set on hpa), that pods can grab a cache from when they are spawned?

e.g. #2 Should I just use a statefulset and forget about autoscaling?

TIA,
Naseem


#2

Naseem_Ullah

    January 10

Hi,

I have a use case where every pod would require it’s own cache volume,

but when the pod is deleted, I do not want to delete this cache volume, I want to make use of all the files that have been downloaded in them.

I want to be able to autoscale my pods.

Furthermore, the caches are interchangeable (but volume to pod interchangeability is not a requirement by any means).

The cache volumes contain very similar data (depending on what request a pod received, it will put stuff in that cache, e.g. if every pod received the exact same requests, the contents would be the same, if required data is not in cache, pod will download it into cache)

These caches have a small amount of data, but it is constantly changing (and slightly growing).

The pods cannot share a RWX cache.

I have been thinking about this for many weeks and cannot figure it out. Maybe i need to drop the idea of autoscaling and just have a fixed amount of pods each with their cache volume…

What would you do?

Fundamentally, persistent volumes and auto-scaling are mutuallyincompatible precisely because the policy space for the intersection is so unclear. As you suggest, we would need a concept of a pool of volume claims that could be doled out. The good news is tha this is not an intractable problem, just not solved by the core of the system (yet?)

I see 2 answers, bith require some work on your end.

  1. Define your own workload object (via CRD / operators). Use volume snapshots (which are not GA yet). Take a snapshot of a populated cache periodically. Whenever you make a new instance of your thing, create anew PVC which restores from the latest snapshot.

  2. Define your own workload object and a concept of volume pool. When you create a new thing, allocate a claim from your pool.

Both of these allow you to define the EXACT policy of how to reuse volumes.

e.g. Would it be possible to have a pool of cache volumes (equal to max replicas set on hpa), that pods can grab a cache from when they are spawned?

HPA->Deployment means that the pods are all identical, which means they will use the same PVClaim. That is not what you want. You need your own workload controller. I think HPA can target other resources, but I am not sure here…


#3

Thank you so much Tim. I’m pretty unfamiliar with CRDs and Operators atm (according to your lightning talk at latest kubecon they are the future so I’d better ramp up!. At least now I know which direction to head towards to solve this problem.


#4

Could you please juste confirm if that by using either of the 2 approaches you mentioned, pod autoscaling would be possible? I’m thrown off by the closing comment. Do you mean horizontal pod autoscaling would be implemented via my own workload controller, not by the typical hpa object?