Ideas? Autoscaling with cache volumes that should not be deleted

General Discussions
1

Hi,

I have a use case where every pod would require it’s own cache volume,
but when the pod is deleted, I do not want to delete this cache volume, I want to make use of all the files that have been downloaded in them.
I want to be able to autoscale my pods.
Furthermore, the caches are interchangeable (but volume to pod interchangeability is not a requirement by any means).
The cache volumes contain very similar data (depending on what request a pod received, it will put stuff in that cache, e.g. if every pod received the exact same requests, the contents would be the same, if required data is not in cache, pod will download it into cache)
These caches have a small amount of data, but it is constantly changing (and slightly growing).

The pods cannot share a RWX cache.

I have been thinking about this for many weeks and cannot figure it out. Maybe i need to drop the idea of autoscaling and just have a fixed amount of pods each with their cache volume…

What would you do?

e.g. Would it be possible to have a pool of cache volumes (equal to max replicas set on hpa), that pods can grab a cache from when they are spawned?

e.g. #2 Should I just use a statefulset and forget about autoscaling?

TIA,
Naseem

2

Naseem_Ullah

    January 10

Hi,

I have a use case where every pod would require it’s own cache volume,

but when the pod is deleted, I do not want to delete this cache volume, I want to make use of all the files that have been downloaded in them.

I want to be able to autoscale my pods.

Furthermore, the caches are interchangeable (but volume to pod interchangeability is not a requirement by any means).

The cache volumes contain very similar data (depending on what request a pod received, it will put stuff in that cache, e.g. if every pod received the exact same requests, the contents would be the same, if required data is not in cache, pod will download it into cache)

These caches have a small amount of data, but it is constantly changing (and slightly growing).

The pods cannot share a RWX cache.

I have been thinking about this for many weeks and cannot figure it out. Maybe i need to drop the idea of autoscaling and just have a fixed amount of pods each with their cache volume…

What would you do?

Fundamentally, persistent volumes and auto-scaling are mutuallyincompatible precisely because the policy space for the intersection is so unclear. As you suggest, we would need a concept of a pool of volume claims that could be doled out. The good news is tha this is not an intractable problem, just not solved by the core of the system (yet?)

I see 2 answers, bith require some work on your end.

  1. Define your own workload object (via CRD / operators). Use volume snapshots (which are not GA yet). Take a snapshot of a populated cache periodically. Whenever you make a new instance of your thing, create anew PVC which restores from the latest snapshot.

  2. Define your own workload object and a concept of volume pool. When you create a new thing, allocate a claim from your pool.

Both of these allow you to define the EXACT policy of how to reuse volumes.

e.g. Would it be possible to have a pool of cache volumes (equal to max replicas set on hpa), that pods can grab a cache from when they are spawned?

HPA->Deployment means that the pods are all identical, which means they will use the same PVClaim. That is not what you want. You need your own workload controller. I think HPA can target other resources, but I am not sure here…

1 Like
5

HPA + VolumeSnapshot should work for my use case. volume snapshots are not available yet in (the current stable version of) GKE.

FWIW today these hpa managed replicas all have an init container that pulls data from GCS and puts it in an emptyDir