Hi guys! I am not new to kubernetes but I am not an expert either. Have a problem and was wondering if you have any suggestions as to the best way to move forward with it.
We have two services (each in their own container), a temporal worker and a dependent data science service (which can be memory-intensive). We have pre-processing in the former, hand off to latter and then post-processing in the former again. For each task, we want to be able to scale up the solution but in tandem (each new worker automatically gets a dedicated data science service) to avoid a noisy neighbor situation. We can scale using KEDA’s temporal scaler (targetQueueSize: “1”, for example) and I assume it would be as simple assigning both containers to the same pod. Is that correct?
The only problem being that if the data science service fails (out of memory, for example), the temporal worker goes with it… is there a different approach perhaps that would alleviate this?