Task queue systems like Celery provide a means to submit tasks to a pool of workers and poll for results to come back. I’m wondering if we still need this kind of system and would like to hear your thoughts. Here’s my alternative proposal:
Create a service that simply performs the long-running/heavy-lifting task directly in the request handler.
Use a service mesh or reverse proxy to perform request buffering/retry timeouts
Autoscale the service based on requests per second
Wouldn’t that work out to be somewhat equivalent? Am I overlooking anything significant?
I’ve used Celery before outside of K8s to automate background moving of large files for processing and enjoyed working with it.
I wouldn’t be a fan of long-running HTTP requests. Just because K8s can allow you to scale the service I would still think that pattern is bad.
Would you have implemented this way outside of K8s if you could autoscale VMs for example?
It sounds like you already have a Celery based system which is probably battle tested.
Why not just move that to K8s and scale Celery workers as workers as needed?
I’m mostly just thinking about it. It occurred to me that such a service can be seen as just another service, the only difference being that it would need to scale much sooner.
What I’m really in need of is a Celery like system that I can call from the browser, and await the results as they are being computed on a worker. I’m not aware of any system like that. I guess it would need to be based on websockets to get instant feedback once the job is done, so there is need of a type of intermediate service between the client and the workers…
Your idea is interesting, but I wonder if in the long run you’d end up reinventing something like celery anyway.
Celery describes itself as a distributed task queue. It provides multiple capabilities:
distributing workloads to workers (via external broker like rabbitmq, etc)
queuing up jobs that are not immediately assignable to workers (more jobs than workers)
a mechanism to return results from workers to the requester
job persistence
job state tracking (and I think it can retry jobs where the worker failed)
Your proposed alternative could work for some types of workloads, but I think it will fall short on tasks that are long running or when tasks arrive faster than you can scale up the number of handlers. There’s also some limit to how many handlers you can run. What process will scale down handlers?
Where will jobs be queued while waiting for a handler? I suppose a service mesh can accept some backlogged connections, but how many? Are those waiting jobs durable? Will they survive a restart of the mesh proxy, or the node?
Celery tasks are async, but I think http/grpc service mesh is mostly for synchronous calls. How long will the service mesh wait for a response from the handler before timing out?
What process or controller is going to autoscale your handlers? One way would be to send all requests through a single process so that requests/second can be counted or time waiting in queue is tracked, etc. Is that process/controller going to retain state for jobs not yet handled by workers? Will it store state so it can survive a restart w/o losing jobs?
Don’t forget day 2 operations, upgrading handlers, new attributes on jobs, new types of jobs.
but I think it will fall short on tasks that are long running or when tasks arrive faster than you can scale up the number of handlers.
In practise I agree, but for my purely theoretical argument, I assumed that we have the technology and technology to instantly and infinitely add and remove more handlers. Then why would a task ever need to wait/be queued? We can always instantly add a worker for an incoming request. Idle handlers can be spinned down after some amount of time has passed. Typical day 2 operations would then vanish, since we no longer need to treat API servers and background workers differently. That’s the idea, at least.
We use Celery inside K8s in my company; many specific queues, many specific workers, one pathology : some workers are often idle, occupying some resources for nothing, while some others are fully busy.
We are also questioning the relevance of Celery inside K8s. Celery knows nothing about the cluster state, so how could it schedule tasks fairly and efficiently on such a system ? Why having a blind task distribution system inside a bigger system having itself job scheduling capabilities ? We are quite perplex.
