Moving a Laravel Scheduler to Kubernetes with Non-idempotent Cron Jobs

General Discussions
1

We’re in the process of transitioning a Laravel application to Kubernetes using EKS and k9s. For the most part, the application seems to be a good fit. However, we’ve run into a challenge related to the Laravel scheduler.

Background and Challenges:

  1. Our Laravel application has a scheduler that relies on a cron job: php artisan scheduler:run.
  2. The scheduled jobs are not idempotent. This means that missing a job or running the same job multiple times could have serious negative impacts.
  3. Stopping a job midway is equally problematic.
  4. Due to business constraints and associated costs, any meaningful revising the code to change this behavior isn’t an option. We need to migrate with the code as it is or with minimal and safe changes.
  5. Kubernetes’ CronJob feature isn’t a viable solution because it requires jobs to be idempotent. We verified this in the official documentation.
  6. Our current workaround involves using a persistent EC2 instance outside of EKS. Instead of running php artisan scheduler:run, we execute docker run "php artisan scheduler:run..." via a cron job on this EC2. While this provides a reasonable compromise, it has drawbacks.
  7. Our main dissatisfaction is that part of the application now sits outside the EKS cluster. This demands a different CI/CD approach, additional access maintenance, and, fundamentally, a bifurcated infrastructure.

What I’m seeking

I’m seeking alternative solutions or best practices that could help us:

  • Maintain the non-idempotent nature of our jobs.
  • Keep our infrastructure uniform and managed within the EKS cluster.
  • Ensure the safe and reliable execution of our Laravel scheduler.

Any suggestions or insights on how we can tackle this would be greatly appreciated! Alternatively, reassurance that our present approach is the most prudent given our constraints would be equally valued. :slight_smile: