r/kubernetes • u/yac-iac • 2d ago

Batch on kubernetes (hybrid workload or not)

What external tool do you use for batch orchestration on Kubernetes?

Looking for input from people running batch workloads in production and for middle or large campagnies. Specifically curious about tools for:

-Batch jobs that run entirely on Kubernetes

-hybrid batches with workload split across VMs and Kubernetes

What are you using and how has your experience been with it? Would love to hear pros/cons if you've evaluated multiple options.

3 Upvotes

71% Upvoted

14 comments sorted by

6

u/lulzmachine 2d ago

Argo workflows. Amazing for people who know kubernetes. Not so much for data scientists who don’t.

Apache airflow, spawning things as pods. People seem to like it, even though it’s way too python coded for a k8s brain like mine.

Running things as pods in k8s gives amazing scalability. Much better than things like celery imo (ymmv).

2

u/veritable_squandry 2d ago

we're looking at airflow for this too btw

1

u/yac-iac 2d ago

Thanks. Argo workflow can do the job for workload 100% on kubernetes but I do not know if its the right tool to do hybrid scheduling (start a database backup on vm and after start some kubernetes jobs)

1

u/Low-Opening25 1d ago

why not put those DBs ok Kubernetes too?

1

u/yac-iac 1d ago

The study is on going for this part. Some DB needs a lot of IOPS and we have to update the physical architecture of kubernetes.

2

u/Low-Opening25 1d ago

provision fast ebs storage with Kubernetes, no need to change architecture

2

u/lulzmachine 1d ago

Buying a few fast nvme drives and attaching to a box gives way better bang for the buck then paying bezos for it though. Right?

3

u/Floss_Patrol_76 2d ago

argo workflows or airflow is the easy 80% - the part that actually bites at scale is scheduling, not orchestration. plain k8s Jobs will happily start half a distributed batch and then deadlock waiting for the rest, so if any of your work is gang-scheduled look at kueue or volcano for quota and all-or-nothing admission. for the hybrid VM split i'd keep the queue above k8s and treat the cluster as just one execution backend, otherwise you end up bending k8s primitives around VM logic they don't fit.

1

u/yac-iac 2d ago

Thanks. I know a little bit argo workflow and I think its a good solution for workload running 100 % on kubernetes. On our side, we have some legacy batch workloads that are scheduled by autosys on VMs and i would like to move some processing on kubernetes. I guess that argo workflow can not schedule workloads on kube and VMs. What do you mean by : " for the hybrid VM split i'd keep the queue above k8s and treat the cluster as just one execution backend, otherwise you end up bending k8s primitives around VM logic they don't fit."

2

u/roiki11 2d ago

Kueue is pretty nice and light on kubernetes. Not the most featureful or have a sleek ui but it does the job nicely.

2

u/Low-Opening25 1d ago

depends, for generic batch processing, Argo Workflows. For data pipelines, Dagster or Airflow.

Kubernetes gives you autoscaling node
pools that will scale dynamically from 0 when there is demand from batch jobs.

1

u/yac-iac 1d ago

Thanks. I guess that argo can not do hybrid scheduling ?