This one covers the other half: how we bend the default Kubernetes scheduler to keep GPU workloads in the cluster and spill only when it's full, using dynamic taints driven by a PromQL signal, the Descheduler, and Kueue.

Introduction In the previous post we showed how we built a system that lets us scale anywhere, using the Virtual Kubelet project.

But we skipped a big chunk of that system: how we schedule around the VK node to maximize cluster usage and minimize cost.

Some approaches we have seen modify the Kubernetes scheduler, by changing its configuration or by adding plugins as steps.

Others replace it entirely with schedulers like Volcano , Apache YuniKorn , or KAI-Scheduler .

None of these capture our business needs out of the box, so we took a different route: make the default Kubernetes scheduler work for us.

We could have written plugins or modified the default scheduler ourselves.

During the project, though, we noticed we did not need to.

With a bit of cleverness and a few tools built on top, the default scheduler could still do the job.

…