Lend and borrow resources across teams with AWS Batch’s quota management for SageMaker Training jobs

AWS added quota management to AWS Batch, letting administrators divide shared GPU capacity into quota shares for teams, projects or workloads.

• The company says customers managing multi-team training environments asked for a way to define per-team allocations, share idle capacity automatically and run priority workloads first.
• Each quota share carries capacity limits and a resource-sharing strategy: RESERVE keeps idle compute for the share, LEND lends it to others, and LEND_AND_BORROW both lends idle capacity and borrows when the share has work.
• A quota share acts as a virtual queue nested inside a job queue, and AWS Batch prioritizes shares that are under their limits as it iterates through them.
• When jobs arrive in a share that lent out capacity, cross-share preemption reclaims the resources, so administrators do not have to find and terminate borrowing jobs themselves.
• Priority-based in-share preemption can also be turned on to preempt lower-priority jobs in the SCHEDULED, STARTING or RUNNING state.
• Setting it up requires a Scheduling Policy with a quotaSharePolicy, a Service Environment specifying instance capacity, and a SageMaker Training job queue connected to both; the example uses the SageMaker Python SDK v3 and three quota shares each capped at one ml.g5.xlarge instance.