# Together AI — Auto repair for Together GPU Clusters: from GPU fault to healthy node

- Company: Together AI (together.ai)
- Announced: 2026-10-27
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.together.ai/blog/auto-repair-for-together-gpu-clusters
- Record: https://forck.live/items/28474-auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node
- Subject: Inference platform

Together AI has explained how automatic node remediation works on its GPU clusters, a capability it says customers and its internal inference platform have used to improve workload resiliency and cut downtime. The system runs passive health checks on every node, both inside the customer's VM and on the host underneath, so GPU faults visible from the guest and host or fabric faults visible from the host land in one place. Rather than keying on a single NotReady signal, which looks the same whether a node is broken or briefly unreachable, auto repair requires a failed health check behind every repair and attaches supporting evidence to each one. The platform recommends the lightest action that fixes a given fault and maps each signal to it, while customers decide per fault whether a repair waits for approval or runs unattended, including per-Xid consent inside the Xid group. Wait-for-idle controls hold a repair until the node is actually idle, and a built-in circuit breaker prevents mass replacement when a storage or network event drops many nodes to NotReady at once. Together AI says passive health checks, repair recommendations, the auto-remediation policy, wait-for-idle controls, the escalation ladder, safety rails and API and console repair are generally available today, while node-label based remediation contracts that let customers bring detection systems such as NVSentinel or custom controllers are in preview.

## Evidence

Verbatim from https://www.together.ai/blog/auto-repair-for-together-gpu-clusters:

> Health checks find the fault, the platform picks the lightest fix, and you decide what runs without a person in the loop, down to individual Xid codes.
> 
> Authors
> 
> Pavneet Ahluwalia, Dongsheng Zhang, Barbara Kazanowska
> 
> Table of contents
> 
> 40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...
> 
> Links in this article
> 
> At cluster scale, GPU faults are a reality. An Xid error, a double-bit ECC error, a GPU falling off the PCIe bus or a flapping InfiniBand link can stall a distributed training job or take an inference replica down. Spotting that something is wrong is usually the easy part. The hard part is everything between the fault and a healthy node again
> 
> Earlier this year, we announced the launch of our fully Automatic node remediation features. Our GPU clusters customers and our internal Inference platform has been using this capability to help drive workload resiliency and minimize down time. Our customers love this capability, and we wanted to take some time break the system down and explain how it works and why we designed it this way.

---

Record: https://forck.live/items/28474-auto-repair-for-together-gpu-clusters-from-gpu-fault-to-healthy-node
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md