From the source
Together AI has explained how automatic node remediation works on its GPU clusters, a capability it says customers and its internal inference platform have used to improve workload resiliency and cut downtime.
- The system runs passive health checks on every node, both inside the customer's VM and on the host underneath, so GPU faults visible from the guest and host or fabric faults visible from the host land in one place.
- Rather than keying on a single NotReady signal, which looks the same whether a node is broken or briefly unreachable, auto repair requires a failed health check behind every repair and attaches supporting evidence to each one.
- The platform recommends the lightest action that fixes a given fault and maps each signal to it, while customers decide per fault whether a repair waits for approval or runs unattended, including per-Xid consent inside the Xid group.
- Wait-for-idle controls hold a repair until the node is actually idle, and a built-in circuit breaker prevents mass replacement when a storage or network event drops many nodes to NotReady at once.
- Together AI says passive health checks, repair recommendations, the auto-remediation policy, wait-for-idle controls, the escalation ladder, safety rails and API and console repair are generally available today, while node-label based remediation contracts that let customers bring detection systems such as NVSentinel or custom controllers are in preview.
