From the source
Torch’s distributed asynchronous checkpoint saves make frequent checkpointing nearly free, enabling more frequent checkpointing and cutting recovery cost.
Model checkpointing by itself is not sufficient, checkpointing the data pipeline prevents silent training-data corruption on resume.
At scale, your training efficiency is determined by a single metric: " goodput ", the proportion of time your GPUs spend on productive computation rather than waiting or recovering from failures.
Because GPU failures are the expected case at scale, the ability to rapidly and automatically recover from a failure is the only way to maintain high goodput and manage your total GPU spend.
Two subsystems make or break that recovery, yet both are routinely treated as afterthoughts: the data pipeline that feeds your accelerators, and the checkpointing mechanism that snapshots state so a job can resume.
Get either one wrong and every failure costs you far more idle GPU time than it should.
…
