From the source
Kernel-level debugging sessions, surprising bottlenecks, and 2 AM fixes don't usually make it into polished papers.
We're sharing them here because we think the process of building at scale is equally valuable as the final results.
Our training stack is a heavily modified version of Torchtitan .
After evaluating multiple pre-training frameworks in early 2025 we decided to centralize our efforts around Torchtitan as it provided a straightforward path to accessing the latest features from the PyTorch team.
While Torchtitan has been an enormous success for us, the upstream version has been largely conservative with the velocity of changes and integration of more bespoke features.
Among the features we've integrated over the past year are variable length masked training (used for Hermes 4 ), asynchronous reinforcement learning with Atropos (see NousCoder ), the Muon optimizer , LoRA support, inline downstream evaluations, and a host of new model architectures and processed data formats.
…