From the source
A 2–3× wall-clock speedup on LLM pretraining at matched FLOPs, without changing the final model architecture, optimizer, tokenizer, or training data.
During the first 20–40% of training, the model reads and predicts bags of s contiguous tokens — averaging their embeddings on the input side, predicting the next bag with a modified cross-entropy on the output side.
For the remainder of the run, it trains normally on next-token prediction.
The inference-time model is identical to one produced by conventional pretraining.
Validated at 270M, 600M, and 3B dense, and at 10B-A1B MoE.
We introduce Token Superposition Training ( TST ), a modification to the standard LLM pretraining loop that produces substantial wall-clock speedups at fixed compute, without requiring any change to the model architecture, the optimizer, the parallelism strategy, the tokenizer, or the training data.
On a 10B-A1B mixture-of-experts model trained to 2T tokens, TST reaches a lower final training loss than a matched-FLOPs baseline in roughly 40% of the wall-clock, and beats that baseline on HellaSwag, ARC-Easy, ARC-Challenge, and MMLU.
…