# Nous Research — Efficient pretraining with token superposition

- Company: Nous Research (nousresearch.com)
- Announced: 2026-02-01T15:00:00+00:00
- Category: not stated
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://nousresearch.com/token-superposition
- Record: https://forck.live/items/24024-efficient-pretraining-with-token-superposition
- Subject: Hermes models

TL;DR. A 2–3× wall-clock speedup on LLM pretraining at matched FLOPs, without changing the final model architecture, optimizer, tokenizer, or training data. During the first 20–40% of training, the model reads and predicts bags of s contiguous tokens — averaging their embeddings on the input side, predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense, and at 10B-A1B MoE. We introduce Token Superposition Training ( TST ), a modification to the standard LLM pretraining loop that produces substantial wall-clock speedups at fixed compute, without requiring any change to the model architecture, the optimizer, the parallelism strategy, the tokenizer, or the training data. On a 10B-A1B mixture-of-experts model trained to 2T tokens, TST reaches a lower final training loss than a matched-FLOPs baseline in roughly 40% of the wall-clock, and beats that baseline on HellaSwag, ARC-Easy, ARC-Challenge, and MMLU. …

---

Record: https://forck.live/items/24024-efficient-pretraining-with-token-superposition
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md