From the source
Cartesia says its hierarchical networks (H-Nets) can replace hand-designed tokenizers by learning to chunk raw bytes directly inside the model, and it reports large efficiency gains from scaling them.
- The company says carefully controlled 1-stage H-Nets deliver up to 3-5x compute efficiency gains purely through learned tokenization, measured against a standard scaling ladder for tokenized Transformers.
- A more flexible 2-stage H-Net, scaled up, outperformed a matched open-source model while using 10x less compute and data, according to Cartesia.
- The company says H-Nets show behaviors not previously seen in language models, including in-context tokenization, and that ongoing work in biology and vision suggests the architecture can span modalities.
- Cartesia frames the work as a test of the bitter lesson: tokenization is one of the last hand-designed parts of the language-modeling pipeline, and byte-level models are a way to learn end-to-end from raw data.
- The results appear in the first of two posts, with a second planned to explore a larger range of models and why H-Nets are more efficient for language modeling.
