# Cartesia — From Tokens to Bytes II: Learning from Raw Data

- Company: Cartesia (cartesia.ai)
- Announced: 2026-10-28
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.cartesia.ai/blog/from-tokens-to-bytes-part-2
- Record: https://forck.live/items/28893-from-tokens-to-bytes-ii-learning-from-raw-data
- Subject: Sonic / Ink

Cartesia published the second post in its "From Tokens to Bytes" series, examining how input processing choices such as tokenization and chunking affect language model performance. The post studies two kinds of scaling laws: compute-to-loss, which measures pretraining efficiency, and loss-to-loss, which relates pretraining strength in loss or bits-per-byte to downstream performance. Cartesia says a central tension emerges: standard language models over compressed inputs have stronger compute efficiency, while models with access to finer-grained data such as byte-level inputs generalize better. H-Nets, the hierarchical networks from the first post, are said to have the best of both worlds, and Cartesia provides evidence that their end-to-end nature is essential. The post also argues that the belief the tokenizer has no major impact on loss-to-loss laws is a myth, citing prior work that stated otherwise. It details the pretokenization step, a hand-crafted regex that pre-segments the corpus, and notes that pretokenizers are not learned from data at all.

## Evidence

Verbatim from https://www.cartesia.ai/blog/from-tokens-to-bytes-part-2:

> In the first post of this series, we saw remarkable efficiency improvements from hierarchical networks over standard tokenized language models. In this post, we’ll dissect some of these empirical phenomena more deeply through analyzing a wide range of methods. Our ultimate goal is to gain more clarity on the following question: if our goal is to train the best language models we can, given no priors , what are the tradeoffs of different approaches? And how much are those priors buying us right now?
> 
> Our primary tool will be carefully studying two types of laws:
> 
> Compute-to-loss: the standard scaling law, measuring how efficient a model is at pretraining.
> 
> Loss-to-loss: the relationship between a model’s pretraining strength (measured in loss or bits-per-byte) and its downstream performance.
> 
> Loss-to-loss laws capture some measure of a model’s generalization ability, and we use them as a proxy for how well a pretrained model actually understands its data. We’ll introduce a sequence of carefully controlled models to understand how architecture decisions (particularly those around how input data is processed) affect model performance:
> 
> A central tension that emerges is

---

Record: https://forck.live/items/28893-from-tokens-to-bytes-ii-learning-from-raw-data
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md