From the source
Cartesia published the second post in its "From Tokens to Bytes" series, examining how input processing choices such as tokenization and chunking affect language model performance.
- The post studies two kinds of scaling laws: compute-to-loss, which measures pretraining efficiency, and loss-to-loss, which relates pretraining strength in loss or bits-per-byte to downstream performance.
- Cartesia says a central tension emerges: standard language models over compressed inputs have stronger compute efficiency, while models with access to finer-grained data such as byte-level inputs generalize better.
- H-Nets, the hierarchical networks from the first post, are said to have the best of both worlds, and Cartesia provides evidence that their end-to-end nature is essential.
- The post also argues that the belief the tokenizer has no major impact on loss-to-loss laws is a myth, citing prior work that stated otherwise.
- It details the pretokenization step, a hand-crafted regex that pre-segments the corpus, and notes that pretokenizers are not learned from data at all.
