# Hugging Face — Tokenization in Transformers v5: Simpler, Clearer, and More Modular

- Company: Hugging Face (huggingface.co)
- Announced: 2025-12-18T00:00:00+00:00
- Category: developer-tool-release
- Subject: Platform
- Source: https://huggingface.co/blog/tokenizers
- Record: https://forck.live/items/1569-tokenization-in-transformers-v5-simpler-clearer-and-more-modular

Hugging Face announces Transformers v5, a major redesign of tokenization that separates tokenizer architecture from trained vocabulary, making tokenizers inspectable, customizable, and trainable with less friction.

## Evidence

Verbatim from https://huggingface.co/blog/tokenizers:

> Transformers v5 redesigns how tokenizers work. The big tokenizers reformat separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can inspect, customize, and train from scratch with far less friction.

---

Record: https://forck.live/items/1569-tokenization-in-transformers-v5-simpler-clearer-and-more-modular
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md