Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Hugging Face announces Transformers v5, a major redesign of tokenization that separates tokenizer architecture from trained vocabulary, making tokenizers inspectable, customizable, and trainable with less friction.

From the source
Transformers v5 redesigns how tokenizers work. The big tokenizers reformat separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can inspect, customize, and train from scratch with far less friction.
huggingface.co

Visit source ↗huggingface.co

Announced: 18 Dec 2025 00:00 UTC
Detected: 03 Aug 2026 21:12 UTC (+228d after)

Other stories

Hugging Face · 28 AugThe Open ASR Leaderboard Adds Its First Global South LanguageCapability change

xAI · 29 AugGrok Bot now works with XCapability change

GitHub · 28 AugGitHub Copilot in Visual Studio — August update

GitHub · 28 AugGitHub Copilot weekly releases — August 24Capability change

Amazon · 28 AugBatch write and discover records in Amazon SageMaker Feature Store

API release

Perplexity · 28 AugGLM 5.3API release