Hugging Face released tokenizers v1, a major performance update to its tokenization library that aims to be tens of times faster than v0.23 while preserving identical token IDs, API, vocabulary and merge ranks.

The update introduces optimizations including SIMD-based bitstream splitting instead of regex, thread-local word caching for repeated pre-tokens, and improvements across normalization, pre-tokenization, model, and post-processing stages to prevent tokenization from becoming a bottleneck in machine learning workflows.