From the source
Hugging Face released tokenizers v1, a major performance update to its tokenization library that aims to be tens of times faster than v0.23 while preserving identical token IDs, API, vocabulary and merge ranks.
- The update introduces optimizations including SIMD-based bitstream splitting instead of regex, thread-local word caching for repeated pre-tokens, and improvements across normalization, pre-tokenization, model, and post-processing stages to prevent tokenization from becoming a bottleneck in machine learning workflows.






