Hugging Face· PlatformStory of the weekCapability change

Streaming datasets: 100x More Efficient

Hugging Face announced significant performance improvements to its datasets library's streaming feature, achieving up to 100x fewer startup requests, 10x faster data resolution, and 2x faster streaming speed, enabling efficient training on multi-TB datasets without downloading.

From the source

We've improved streaming to have 100x fewer requests, → 10× faster data resolution → 2x sample/sec, → 0 worker crashes at 256 concurrent workers.

huggingface.co