Hugging Face announced significant performance improvements to its datasets library's streaming feature, achieving up to 100x fewer startup requests, 10x faster data resolution, and 2x faster streaming speed, enabling efficient training on multi-TB datasets without downloading.
From the source
We've improved streaming to have 100x fewer requests, → 10× faster data resolution → 2x sample/sec, → 0 worker crashes at 256 concurrent workers.
