From the source
Hugging Face's Xet team announces a new chunk-based approach using blocks, shards, and key chunks to accelerate uploads and downloads on the Hub, with open-sourced components xet-core and hf_xet integrated with huggingface_hub.
From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub
Infrastructure
From the source
To bring faster file transfers to the Hub, we've open-sourced xet-core and hf_xet, an integration with huggingface_hub which uses a chunk-based approach written in Rust.
huggingface.co
