From the source
Hugging Face announces a new storage approach using content-defined chunking (CDC) to improve storage efficiency and iteration speed.
By storing files as chunks and transferring only modified chunks, the method reduces upload/download times and storage costs.
Benchmarks show a 50% improvement in storage and transfer performance, and initial research on the Hub indicates potential savings of up to 100 TB for PyTorch model checkpoints.
