# Hugging Face — From Files to Chunks: Improving HF Storage Efficiency

- Company: Hugging Face (huggingface.co)
- Announced: 2024-11-20T00:00:00+00:00
- Category: infrastructure-release
- Subject: Platform
- Models affected: openai-community/gpt2
- Source: https://huggingface.co/blog/from-files-to-chunks
- Record: https://forck.live/items/1795-from-files-to-chunks-improving-hf-storage-efficiency

Hugging Face announces a new storage approach using content-defined chunking (CDC) to improve storage efficiency and iteration speed. By storing files as chunks and transferring only modified chunks, the method reduces upload/download times and storage costs. Benchmarks show a 50% improvement in storage and transfer performance, and initial research on the Hub indicates potential savings of up to 100 TB for PyTorch model checkpoints.

## Evidence

Verbatim from https://huggingface.co/blog/from-files-to-chunks:

> Hugging Face's Xet team is taking a different approach to storage by storing files as chunks. By only transferring modified chunks, we can dramatically improve both storage efficiency and iteration speed while ensuring reliable access to evolving datasets and models.

---

Record: https://forck.live/items/1795-from-files-to-chunks-improving-hf-storage-efficiency
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md