Hugging Face· Platform

From Files to Chunks: Improving HF Storage Efficiency

Hugging Face stores over 30 PB of models, datasets, and spaces in Git LFS repositories . Because Git stores and versions at the file level, any change to a file requires re-uploading the full asset – expensive operations when average Parquet and CSV files on the Hub range between 200-300 MB, average Safetensor files around 1 GB, and GGUF files can exceed 8 GB. Imagine modifying just a single line of metadata in a GGUF file and waiting for the multi-gigabyte file to upload; in addition to user time and transfer costs, Git LFS also then needs to save full versions of both files, bloating storage costs. The plot below illustrates the growth of LFS storage in model, dataset, and space repositories on the Hub between March 2022 and September 2024: Hugging Face's Xet team is taking a different approach to storage by storing files as chunks. …

Announced

20 Nov 2024 00:00 UTC

Detected

03 Aug 2026 21:12 UTC (+621d after)