Hugging Face· Platform

Improving Parquet Dedupe on Hugging Face Hub

The Xet team at Hugging Face is working on improving the efficiency of the Hub's storage architecture to make it easier and quicker for users to store and update data and models. As Hugging Face hosts nearly 11PB of datasets with Parquet files alone accounting for over 2.2PB of that storage, optimizing Parquet storage is of pretty high priority. Most Parquet files are bulk exports from various data analysis pipelines or databases, often appearing as full snapshots rather than incremental updates. Data deduplication becomes critical for efficiency when users want to update their datasets on a regular basis. Only by deduplicating can we store all versions as compactly as possible, without requiring everything to be uploaded again on every update. In an ideal case, we should be able to store every version of a growing dataset with only a little more space than the size of its largest version. Our default storage algorithm uses byte-level Content-Defined Chunking (CDC) , …

Announced

05 Oct 2024 00:00 UTC

Detected

03 Aug 2026 21:12 UTC (+667d after)