# Hugging Face — Improving Parquet Dedupe on Hugging Face Hub

- Company: Hugging Face (huggingface.co)
- Announced: 2024-10-05T00:00:00+00:00
- Category: research-paper
- Subject: Platform
- Source: https://huggingface.co/blog/improve_parquet_dedupe
- Record: https://forck.live/items/1820-improving-parquet-dedupe-on-hugging-face-hub

The Xet team at Hugging Face is researching and experimenting with techniques to improve Parquet file deduplication on the Hugging Face Hub, including content-defined row groups and relative offsets.

## Evidence

Verbatim from https://huggingface.co/blog/improve_parquet_dedupe:

> The Xet team at Hugging Face is working on improving the efficiency of the Hub's storage architecture to make it easier and quicker for users to store and update data and models.

---

Record: https://forck.live/items/1820-improving-parquet-dedupe-on-hugging-face-hub
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md