# Hugging Face — Parquet Content-Defined Chunking

- Company: Hugging Face (huggingface.co)
- Announced: 2025-07-25T00:00:00+00:00
- Category: capability-change
- Subject: Platform
- Source: https://huggingface.co/blog/parquet-cdc
- Record: https://forck.live/items/1649-parquet-content-defined-chunking

(CDC) is now available in PyArrow and Pandas, enabling efficient deduplication of Parquet files on Hugging Face's Xet storage layer, reducing data transfer and storage costs.

## Evidence

Verbatim from https://huggingface.co/blog/parquet-cdc:

> Parquet Content-Defined Chunking (CDC) is now available in PyArrow and Pandas, enabling efficient deduplication of Parquet files on content-addressable storage systems like Hugging Face's Xet storage layer.

---

Record: https://forck.live/items/1649-parquet-content-defined-chunking
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md