# Hugging Face — Large-scale Near-deduplication Behind BigCode

- Company: Hugging Face (huggingface.co)
- Announced: 2023-05-16
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://huggingface.co/blog/dedup
- Record: https://forck.live/items/2059-large-scale-near-deduplication-behind-bigcode
- Subject: Platform

The large-scale near-deduplication approach used in the BigCode project, discussing motivations, methods, and results.

## Evidence

Verbatim from https://huggingface.co/blog/dedup:

> Large-scale Near-deduplication Behind BigCode

---

Record: https://forck.live/items/2059-large-scale-near-deduplication-behind-bigcode
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md