# Hugging Face — Deep Learning over the Internet: Training Language Models Collaboratively

- Company: Hugging Face (huggingface.co)
- Announced: 2021-07-15
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://huggingface.co/blog/collaborative-training
- Record: https://forck.live/items/2253-deep-learning-over-the-internet-training-language-models-collaboratively
- Subject: Platform
- Models affected: sahajBERT

Hugging Face describes DeDLOC, a method for collaborative distributed training over the internet, and demonstrates its use by pretraining sahajBERT, a Bengali language model, with 40 volunteers.

## Evidence

Verbatim from https://huggingface.co/blog/collaborative-training:

> we describe DeDLOC — a new method for collaborative distributed training that can adapt itself to the network and hardware constraints of participants. We show that it can be successfully applied in real-world scenarios by pretraining sahajBERT, a model for the Bengali language, with 40 volunteers.

---

Record: https://forck.live/items/2253-deep-learning-over-the-internet-training-language-models-collaboratively
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md