From the source
Hugging Face describes DeDLOC, a method for collaborative distributed training over the internet, and demonstrates its use by pretraining sahajBERT, a Bengali language model, with 40 volunteers.
Deep Learning over the Internet: Training Language Models Collaboratively
Research paper
From the source
we describe DeDLOC — a new method for collaborative distributed training that can adapt itself to the network and hardware constraints of participants. We show that it can be successfully applied in real-world scenarios by pretraining sahajBERT, a model for the Bengali language, with 40 volunteers.
huggingface.co
