# Hugging Face — Training a language model with 🤗 Transformers using TensorFlow and TPUs

- Company: Hugging Face (huggingface.co)
- Announced: 2023-04-27
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/tf_tpu
- Record: https://forck.live/items/2069-training-a-language-model-with-transformers-using-tensorflow-and-tpus
- Subject: Platform
- Models affected: RoBERTa, BERT

Training a masked language model (RoBERTa) from scratch using TensorFlow and TPUs, covering tokenizer training, data preparation, and model training.

## Evidence

Verbatim from https://huggingface.co/blog/tf_tpu:

> We’re going to train a RoBERTa (base model) from scratch on the WikiText dataset (v1).

---

Record: https://forck.live/items/2069-training-a-language-model-with-transformers-using-tensorflow-and-tpus
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md