# Hugging Face — Pre-Train BERT with Hugging Face Transformers and Habana Gaudi

- Company: Hugging Face (huggingface.co)
- Announced: 2022-08-22
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/pretraining-bert
- Record: https://forck.live/items/2160-pre-train-bert-with-hugging-face-transformers-and-habana-gaudi
- Subject: Platform

To pre-train a BERT-base model from scratch using Hugging Face Transformers and Habana Gaudi on AWS, covering dataset preparation, tokenizer training, and model pre-training with masked language modeling.

## Evidence

Verbatim from https://huggingface.co/blog/pretraining-bert:

> In this Tutorial, you will learn how to pre-train BERT-base from scratch using a Habana Gaudi-based DL1 instance on AWS to take advantage of the cost-performance benefits of Gaudi.

---

Record: https://forck.live/items/2160-pre-train-bert-with-hugging-face-transformers-and-habana-gaudi
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md