From the source
To pre-train a BERT-base model from scratch using Hugging Face Transformers and Habana Gaudi on AWS, covering dataset preparation, tokenizer training, and model pre-training with masked language modeling.
Pre-Train BERT with Hugging Face Transformers and Habana Gaudi
From the source
In this Tutorial, you will learn how to pre-train BERT-base from scratch using a Habana Gaudi-based DL1 instance on AWS to take advantage of the cost-performance benefits of Gaudi.
huggingface.co
