Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models
To warm-start encoder-decoder models using pre-trained language model checkpoints like BERT and GPT2 to avoid expensive pre-training. It provides theoretical background, analysis from Rothe et al. (2020), and a practical code example using the 🤗Transformers library.