# Hugging Face — Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

- Company: Hugging Face (huggingface.co)
- Announced: 2020-11-09
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/warm-starting-encoder-decoder
- Record: https://forck.live/items/2273-leveraging-pre-trained-language-model-checkpoints-for-encoder-decoder-models
- Subject: Platform

To warm-start encoder-decoder models using pre-trained language model checkpoints like BERT and GPT2 to avoid expensive pre-training. It provides theoretical background, analysis from Rothe et al. (2020), and a practical code example using the 🤗Transformers library.

## Evidence

Verbatim from https://huggingface.co/blog/warm-starting-encoder-decoder:

> In this notebook, we will explain in detail how encoder-decoder models can be warm-started, give practical tips based on Rothe et al. (2020), and finally go over a complete code example showing how to warm-start encoder-decoder models with 🤗Transformers.

---

Record: https://forck.live/items/2273-leveraging-pre-trained-language-model-checkpoints-for-encoder-decoder-models
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md