# Hugging Face — Hugging Face Reads, Feb. 2021 - Long-range Transformers

- Company: Hugging Face (huggingface.co)
- Announced: 2021-03-09
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://huggingface.co/blog/long-range-transformers
- Record: https://forck.live/items/2266-hugging-face-reads-feb-2021-long-range-transformers
- Subject: Platform

This is a blog post summarizing four papers on long-range transformers: Longformer, Compressive Transformer, Linformer, and Performer. It discusses their approaches to reducing the quadratic cost of transformer self-attention with respect to sequence length.

## Evidence

Verbatim from https://huggingface.co/blog/long-range-transformers:

> In this reading group, rather than trying to cover every approach (there are so many!), we’ll focus on four main ideas: Custom attention patterns (with Longformer), Recurrence (with Compressive Transformer), Low-rank approximations (with Linformer), Kernel approximations (with Performer).

---

Record: https://forck.live/items/2266-hugging-face-reads-feb-2021-long-range-transformers
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md