# Hugging Face — Understanding BigBird's Block Sparse Attention

- Company: Hugging Face (huggingface.co)
- Announced: 2021-03-31
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/big-bird
- Record: https://forck.live/items/2262-understanding-bigbird-s-block-sparse-attention
- Subject: Platform
- Models affected: BigBird
- Context window: sequences up to a length of 4096

BigBird's block sparse attention mechanism, its implementation, and advantages over BERT's full attention, including that a BigBird RoBERTa-like model is now available in 🤗Transformers.

## Evidence

Verbatim from https://huggingface.co/blog/big-bird:

> BigBird RoBERTa-like model is now available in 🤗Transformers.

---

Record: https://forck.live/items/2262-understanding-bigbird-s-block-sparse-attention
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md