# LG AI Research — [NeurIPS 2022] Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost

- Company: LG AI Research (lgresearch.ai)
- Announced: 2022-12-23
- Category: new-model
- Coverage: not counted
- Announcement: yes
- Group: models
- Source: https://www.lgresearch.ai/blog/view?seq=291
- Record: https://forck.live/items/4912-neurips-2022-transformers-meet-stochastic-block-models-attention-with-data
- Subject: EXAONE
- Models affected: SBM-Transformer

Proposes SBM-Transformer, a Transformer model that uses a Stochastic Block Model to data-adaptively choose attention sparsity and computational cost, enabling linear complexity in the number of sampled edges while preserving universal approximability.

## Evidence

Verbatim from https://www.lgresearch.ai/blog/view?seq=291:

> Hence, we propose SBM-Transformer, a Transformer model that can data-adaptively choose its attention-sparsity as well as computational cost by endowing each attention head a mixed-membership Stochastic Block Model (SBM).

---

Record: https://forck.live/items/4912-neurips-2022-transformers-meet-stochastic-block-models-attention-with-data
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md