From the source
From the source
This is a blog post summarizing four papers on long-range transformers: Longformer, Compressive Transformer, Linformer, and Performer.
It discusses their approaches to reducing the quadratic cost of transformer self-attention with respect to sequence length.
From the source
In this reading group, rather than trying to cover every approach (there are so many!), we’ll focus on four main ideas: Custom attention patterns (with Longformer), Recurrence (with Compressive Transformer), Low-rank approximations (with Linformer), Kernel approximations (with Performer).
huggingface.co
