# Hugging Face — A failed experiment: Infini-Attention, and why we should keep trying?

- Company: Hugging Face (huggingface.co)
- Announced: 2024-08-14T00:00:00+00:00
- Category: research-paper
- Subject: Platform
- Models affected: Llama 3 8B
- Source: https://huggingface.co/blog/infini-attention
- Record: https://forck.live/items/1838-a-failed-experiment-infini-attention-and-why-we-should-keep-trying

The blog post reports that Infini-Attention, a method for extending context length, fails to perform well; its performance degrades with increased memory compression, and existing methods like ring attention, YaRN, and rope scaling remain superior.

## Evidence

Verbatim from https://huggingface.co/blog/infini-attention:

> Infini-attention's performance gets worse as we increase the number of times we compress the memory, and to the best of our knowledge, ring attention, YaRN and rope scaling are still the best ways for extending a pretrained model to longer context length.

---

Record: https://forck.live/items/1838-a-failed-experiment-infini-attention-and-why-we-should-keep-trying
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md