Sakana AI· Sakana modelsResearch paper

Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

Sakana AI introduces DroPE, a method to extend the context length of pretrained LLMs by dropping positional embeddings during inference, requiring less than 1% of the original pretraining budget and outperforming established methods on LongBench and RULER.

From the source

We’re excited to introduce DroPE: Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings!

sakana.ai

Announced

11 Jan 2026 15:00 UTC

Detected

28 Aug 2026 21:48 UTC (+229d after)