Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
Research paper
Sakana AI introduces DroPE, a method to extend the context length of pretrained LLMs by dropping positional embeddings during inference, requiring less than 1% of the original pretraining budget and outperforming established methods on LongBench and RULER.
