# Hugging Face — You could have designed state of the art positional encoding

- Company: Hugging Face (huggingface.co)
- Announced: 2024-11-25T00:00:00+00:00
- Subject: Platform
- Source: https://huggingface.co/blog/designing-positional-encoding
- Record: https://forck.live/items/1793-you-could-have-designed-state-of-the-art-positional-encoding

This blog post walks through the step-by-step discovery of state-of-the-art positional encoding in transformer models, iteratively improving the approach to arrive at RoPE, used in Llama 3.2 and modern transformers.

## Evidence

Verbatim from https://huggingface.co/blog/designing-positional-encoding:

> This post walks you through the step-by-step discovery of state-of-the-art positional encoding in transformer models. We will achieve this by iteratively improving our approach to encoding position, arriving at Rotary Positional Encoding (RoPE) used in the latest LLama 3.2 release and most modern transformers.

---

Record: https://forck.live/items/1793-you-could-have-designed-state-of-the-art-positional-encoding
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md