RoPE: How 2D Rotations Enable Transformers to Handle Long Contexts

Rotary Position Embedding (RoPE) uses 2D rotations to encode token positions, preserving semantic features while enabling relative distance sensitivity. Unlike absolute positional encodings that add static sine/cosine waves, RoPE rotates query and key vectors, allowing attention to focus on token distances. This technique has been widely adopted in models like LLaMA and GPT-NeoX for improved long-context performance.