Rotary Positional Embeddings

Also known as RoPE

Rotary positional embeddings apply position-dependent rotations to query and key vectors, making attention scores reflect relative offsets between tokens. Common approaches adjust or learn rotation frequencies, apply rotations to selected channels or heads, and extend RoPE to spatial, temporal, graph, and geometric relationships. These methods are used in language models and in vision, video, and scientific applications, with evaluations including long-context retrieval, reasoning, and language modeling. A central challenge is extending context reliably without harming short-context performance: frequency scaling and other adaptations can help, but positional and semantic distinctions may still break down at long lengths.

Latest papers

98 papers12 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoff
Feb 17, 2026Patrick Pynadath, Ruqi ZhangAutoregressive Modeling Self-Attention
Department of Computer Science, Purdue University
EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs
Dec 11, 2025Chao Gong, Depeng Wang, Zhipeng Wei +3Rotary Positional Embeddings Multimodal Large Language Models
1Fudan University · 2Ant Group · 3UC Berkley
Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer
Nov 24, 2025Haoyu Wu, Jingyi Xu, Qiaomu Miao +2Rotary Positional Embeddings Video Diffusion Models
Stony Brook University, NY, USA · UNC-Charlotte, NC, USA
Selective Rotary Position Embedding
Nov 21, 2025Sajad Movahedi, Timur Carstensen, Arshia Afzal +3Softmax Attention Rotary Positional Embeddings
ELLIS Institute Tübingen1, LIONS, EPFL2, University of Freiburg3, Max-Planck-Institute for Intelligent Systems4, Prior Labs5
Kairos: Toward Adaptive and Parameter-Efficient Time Series Foundation Models
Sep 30, 2025Kun Feng, Shaocheng Lan, Yuchen Fang +6Rotary Positional Embeddings Zero-Shot Time Series Forecasting
School of Information Science and Technology, ShanghaiTech University, Shanghai, China · Ant Group, Shanghai, China
Rotary Position Encodings for Graphs
Sep 26, 2025Isaac Reid, Arijit Sehanobish, Cederik Höfs +7Graph Positional Encoding Rotary Positional Embeddings
University of Cambridge · Google DeepMind · Independent Researcher +3
CineScale: Tuning-Free High-Resolution Video Generation
Aug 21, 2025Gordon Chen, Haonan Qiu, Ning Yu +3Rotary Positional Embeddings Video Diffusion Models
Nanyang Technological University · Netflix Eyeline Studios
ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models
May 27, 2025Bozhou Li, Wentao ZhangRotary Positional Embeddings Vision-Language Models
Peking University Beijing, China

Feb 17, 2026J/K move · Enter open · S save

Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoff

Patrick Pynadath, Ruqi Zhang

Department of Computer Science, Purdue University

Autoregressive Modeling Self-Attention Autoregressive Diffusion Rotary Positional Embeddings

Open paper AskPDF arXiv

Any-order autoregressive models (AO-ARMs) offer a promising path toward efficient masked diffusion by enabling native key-value caching, but competitive performance has so far required two-stream attention, typically motivated as a means of decoupling token content from position. In this work, we argue that two-stream attention may be serving a more subtle role. We identify a structural-semantic tradeoff in any-order generation: the hidden representation at each step must simultaneously attend to semantically informative tokens for prediction and structurally recent tokens for summarization, objectives that compete for attention capacity in a single stream but can specialize across two streams. To isolate this tradeoff from position-content separation, we propose Decoupled RoPE, a modification to rotary position embeddings that provides target position information without revealing target content. Decoupled RoPE performs competitively at short sequence lengths--where semantic and structural proximity coincide--but degrades as sequence length increases and the two orderings diverge. These results suggest that the success of two-stream attention stems not merely from separating position from content, but from circumventing the deeper structural-semantic tradeoff inherent to any-order generation.