Rotary Positional Embeddings

Also known as RoPE

Rotary positional embeddings apply position-dependent rotations to query and key vectors, making attention scores reflect relative offsets between tokens. Common approaches adjust or learn rotation frequencies, apply rotations to selected channels or heads, and extend RoPE to spatial, temporal, graph, and geometric relationships. These methods are used in language models and in vision, video, and scientific applications, with evaluations including long-context retrieval, reasoning, and language modeling. A central challenge is extending context reliably without harming short-context performance: frequency scaling and other adaptations can help, but positional and semantic distinctions may still break down at long lengths.

Latest papers

98 papers12 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

POD-RoPE: Rotary Position Encoding over Physical Observation Distributions for Embodied AI
Oct 24, 2026Xiaodi Huang, Zilin Wang, Jinpeng Liu +4Positional Encoding Rotary Positional Embeddings
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Sharpa +1
ActionRoPE: Action as Coordinates for Spatially Consistent Control in Planar-Camera Game World Models
Oct 19, 2026Zhiyang Deng, Yuzhe Wu, Zijun Lin +1Action-Conditioned Video Generation World Models
Tencent · National University of Singapore · Nanyang Technological University
DuET: Note-Frame Dual Encoding for Piano Performance-to-Score Transcription
Oct 15, 2026Wei Zeng, Ye WangAutomatic Music Transcription Transformer
Integrative Sciences and Engineering Programme, NUS Graduate School · School of Computing, National University of Singapore
Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
Oct 15, 2026Wanyang Hong, Zhaoning ZhangKV-Cache Management KV-Cache Compression
National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.
WorldRoPE: Probabilistic World-Centric Rotary Embedding for Multi-View Attention
Oct 12, 2026Mijeong Kim, Kyujin Lim, Bohyung HanRotary Positional Embeddings Multi-View 3D Reconstruction
ECE, Seoul National University · IPAI, Seoul National University
Chronos Policy: Learning Robot Policies on Physical Time
Oct 9, 2026Changhao Wang, Jaden Clark, Shuran SongRobot Policy Learning Robot Learning
Stanford University
HySPE: Positional Encoding via Symplectic Dual Shears
Oct 7, 2026Zhongping JiPositional Encoding Length Generalization
SO(3)-RoPE for Spherical Transformers
Oct 5, 2026Christian Libner, Chase van de Geijn, Alexander S. Ecker +1Rotary Positional Embeddings Transformer
Institute of Computer Science & CIDAS, University of Göttingen · Max Planck Institute for Dynamics and Self-Organization, Göttingen · MIT CSAIL
WavePrune: One period is often enough for RoPE
Oct 3, 2026Guancheng Du, Luotian Huang, Shaowen Wang +2Rotary Positional Embeddings Positional Encoding
Tsinghua University
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Aperture: Merge-Consistent Rotary States for Compressed Tokens
Sep 29, 2026Yuhao Du, Shunian ChenRotary Positional Embeddings Visual Token Merging
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
Sep 28, 2026Jarod Lévy, Mathurin Videau, Jad Yehya +3Rotary Positional Embeddings Long-Context Language Modeling
Meta AI, Paris · Inria, Université Paris-Saclay, Palaiseau, France
Analytic-Walk Rotary Positional Encodings for Graphs
Sep 26, 2026Jiaqing Xie, Yuxin Wang, Xipeng QiuGraph Positional Encoding Rotary Positional Embeddings
Shanghai Innovation Institute · Fudan University
T-RoPE: Time-Aware Rotary Position Embedding for Sequential Recommendation
Sep 24, 2026Yang Liu, Noel Loo, Ali Khanafer +4Rotary Positional Embeddings Sequential Recommendation
Shopify · Massachusetts Institute of Technology · Liquid AI
Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Sep 20, 2026Yueyan Li, Haibo Wang, Caixia Yuan +1Rotary Positional Embeddings Physical Consistency in Video Generation
Beijing University of Posts and Telecommunications
G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity
Sep 14, 2026Shuo Zhang, Xin Su, Wei Wang +8Rotary Positional Embeddings 3D ViTs
Wuhan University, Wuhan, China · Rongyun Robot (Guizhou) Co., Ltd. · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Exact Finite Attention Responses From RoPE Derivatives
Sep 12, 2026Julie Huang, Maggie Chlon, Gregory Gutin +1Attention Approximation Causal Interventions in Language Models
Hassana Labs · Department of Computer Science, Royal Holloway, University of London · University of Oxford
$\text{GSF-}χ$ : Global Stereochemical Fields for Chiral Graph Transformers
Sep 11, 2026Jiaqing Xie, Yuxin Wang, Xipeng QiuRotary Positional Embeddings Molecular Property Prediction
Shanghai Innovation Institute · Fudan University
Content-Based Addressing for Long Context
Sep 7, 2026Mahesh GodavartiRotary Positional Embeddings Long-Context Language Modeling
A Carrot, Inc.
RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Rotary Positional Embeddings Low-Rank Attention
Peking University, Melon Group · Alibaba Group · University of Electronic Science and Technology of China +1
RoPE attention is an exact forward-pass gradient step with softmax intact
Sep 6, 2026Julie Huang, Maggie Chlon, Leon ChlonSoftmax Attention Rotary Positional Embeddings
Hassana Labs · Department of Information Engineering, Oxford University
WEECFP-SuRGE: A Position-Aware Substructure Encoding Method for Molecular Property Prediction
Sep 4, 2026Robert EppsMolecular Representation Learning Molecular Property Prediction
Eli Lilly and Company Lilly Corporate Center, Indianapolis, IN, United States of America
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional Embeddings Linear Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China
MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation
Sep 1, 2026Zhijian Qiao, Xinjiang Wang, Jiajie Chen +5Rotary Positional Embeddings Camera-Controlled Video Generation
The Hong Kong University of Science and Technology · Zhuoyu Technology
Higher-Dimensional Rotary Position Embedding
Aug 30, 2026Yixing Li, Ruobing Xie, Yudong Zhang +3Rotary Positional Embeddings Transformer Attention
The Chinese University of Hong Kong · Tencent Hunyuan · Tsinghua University +1
HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization
Aug 23, 2026Yingying Yan, Jiaqi Tang, Wei Wei +6Visual Tokenization Remote Sensing Image Understanding
Hong Kong University of Science and Technology Hong Kong, China
Disentangling the Expressivity of RoPE
Aug 12, 2026Selim Jerad, Anej Svete, Jiaoda Li +1Rotary Positional Embeddings Transformer Expressivity
Toyota Technological Institute at Chicago · ETH Zürich
Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
Aug 9, 2026Jiguo LiRotary Positional Embeddings Long-Context Modeling
ControlRef: Efficient Layout-Guided Multi-Instance Generation via Anchored 4D-RoPE
Aug 7, 2026Yunkai Yang, Yudong Zhang, Xinying Chen +7Rotary Positional Embeddings Diffusion Transformer
1Sun Yat-Sen University · 2Tsinghua University · 3Beijing Institute of Technology +1
MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Aug 4, 2026Tong Ling, Hang Lei, Feng Xiao +5Rotary Positional Embeddings Masked Diffusion Models
University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China +1

Oct 24, 2026J/K move · Enter open · S save

POD-RoPE: Rotary Position Encoding over Physical Observation Distributions for Embodied AI

Xiaodi Huang, Zilin Wang, Jinpeng Liu, Pengchuan Xiao+3

Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Sharpa+1

Positional Encoding Rotary Positional Embeddings Robotic Perception

Open paper AskPDF arXiv

Embodied perception requires positional representations that capture both physical geometry and the spatial and temporal extent of observations. We introduce POD-RoPE (Physical Observation Distribution RoPE), which represents token observation footprints as distributions over camera centers, viewing directions, and sampling times. Under independent sampling, expected rotations encode relative relationships between distributions; when each distribution collapses to a single address, the construction recovers pointwise RoPE. The same rotation statistics support Q/K matching and coherence-dependent V/O transformations, allowing observation geometry to guide both information selection and content aggregation. Per-token pre-aggregation avoids increasing sequence length or processing pairs of observation nodes. We evaluate POD-RoPE across four robot policy architectures, simulation and real-world manipulation, and complementary reconstruction, depth-estimation, and novel-view-synthesis tasks. Integrating POD-RoPE into Fast-WAM improves mean success from 91.83% to 92.88% across all 50 RoboTwin 2.0 tasks, and from 32.5% to 52.5% on two real-robot tasks on Sharpa North. Controlled reconstruction ablations support the contributions of distribution encoding and V/O transformations. In H20 model-block benchmarks, POD-RoPE achieves median forward-plus-backward speedups of

1.91\times

over P-RoPE and

5.34\times

over RayRoPE, with a median latency overhead of 17.4% over Standard RoPE. These results support physical observation distributions as an effective and computationally efficient basis for geometry-aware attention.