Rotary Positional Embeddings

Also known as RoPE

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 40% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 72

All topics

Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention
Jul 7, 2026Li HengyuRotary Positional Embeddings Attention Head Analysis
Institute for Solid State Physics, The University of Tokyo
The Wiola Architecture for Efficient Small Language Models
Jul 1, 2026Aryuemaan Kumar Chowdhury, Afreen Shaik, Yaparla Bhargavi +1Rotary Positional Embeddings Token Merging
Research and Development, Oscowl Ai IIT Hyderabad Hyderabad, India · Research and Development, Oscowl Ai Hyderabad, India
RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Mixed-Precision Quantization Rotary Positional Embeddings
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Independent Researcher, China
SpheRoPE: Zero-Shot Optimization-Free 360 Panorama Generation with Spherical RoPE
Jun 30, 2026Or Hirschorn, Aaron Olender, Eli Alshan +3Video-Reference Video Generation Rotary Positional Embeddings
1Amazon Prime Video · 2Tel-Aviv University · 3Hebrew University of Jerusalem
Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling
Jun 26, 2026Changze Lv, Zhenghua Wang, Yiran Ding +9Rotary Positional Embeddings Long-Context Language Modeling
Fudan University · Shanghai Key Laboratory of Intelligent Information Processing · Westlake University
RoPEMover: Depth-Aware Object Relocation via Positional Embeddings
Jun 25, 2026Ipek Oztas, Duygu Ceylan, Aybars Bugra Aksoy +1Rotary Positional Embeddings Diffusion Transformer
Bilkent University · Brown University · Adobe Research
Why Do Accumulated Transformations Extrapolate?
Jun 23, 2026Mahesh GodavartiRotary Positional Embeddings Length Generalization
A Carrot, Inc.
Controllable Texture Tiling with Transformed RoPE-Enhanced Diffusion Models
Jun 22, 2026Junrong Huang, Zhiyuan Zhang, Rui Tang +2Rotary Positional Embeddings Diffusion Transformer
City University of Hong Kong, China · Manycore Tech Inc., China · Hong Kong University of Science and Technology, China
Does RoPE Prevent or Degrade Retrieval Heads? A Mechanistic Analysis Across Model Families
Jun 19, 2026Cengizhan BayramRotary Positional Embeddings Long-Context Retrieval
Independent Researcher
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh +2Rotary Positional Embeddings Transformer
National University of Singapore · Center for AI Research, VinUniversity · Technical University of Munich +1
SHERPA: Seam-aware Harmonized ERP Adaptation for Open-Domain 360 $^\circ$ Panorama Generation
Jun 10, 2026Jungwoon Kang, Jaehun Kim, Yiwon Yu +3Rotary Positional Embeddings Image Generation
Yonsei University
nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding
Jun 10, 2026Boyang Li, Yulin Wu, Sizhe Xu +5Rotary Positional Embeddings Positional Encoding
New York University, New York, NY, USA · Peking University, Beijing, China · Imperial College London, London, UK +1
RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
Jun 9, 2026Alejandro García-Castellanos, Maurice Weiler, Erik J BekkersRotary Positional Embeddings Self-Attention
AMLab, University of Amsterdam · MIT CSAIL
GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models
Jun 8, 2026Yu Luo, Kun Hu, Mengwei He +7Rotary Positional Embeddings Geospatial Foundation Models
The University of Sydney · Edith Cowan University · Adelaide University +3
Where does Absolute Position come from in decoder-only Transformers?
Jun 4, 2026Valeria Ruscio, Umberto Nanni, Fabrizio SilvestriRotary Positional Embeddings Transformer Interpretability
Intuition Machines · Sapienza University of Rome
PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention
Jun 3, 2026Yaobo ZhangRotary Positional Embeddings Positional Encoding
School of Physics, Ningxia University Yinchuan, 750021, China
Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
May 31, 2026Athanasios ZerisRotary Positional Embeddings Positional Encoding
Independent Researcher, Athens, Greece.
Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
May 29, 2026Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3Rotary Positional Embeddings Attention Head Analysis
CENIA & Faculty of Mathematics UC Santiago, Chile · CENIA Santiago, Chile · IMC UC & CENIA Santiago, Chile
Periodic RoPE for Infinite Context LLMs
May 27, 2026Simin HuoRotary Positional Embeddings Long-Context Language Modeling
Shanghai Jiao Tong University
LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation
May 26, 2026Gabriele Cesa, Thomas Hehn, Aleix Torres-Camps +4Rotary Positional Embeddings Parallel Decoding
Qualcomm AI Research∗
SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers
May 21, 2026Javad Rajabi, Kimia Shaban, Koorosh Roohi +2Rotary Positional Embeddings Diffusion Model Guidance
University of Toronto · Vector Institute
RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
May 20, 2026Yuxi Liu, Zekun Zhang, Yixiang Cai +3Rotary Positional Embeddings Low-Rank Attention
Peking University · University of Electronic Science and Technology of China · Beijing University of Posts and Telecommunications
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
TIE: Time Interval Encoding for Video Generation over Events
May 11, 2026Zhilei Shu, Shangwen Zhu, Zihang Liang +10Rotary Positional Embeddings Video Diffusion Models
University of Science and Technology of China · Matrix Team · Shanghai Jiao Tong University +5
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
ExtraVAR: Stage-Aware RoPE Remapping for Resolution Extrapolation in Visual Autoregressive Models
May 11, 2026Feihong Yan, Shaoyu Liu, Haixuan Wang +4Rotary Positional Embeddings Visual Autoregressive Models
Beijing Institute of Technology · Xidian University · Northeastern University at Qinhuangdao +3
Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
May 5, 2026Yaobo ZhangRotary Positional Embeddings Self-Attention
School of Physics, Ningxia University Yinchuan, 750021, China
TCDA: Thread-Constrained Discourse-Aware Modeling for Conversational Sentiment Quadruple Analysis
May 3, 2026Xinran Li, Xinze Che, Yifan Lyu +2Rotary Positional Embeddings Graph Neural Networks
School of Software, Dalian University of Technology, Dalian, China · School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China
Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Models
May 1, 2026Chenyu Zhang, Xinchen Lyu, Chenshan Ren +4Rotary Positional Embeddings Wireless Foundation Models
National Engineering Research Center for Mobile Network Technologies, Beijing University of Posts and Telecommunications, Beijing 100876, China · Department of Broadband Communication, Pengcheng Laboratory, Shenzhen 518055, China · Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing 100081, China +1
RETO: A Rotary-Enhanced Transformer Operator for High-Fidelity Prediction of Automotive Aerodynamics
Apr 30, 2026Bojun Zhang, Huiyu Yang, Yunpeng Wang +4Rotary Positional Embeddings PDE Operator Learning
Department of Mechanics and Aerospace Engineering, Southern University of Science and Technology, Shenzhen, 518055, China · Shenzhen Key Laboratory of Complex Aerospace Flows, Southern University of Science and Technology,Shenzhen 518055, China · Ningbo Key Laboratory of Advanced Manufacturing Simulation, Eastern Institute of Technology, Ningbo, 315200, China +1

Jul 7, 2026, cs.LGJ/K move · Enter open · S save

Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

Li Hengyu

Institute for Solid State Physics, The University of Tokyo

Rotary Positional Embeddings Attention Head Analysis Induction Heads Multi-Head Attention

Open paper AskPDF arXiv

The pre-softmax score of an attention head is a bilinear form

score(i,j) = x_i^T M x_j

in a learned operator

M = W_q^T W_k

. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK (learned-absolute and ALiBi); the model-level separation is perfect at every top-k examined (exact permutation

p=0.029

), and zeroing the per-frequency RoPE phase

Im(M_t)

eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay (four pre-registered contrasts,

q_BH <= 0.016

). The cost structure exposes each scheme's default: imposing symmetry slows learned-absolute models by a factor of 2.9, whereas a RoPE head with a fully symmetric static M still routes directionally via the phase channel, impossible under absolute positions. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution: a fingerprint sculpted after function, not a hard constraint upon it.

Momentum

Related topics

Latest papers 72