Rotary Positional Embeddings

Also known as RoPE

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 40% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 72

All topics

HySPE: Positional Encoding via Symplectic Dual Shears
Oct 7, 2026Zhongping JiPositional Encoding Length Generalization
SO(3)-RoPE for Spherical Transformers
Oct 5, 2026Christian Libner, Chase van de Geijn, Alexander S. Ecker +1Rotary Positional Embeddings Transformer
Institute of Computer Science & CIDAS, University of Göttingen · Max Planck Institute for Dynamics and Self-Organization, Göttingen · MIT CSAIL
RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures
Sep 30, 2026Yuyang Wu, Yufeng Du, Hao PengRotary Positional Embeddings Long-Context Language Model Evaluation
University of Illinois at Urbana-Champaign, USA
Aperture: Merge-Consistent Rotary States for Compressed Tokens
Sep 29, 2026Yuhao Du, Shunian ChenRotary Positional Embeddings Visual Token Merging
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
Sep 28, 2026Jarod Lévy, Mathurin Videau, Jad Yehya +3Rotary Positional Embeddings Long-Context Language Modeling
Meta AI, Paris · Inria, Université Paris-Saclay, Palaiseau, France
Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Sep 20, 2026Yueyan Li, Haibo Wang, Caixia Yuan +1Rotary Positional Embeddings Physical Consistency in Video Generation
Beijing University of Posts and Telecommunications
G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity
Sep 14, 2026Shuo Zhang, Xin Su, Wei Wang +8Rotary Positional Embeddings 3D ViTs
Wuhan University, Wuhan, China · Rongyun Robot (Guizhou) Co., Ltd. · Institute of Automation, Chinese Academy of Sciences, Beijing, China
$\text{GSF-}\chi$ : Global Stereochemical Fields for Chiral Graph Transformers
Sep 14, 2026Jiaqing Xie, Yuxin Wang, Xipeng QiuRotary Positional Embeddings Molecular Property Prediction
Shanghai Innovation Institute · Fudan University
Content-Based Addressing for Long Context
Sep 7, 2026Mahesh GodavartiRotary Positional Embeddings Long-Context Language Modeling
A Carrot, Inc.
RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Rotary Positional Embeddings Low-Rank Attention
Peking University, Melon Group · Alibaba Group · University of Electronic Science and Technology of China +1
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional Embeddings Linear Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China
MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation
Sep 1, 2026Zhijian Qiao, Xinjiang Wang, Jiajie Chen +5Rotary Positional Embeddings Camera-Controlled Video Generation
The Hong Kong University of Science and Technology · Zhuoyu Technology
Higher-Dimensional Rotary Position Embedding
Aug 30, 2026Yixing Li, Ruobing Xie, Yudong Zhang +3Rotary Positional Embeddings Transformer Attention
The Chinese University of Hong Kong · Tencent Hunyuan · Tsinghua University +1
Disentangling the Expressivity of RoPE
Aug 12, 2026Selim Jerad, Anej Svete, Jiaoda Li +1Rotary Positional Embeddings Transformer Expressivity
Toyota Technological Institute at Chicago · ETH Zürich
Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
Aug 9, 2026Jiguo LiRotary Positional Embeddings Long-Context Modeling
ControlRef: Efficient Layout-Guided Multi-Instance Generation via Anchored 4D-RoPE
Aug 7, 2026Yunkai Yang, Yudong Zhang, Xinying Chen +7Rotary Positional Embeddings Diffusion Transformer
1Sun Yat-Sen University · 2Tsinghua University · 3Beijing Institute of Technology +1
MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Aug 4, 2026Tong Ling, Hang Lei, Feng Xiao +5Rotary Positional Embeddings Masked Diffusion Models
University Chinese Academic of Science · HUJING Digital Media & Entertainment Group · State Key Laboratory for Novel Software Technology, Nanjing University, China +1
Inter-Residue Geometry Attention for Antibody-Specific Epitope Prediction
Aug 2, 2026Chuanliu Fan, Nan Yu, Junjie Wu +1Rotary Positional Embeddings Positional Encoding
School of Computer Science and Technology, Soochow University · Institute of Artificial Intelligence, Soochow University
Labeled Incidence Structures for Native Transformer Modeling of Text, Knowledge Graphs, and Hypergraphs
Jul 29, 2026Mahesh GodavartiRotary Positional Embeddings Knowledge Graphs
A Carrot, Inc
Journey Operators for Structured Multi-Axis Composition
Jul 29, 2026Mahesh GodavartiRotary Positional Embeddings Positional Encoding
A Carrot, Inc
ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling
Jul 29, 2026Yiwen Chen, Joshua Ainslie, Krzysztof Choromanski +4Fourier Feature Embeddings Rotary Positional Embeddings
YouTube · Google DeepMind
Phase Structure in Rotary Attention: A Spectral Framework for Semantic Continuity and Execution-Boundary Governance
Jul 28, 2026Abraham ChachamovitsRotary Positional Embeddings Transformer Attention
ENTRUST AI
Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere
Jul 27, 2026Hao YeSoftmax Attention Rotary Positional Embeddings
Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Jul 23, 2026Ji Ho BaeRotary Positional Embeddings In-Context Retrieval
JRTI, Seoul, 05855, Republic of Korea
Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention
Jul 21, 2026Subham Singh, Ashutosh Mishra, Subha RautSoftmax Attention Rotary Positional Embeddings
Department of Mathematics and Statistics, Mississippi State University · Independent Researcher
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Jul 17, 2026Haodong Wen, Yiran Zhang, Yingfa Chen +1Language Model Pretraining Rotary Positional Embeddings
Tsinghua University
2D Rotary Position Embedding for Scene Text Recognition with Transformers
Jul 15, 2026Zobeir RaisiRotary Positional Embeddings Cross-Attention
Electrical Engineering Department, Chabahar Maritime University, Chabahar, Iran.
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
Jul 11, 2026Petros Karypis, Sean O'Brien, Shreyas Kadekodi +2Language Model Pretraining Rotary Positional Embeddings
UC San Diego · Independent Researcher
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
Jul 8, 2026Xinyi Wu, Siyuan Liu, Ali JadbabaieRotary Positional Embeddings Long-Context Language Modeling
MIT IDSS · IIIS, Tsinghua University
Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
Jul 8, 2026Haozhan Tang, Zerui Wang, Yuxian Gu +2Rotary Positional Embeddings Long-Context Language Modeling
NVIDIA

Oct 7, 2026, cs.CLJ/K move · Enter open · S save

HySPE: Positional Encoding via Symplectic Dual Shears

Zhongping Ji

Positional Encoding Length Generalization Rotary Positional Embeddings

Open paper AskPDF arXiv

We introduce Hyperbolic Symplectic Positional Encoding (HySPE), grounding positional attention in non-compact symplectic transformations. While canonical Rotary Position Embedding (RoPE) parameterizes the compact, elliptic branch of

\Sp(2,\R)

via rotations, HySPE operationalizes its hyperbolic branch via a damped symmetric composition of dual shears, yielding a conformally symplectic contraction with two spectral decay rates per channel pair. To eliminate the exponential representation drift inherent to naive absolute factorizations, we diagonalize the operator in its invariant eigenbasis and introduce blockwise coordinate rebasing with adaptive centered execution. This guarantees length-independent numerical bounds while matching cached RoPE forward latency (7.21,ms on an RTX 4090). On TinyShakespeare, HySPE-UltraLong maintains an invariant perplexity of 4.810 up to

16\times

zero-shot extrapolation (

L=4096

), whereas RoPE degrades to 131.198. Scaled to a 51M-parameter subword Transformer on WikiText-103 (

L_{\text{train}}=512

), HySPE closely matches RoPE in-domain while robustly extrapolating to length 8192, reducing tail perplexity by 83.9% over RoPE. While these controlled experiments establish HySPE's extrapolation robustness and numerical stability, evaluating its scaling behavior on large-scale foundation models remains an important direction for future investigation.

Momentum

Related topics

Latest papers 72