Positional Encoding

Follow this topic Ask about this topic

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

116 papers19 papers in the last four weeks, up 73% on the four weeks before. 0.1% of all new papers.

GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models
Jun 8, 2026Yu Luo, Kun Hu, Mengwei He +7Rotary Positional Embeddings Geospatial Foundation Models
The University of Sydney · Edith Cowan University · Adelaide University +3
PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention
Jun 3, 2026Yaobo ZhangRotary Positional Embeddings Relative Positional Encoding
School of Physics, Ningxia University Yinchuan, 750021, China
LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding
Jun 3, 2026Haocheng Xia, Mihir Pamnani, Hanxi Fang +2Retrieval-Augmented Generation KV Caching
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign · Nexla · Amazon +1
Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
Jun 1, 2026Qian Li, Xinyu Mao, Shang-Hua TengTransformer Expressivity Transformer
Shenzhen Research Institute of Big Data · University of Southern California
Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
May 31, 2026Athanasios ZerisRotary Positional Embeddings Positional Encoding
Independent Researcher, Athens, Greece.
ReFLEX: Length-Generalizable CSI Denoising for MIMO-OFDM via Relative-Frequency Bias
May 29, 2026Zhibin Zhang, Robert Potekhin, Ziwei Wan +2Transformer Relative Positional Encoding
Moscow Institute of Physics and Technology (State University), Moscow, Russia · Yangtze Delta Region Academy, Beijing Institute of Technology (Jiaxing), Jiaxing 314019, China · School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
May 28, 2026Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1Transformer Self-Attention
The University of Tokyo, Tokyo, Japan · RIKEN Advanced Intelligene Project, Tokyo, Japan · Kyoto University, Kyoto, Japan
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiTransformer Interpretability Disentangled Representation Learning
Sorbonne Université, CNRS, ISIR, Paris, France · Orange Innovation
Positional Encodings Anchor Spatial Structure in Vision Transformers: A Geometric Perspective on Robustness
May 28, 2026Mahmoud MannesDistribution Shift Robustness Vision Transformer
ESSTHS
Benchmarking Positional Encoding Strategies for Transformer-Based EEG Foundation Models
May 28, 2026Ayse Betul Yuce, Sebastian StoberMI Classification EEG Decoding
Department of Computer Science, Otto von Guericke University, Magdeburg, Germany
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Helix4D: Complex 4D Mesh Generation
May 25, 2026Jiraphon Yenphraphai, Jianqi Chen, Jian Wang +63D Mesh Generation Positional Encoding
1Snap · 2Purdue University · 3KAUST
Weierstrass Positional Encoding for Vision Transformers
May 20, 2026Zhihang Xin, Rui Wang, Xitong Hu +1Vision Transformer Positional Encoding
School of Mathematics and Data Science, Jiangnan University, Wuxi 214122, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
Chessformer: A Unified Architecture for Chess Modeling
May 18, 2026Daniel Monroe, George Eilender, Philip Chalmers +2Transformer Interpretability Game-Playing Agents
University of Toronto · Williams College
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation
May 13, 2026Seonghyun Jin, Youngmin Kim, Sunwoo Park +1Camera-Conditioned Video Generation Camera-Controlled Video Generation
KAIST · Graduate School of AI Korea University · Graduate School of AI KAIST
Rethinking Positional Encoding for Neural Vehicle Routing
May 12, 2026Chuanbo Hua, Federico Berto, Andre Hottung +8Vehicle Routing Positional Encoding
KAIST · Radical Numerics · Bielefeld University +3
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
May 11, 2026Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +23D ViTs Self-Attention
1Seoul National University · 2Independent · 3Google Research +2
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerSelf-Attention Long-Context Language Modeling
Blavatnik School of Computer Science and AI, Tel Aviv University
BRIDGE: Background Routing and Isolated Discrete Gating for Coarse-Mask Local Editing
May 8, 2026Peilin Xiong, Honghui Yuan, Junwen Chen +1Diffusion Transformer Positional Encoding
Department of Informatics, The University of Electro-Communications, Tokyo, Japan
Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
May 5, 2026Yaobo ZhangRotary Positional Embeddings Self-Attention
School of Physics, Ningxia University Yinchuan, 750021, China
Spiking Sequence Machines and Transformers
May 1, 2026Joy BoseTransformer Attention Spiking Neural Networks
Independent Researcher Bengaluru, India
Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling
Apr 27, 2026Hailing Cheng, Daqi Sun, Xinyu LuRotary Positional Embeddings Self-Attention
LinkedIn Inc. Mountain View, CA 94043
PEPS: Positional Encoding Projected Sampling -- Extended
Apr 27, 2026Guillaume Perez, Janarbek Matai, Takahiro HaradaImplicit Neural Representations Positional Encoding
Advanded Micro Devices
A-THENA: Early Intrusion Detection for IoT with Time-Aware Hybrid Encoding and Network-Specific Augmentation
Apr 23, 2026Ioannis Panopoulos, Maria Lamprini A. Bartsioka, Sokratis Nikolaidis +3IoT Intrusion Detection Network Intrusion Detection
National Technical University of Athens, Greece · Samsung AI Center, UK
URoPE: Universal Relative Position Embedding across Geometric Spaces
Apr 20, 2026Yichen Xie, Depu Meng, Chensheng Peng +4Rotary Positional Embeddings Relative Positional Encoding
Applied Intuition · University of California, Berkeley
Learning Where to Embed: Noise-Aware Positional Embedding for Query Retrieval in Small-Object Detection
Apr 16, 2026Yangchen Zeng, Zhenyu Yu, Dongming Jiang +5Small Object Detection Efficient Object Detection
Southeast University · Fudan University · The University of Texas at Dallas +6
Three-Phase Transformer
Apr 15, 2026Mohammad R. Abu AyyashTransformer Language Modeling
Brains Build Research, Ramallah, Palestine

Jun 8, 2026J/K move · Enter open · S save

GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models

Yu Luo, Kun Hu, Mengwei He, Xiaogang Zhu+6

The University of Sydney · Edith Cowan University · Adelaide University+3

Rotary Positional Embeddings Geospatial Foundation Models Vision Foundation Model Adaptation Positional Encoding

Open paper AskPDF arXiv

Remote-sensing foundation models (RSFMs) benefit from pretraining on imagery from multiple sensors and ground sampling distances (GSDs), but such exposure alone does not resolve scale mismatch during downstream adaptation. A fixed token-grid offset can correspond to different ground distances across sensors, making grid-based positional priors physically inconsistent. Meanwhile, heterogeneous spatial granularity means that compact urban regions and homogeneous landscapes may require different positional sensitivities even under the same GSD. Therefore, we propose {GeoRoPE}, a ground-aware, RoPE-compatible, and parameter-efficient spatial adaptation method for RSFMs. GeoRoPE recalibrates token-level positional interactions from two complementary aspects. First, Geo-Coordinate Calibration (GCC) rescales raw token-grid offsets according to the ground distance represented by one token-grid step, producing geo-calibrated relative coordinates across GSDs. Second, Geo-Frequency Calibration (GFC) adjusts the native RoPE frequency with a relation-specific factor, enabling position sensitive adaptation to scene-dependent spatial granularity. GeoRoPE is injected into pretrained RSFMs through a lightweight adapter, preserving the frozen spatial prior while adding geo-aware positional corrections. Experiments across multiple RSFMs, sensors, resolutions, and downstream tasks demonstrate that GeoRoPE improves cross-resolution robustness and scale-sensitive representation learning.