Positional Encoding

Follow this topic Ask about this topic

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

115 papers18 papers in the last four weeks, up 64% on the four weeks before. 0.1% of all new papers.

PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention
Jun 3, 2026Yaobo ZhangRotary Positional Embeddings Relative Positional Encoding
School of Physics, Ningxia University Yinchuan, 750021, China
LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding
Jun 3, 2026Haocheng Xia, Mihir Pamnani, Hanxi Fang +2Retrieval-Augmented Generation KV Caching
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign · Nexla · Amazon +1
Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
Jun 1, 2026Qian Li, Xinyu Mao, Shang-Hua TengTransformer Expressivity Transformer
Shenzhen Research Institute of Big Data · University of Southern California
Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
May 31, 2026Athanasios ZerisRotary Positional Embeddings Positional Encoding
Independent Researcher, Athens, Greece.
ReFLEX: Length-Generalizable CSI Denoising for MIMO-OFDM via Relative-Frequency Bias
May 29, 2026Zhibin Zhang, Robert Potekhin, Ziwei Wan +2Transformer Relative Positional Encoding
Moscow Institute of Physics and Technology (State University), Moscow, Russia · Yangtze Delta Region Academy, Beijing Institute of Technology (Jiaxing), Jiaxing 314019, China · School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
May 28, 2026Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1Transformer Self-Attention
The University of Tokyo, Tokyo, Japan · RIKEN Advanced Intelligene Project, Tokyo, Japan · Kyoto University, Kyoto, Japan
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiTransformer Interpretability Disentangled Representation Learning
Sorbonne Université, CNRS, ISIR, Paris, France · Orange Innovation
Positional Encodings Anchor Spatial Structure in Vision Transformers: A Geometric Perspective on Robustness
May 28, 2026Mahmoud MannesDistribution Shift Robustness Vision Transformer
ESSTHS
Benchmarking Positional Encoding Strategies for Transformer-Based EEG Foundation Models
May 28, 2026Ayse Betul Yuce, Sebastian StoberMI Classification EEG Decoding
Department of Computer Science, Otto von Guericke University, Magdeburg, Germany
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Helix4D: Complex 4D Mesh Generation
May 25, 2026Jiraphon Yenphraphai, Jianqi Chen, Jian Wang +63D Mesh Generation Positional Encoding
1Snap · 2Purdue University · 3KAUST
Weierstrass Positional Encoding for Vision Transformers
May 20, 2026Zhihang Xin, Rui Wang, Xitong Hu +1Vision Transformer Positional Encoding
School of Mathematics and Data Science, Jiangnan University, Wuxi 214122, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
Chessformer: A Unified Architecture for Chess Modeling
May 18, 2026Daniel Monroe, George Eilender, Philip Chalmers +2Transformer Interpretability Game-Playing Agents
University of Toronto · Williams College
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation
May 13, 2026Seonghyun Jin, Youngmin Kim, Sunwoo Park +1Camera-Conditioned Video Generation Camera-Controlled Video Generation
KAIST · Graduate School of AI Korea University · Graduate School of AI KAIST
Rethinking Positional Encoding for Neural Vehicle Routing
May 12, 2026Chuanbo Hua, Federico Berto, Andre Hottung +8Vehicle Routing Positional Encoding
KAIST · Radical Numerics · Bielefeld University +3
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
May 11, 2026Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +23D ViTs Self-Attention
1Seoul National University · 2Independent · 3Google Research +2
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerSelf-Attention Long-Context Language Modeling
Blavatnik School of Computer Science and AI, Tel Aviv University
BRIDGE: Background Routing and Isolated Discrete Gating for Coarse-Mask Local Editing
May 8, 2026Peilin Xiong, Honghui Yuan, Junwen Chen +1Diffusion Transformer Positional Encoding
Department of Informatics, The University of Electro-Communications, Tokyo, Japan
Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
May 5, 2026Yaobo ZhangRotary Positional Embeddings Self-Attention
School of Physics, Ningxia University Yinchuan, 750021, China
Spiking Sequence Machines and Transformers
May 1, 2026Joy BoseTransformer Attention Spiking Neural Networks
Independent Researcher Bengaluru, India
Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling
Apr 27, 2026Hailing Cheng, Daqi Sun, Xinyu LuRotary Positional Embeddings Self-Attention
LinkedIn Inc. Mountain View, CA 94043
PEPS: Positional Encoding Projected Sampling -- Extended
Apr 27, 2026Guillaume Perez, Janarbek Matai, Takahiro HaradaImplicit Neural Representations Positional Encoding
Advanded Micro Devices
A-THENA: Early Intrusion Detection for IoT with Time-Aware Hybrid Encoding and Network-Specific Augmentation
Apr 23, 2026Ioannis Panopoulos, Maria Lamprini A. Bartsioka, Sokratis Nikolaidis +3IoT Intrusion Detection Network Intrusion Detection
National Technical University of Athens, Greece · Samsung AI Center, UK
URoPE: Universal Relative Position Embedding across Geometric Spaces
Apr 20, 2026Yichen Xie, Depu Meng, Chensheng Peng +4Rotary Positional Embeddings Relative Positional Encoding
Applied Intuition · University of California, Berkeley
Learning Where to Embed: Noise-Aware Positional Embedding for Query Retrieval in Small-Object Detection
Apr 16, 2026Yangchen Zeng, Zhenyu Yu, Dongming Jiang +5Small Object Detection Efficient Object Detection
Southeast University · Fudan University · The University of Texas at Dallas +6
Three-Phase Transformer
Apr 15, 2026Mohammad R. Abu AyyashTransformer Language Modeling
Brains Build Research, Ramallah, Palestine
Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings
Apr 9, 2026Logan Hallee, Jason P. GleghornAttention Mechanisms Positional Encoding
Center for Bioinformatics and Computational Biology, University of Delaware · Synthyra, Newark, DE · Department of Biomedical Engineering, University of Delaware

Jun 3, 2026J/K move · Enter open · S save

PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention

Yaobo Zhang

School of Physics, Ningxia University Yinchuan, 750021, China

Rotary Positional Embeddings Relative Positional Encoding Positional Encoding

Open paper AskPDF arXiv

We organize relative-position mechanisms in attention as a learnable Fourier-Jet-Affine position space. The starting point is lag-shift dynamics: a relative-position kernel is a response function of the lag

d=i-j

, and the one-step shift

(Ef)(d)=f(d+1)

gives a compact classification of finite structured responses through constant-coefficient difference modules. In this view, RoPE supplies simple Fourier roots, Jordan-RoPE thickens these roots into finite Fourier jets, and ALiBi supplies the repeated unit-root affine direction. NTK-aware RoPE scaling fits the same structure as a spectral flow of simple Fourier roots: moving the frequency grid generates first Fourier-jet tangent directions, while higher Taylor directions generate higher jets. PJ-RoPE makes these jet directions explicit and learnable, and uses the resulting space to measure task-level sector selection. The framework separates scalar PJ-bias kernels from exact PJ-rotary feature transforms, introduces sector-gate, effective-mass, functional-energy, and leave-one-order-out diagnostics, and stabilizes high-order coordinates with LC/rapidity compactification. Controlled probes recover designed sectors; synthetic teachers show trainable use; small byte-level language runs favor NTK-aware RoPE plus affine recency; symbolic music-token streams keep LC/affine variants strong with measurable high-order corrections; and LC diagnostics quantify the stability-resolution tradeoff.