Positional Encoding

Follow this topic Ask about this topic

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

116 papers19 papers in the last four weeks, up 73% on the four weeks before. 0.1% of all new papers.

Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings
Apr 9, 2026Logan Hallee, Jason P. GleghornAttention Mechanisms Positional Encoding
Center for Bioinformatics and Computational Biology, University of Delaware · Synthyra, Newark, DE · Department of Biomedical Engineering, University of Delaware
Training-free Spatially Grounded Geometric Shape Encoding (Technical Report)
Apr 8, 2026Yuhang HePositional Encoding Geometric Representation Learning
Microsoft Research
On the Geometry of Positional Encodings in Transformers
Apr 6, 2026Giansalvo CirrincioneMultidimensional Scaling Positional Encoding
Laboratoire LTI, Université de Picardie Jules Verne Chemin du Thil, 80025 Amiens, France
Rethinking Position Embedding as a Context Controller for Multi-Reference and Multi-Shot Video Generation
Apr 4, 2026Binyuan Huang, Yuning Lu, Weinan Jia +3Multi-Shot Video Generation Identity-Preserving Video Generation
Wuhan University · University of Science and Technology of China · Hong Kong University of Science and Technology +1
Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models
Mar 25, 2026Shengli Zhou, Minghang Zheng, Feng Zheng +13D Spatial Reasoning Positional Encoding
Department of Computer Science and Engineering, Southern University of Science and Technology · Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University
Coordinate Encoding on Linear Grids for Physics-Informed Neural Networks
Mar 24, 2026Tetsuro Tsuchino, Motoki ShigaNeural PDE Solvers Physics-Informed ML
Graduate School of Engineering, Gifu University, Japan · International Center for Synchrotron Radiation Innovation Smart, Tohoku University, Japan · Unprecedented-scale Data Analytics Center, Tohoku University, Japan +3
What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers
Mar 17, 2026Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss +3Vision Transformer Positional Encoding
Center for Materials Research, Justus Liebig University, 35392 Giessen · Dyson School of Design Engineering, Imperial College London, London · Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus +1
ExPosST: Explicit Positioning with Adaptive Masking for LLM-Based Simultaneous Machine Translation
Mar 16, 2026Yuzhe Shang, Pengzhi Gao, Yazheng Yang +4Machine Translation Decoder-Only Language Models
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., Beijing, China · The University of Hong Kong, Hong Kong, China
Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
Mar 13, 2026Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj +7Spatial Reasoning in Language Models Visual Spatial Reasoning
Cohere Labs Community · Eisai Inc. · Indian Institute of Science, Bangalore +4
Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
Mar 12, 2026Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta +1Rotary Positional Embeddings Positional Encoding
Max Planck Institute for Software Systems · Microsoft, Hyderabad
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
Mar 11, 2026Lin Chen, Bolin Ni, Qi Yang +5Multimodal Large Language Models Long-Context Modeling
1MAIS, Institute of Automation, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · 3Tencent +1
Surgical Repair of Collapsed Attention Heads in ALiBi Transformers
Mar 10, 2026Palmer SchallonAttention Head Analysis Positional Encoding
Independent researcher
Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding
Mar 7, 2026Lance Legel, Qin Huang, Brandon Voelker +9World Model Learning Geospatial Representation Learning
Ecological Intelligence Lab Ecodash.ai · School of Complex Adaptive Systems Arizona State University · Geosensing Systems Engineering & Sciences Lab University of Houston +6
Rank-Factorized Implicit Neural Bias: Scaling Super-Resolution Transformer with FlashAttention
Mar 6, 2026Dongheon Lee, Seokju Yun, Jaegyun Im +1Image Super-Resolution Attention Mechanisms
University of Seoul · KAIST AI
Efficient Point Cloud Processing with High-Dimensional Positional Encoding and Non-Local MLPs
Mar 4, 2026Yanmei Zou, Hongshan Yu, Yaonan Wang +4Point Cloud Learning Point Cloud Classification
the School of Artificial Intelligence and Robotics, Quanzhou Institute of Industrial Design and Machine Intelligence Innovation, Hunan University, Changsha 410082, China. · the College of Engineering and Design, Hunan Normal University, Changsha 410081, China. · the College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China. +1
SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs
Feb 26, 2026Guanting Ye, Qiyan Zhao, Wenhao Yu +7Positional Encoding 3D VLMs
University of Macau · Shanghai Jiaotong University · University of Science and Technology of China +2
An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention
Feb 25, 2026Madhusudan Ghosh, Rishabh GuptaLong-Context Modeling Long-Context Language Model Evaluation
Indian Association for the Cultivation of Science, Kolkata, India · Bosch Research and Technology Centre, Bangalore, India
Trie-Aware Transformers for Generative Recommendation
Feb 25, 2026Zhenxiang Xu, Jiawei Chen, Sirui Chen +5Generative Recommendation Sequential Recommendation
State Key Laboratory of Blockchain and Data Security, Zhejiang University. · College of Computer Science and Technology, Zhejiang University. · Zhejiang University Hangzhou, China +2
RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
Jan 21, 2026Yu Wu, Minsik Jeon, Jen-Hao Rick Chang +2Multi-View 3D Reconstruction Novel View Synthesis
Carnegie Mellon University · Apple
SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language Models VLMs for Autonomous Driving
Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4
EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs
Dec 11, 2025Chao Gong, Depeng Wang, Zhipeng Wei +3Rotary Positional Embeddings Multimodal Large Language Models
1Fudan University · 2Ant Group · 3UC Berkley
CineScale: Tuning-Free High-Resolution Video Generation
Aug 21, 2025Gordon Chen, Haonan Qiu, Ning Yu +3Rotary Positional Embeddings Video Diffusion Models
Nanyang Technological University · Netflix Eyeline Studios
ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models
May 27, 2025Bozhou Li, Wentao ZhangRotary Positional Embeddings Vision-Language Models
Peking University Beijing, China
Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings
May 19, 2025Adrien Lagesse, Marc LelargeGraph Neural Networks Synthetic Benchmark Generation
INRIA, École Normale Supérieure - PSL, Paris, France.
Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis
Nov 26, 2024Akshita Gupta, Tatiana Likhomanenko, Karren Dai Yang +3Unified Multimodal Models Audio-Visual Synchronization
TU Darmstadt · Apple
HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding
Oct 30, 2023Giorgio AngelottiRelative Positional Encoding Positional Encoding
Independent Researcher.

Apr 9, 2026J/K move · Enter open · S save

Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddings

Logan Hallee, Jason P. Gleghorn

Center for Bioinformatics and Computational Biology, University of Delaware · Synthyra, Newark, DE · Department of Biomedical Engineering, University of Delaware

Attention Mechanisms Positional Encoding

Open paper AskPDF arXiv

Bidirectional transformers are the foundation of many sequence modeling tasks across natural, biological, and chemical language domains, but they are permutation-invariant without explicit positional embeddings. In contrast, unidirectional attention inherently encodes positional information through its triangular mask, enabling models to operate without positional embeddings altogether. Here, we introduce Dual Triangle Attention, a novel bidirectional attention mechanism that separates the query-key subspace of each attention head into two complementary triangular masks: one that attends to past-and-self positions and one that attends to future-and-self positions. This design provides bidirectional context while maintaining the causal mask's implicit positional inductive bias in both directions. Using PyTorch's flex_attention, Dual Triangle Attention is implemented as a single compiled kernel call with no additional parameters beyond standard multi-head attention. We evaluated Dual Triangle Attention across three settings: (1) a synthetic argmax position probe, (2) masked language modeling (MLM) on natural language, and (3) MLM on protein sequences. In the argmax task, both Dual Triangle Attention and causal attention learn positional information without explicit positional embeddings, whereas standard bidirectional attention cannot. In the MLM experiments, Dual Triangle Attention with Rotary Positional Embeddings (RoPE) achieved the best context extension performance and strong performance across the board. These findings suggest that Dual Triangle Attention is a viable attention mechanism for bidirectional transformers, with or without positional embeddings.