Positional Encoding

Follow this topic Ask about this topic

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

115 papers18 papers in the last four weeks, up 64% on the four weeks before. 0.1% of all new papers.

Wavenumber-Aware Deep Learning for Robust Analysis of Raman Spectra
Oct 22, 2026Jihwan Kim, Jaeseong Heo, Yujin Kim +7Neural Network Robustness Positional Encoding
Department of Chemistry, Kookmin University, 77 Jeongneung-ro, Seongbuk-gu, Seoul, 02707, Republic of Korea. · BB21+, Department of Chemistry, Pukyong National University, Busan, 48513, Republic of Korea. · Department of Chemical Engineering, Myongji University, Yongin, 17058, Republic of Korea. +4
HRoPE: Homography Relative Positional Encoding
Oct 20, 2026Huachen Gao, Yang Yang, Xiaoqian Liang +6Positional Encoding Relative Positional Encoding
Zhejiang University · Westlake University · Huazhong University of Science and Technology +2
Copying Under Repetition: From Prefix Matching to Position Hashing
Oct 19, 2026Anxin Guo, Tianlong Huang, Chenxiao Yang +1Transformer Repetition in Language Generation
Northwestern University · Duke University · Toyota Technological Institute at Chicago
Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting
Oct 14, 2026Xinjie Lan, Yunbo Mei, Leheng Zhou +1Long-Term Time Series Forecasting Non-Stationary Time Series Forecasting
Rice University · University of Hong Kong · University of North Carolina at Charlotte
RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can
Oct 9, 2026Yi-Jen Shih, Puyuan Peng, Haohe Liu +2Full-Duplex Spoken Dialogue Systems Positional Encoding
The University of Texas at Austin · FAIR, Meta Superintelligence Labs
GLIMPSE: Global-Local Indexing with Multi-Precision Spectral Encoding for Explorable Video World Models
Oct 9, 2026Zhufeng Xu, Henglei Lv, Bailin Deng +5Video World Models Memory-Augmented Video Understanding
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Cardiff University +2
HySPE: Positional Encoding via Symplectic Dual Shears
Oct 7, 2026Zhongping JiPositional Encoding Length Generalization
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language Modeling Hybrid Attention
Fudan University · Shanghai Innovation Institute · OpenMOSS Team
LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition
Oct 6, 2026Shikhar Srivastava, Christopher KananTransformer Positional Encoding
University of Rochester
WavePrune: One period is often enough for RoPE
Oct 3, 2026Guancheng Du, Luotian Huang, Shaowen Wang +2Rotary Positional Embeddings Positional Encoding
Tsinghua University
Path Laplacian Encodings for Directed Graphs
Oct 3, 2026Lydia Mezrag, Semih Cantürk, Michael Perlmutter +2Graph Positional Encoding Positional Encoding
Université de Montréal · Mila – Quebec AI Institute · Boise State University +1
Feedforward Novel View Synthesis for Heterogeneous Cameras
Oct 2, 2026Meng Wei, Cheng Zhang, Boying Li +4Novel View Synthesis Positional Encoding
Monash Univeristy · Shanghai Jiaotong University · Nanyang Technological University
Higher-Order Positional Encodings for Graph Representation Learning
Oct 1, 2026Caleb Stam, Aagrim Hoysal, Sanjukta KrishnagopalGraph Representation Learning Positional Encoding
University of California, Santa Barbara
Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval
Sep 29, 2026Eric Enouen, Sainyam GalhotraIn-Context Retrieval Semantic Search
Cornell University
How Local Mixing Encodes Relative Position in Global NoPE Attention
Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated Attention Positional Encoding
Zyphra Research
Pattern Formation in Transformers
Sep 29, 2026Erkan Turan, Gaspard Abel, Maks OvsjanikovTransformer Transformer Attention
LIX, École Polytechnique, IP Paris, France · Centre Borelli, ENS Paris-Saclay, Université Paris-Saclay, France · Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS, 75006 Paris, France
DirectUV: Image-Conditioned UV Texture Generation with Surface-Aware Positional Encoding
Sep 28, 2026Jiantao Lin, Yingjie Xu, Mingzhi Sheng +3Latent Diffusion Models Positional Encoding
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
Sep 28, 2026Jarod Lévy, Mathurin Videau, Jad Yehya +3Rotary Positional Embeddings Long-Context Language Modeling
Meta AI, Paris · Inria, Université Paris-Saclay, Palaiseau, France
MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception
Sep 27, 2026Yuhao Li, Louie Hong Yao, Tianyi Shi +4Spatiotemporal Reasoning Positional Encoding
Wave Intelligence Lab · Department of Computer Science and Engineering, The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory
Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates
Sep 26, 2026Ahmad Bdeir, Niels LandwehrVision Transformer Hyperbolic Representation Learning
Data Science Group University of Hildesheim Hildesheim, Germany
Enhancing Transformer Representations of Symbolic ODE Expressions
Sep 21, 2026Xiyue Fan, Adam Prugel-Bennett, Stuart E. MiddletonTransformer Positional Encoding
University of Southampton
Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure
Sep 20, 2026Shuyang XiangPositional Encoding
G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity
Sep 14, 2026Shuo Zhang, Xin Su, Wei Wang +8Rotary Positional Embeddings 3D ViTs
Wuhan University, Wuhan, China · Rongyun Robot (Guizhou) Co., Ltd. · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Degraded but Not Entirely Ineffective: PE-Based Deformable Graph Neural Networks
Sep 12, 2026Jinhua Wu, Xinliang ZhangGraph Neural Networks Positional Encoding
Henan Polytechnic University, Jiaozuo, 454000, Henan, China
Distance generalization in transformers: why bother with positional encoding?
Sep 10, 2026Daniel Henrik Nevermann, Claudius GrosOOD Generalization Positional Encoding
Institute for Theoretical Physics, Goethe University Frankfurt, Germany
Physics-guided deep metric learning with continuous time embeddings for open-world radar pulse de-interleaving
Sep 8, 2026Vikas Agnihotri, Jasleen KaurMetric Learning Physics-Informed ML
Independent Researcher India · National Institute of Technology, Rourkela India
Content-Based Addressing for Long Context
Sep 7, 2026Mahesh GodavartiRotary Positional Embeddings Long-Context Language Modeling
A Carrot, Inc.
MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation
Sep 1, 2026Zhijian Qiao, Xinjiang Wang, Jiajie Chen +5Rotary Positional Embeddings Camera-Controlled Video Generation
The Hong Kong University of Science and Technology · Zhuoyu Technology
Converse and Collision-Based Achievability for Node Localization with Hybrid Distance-Spectral Graph Positional Encodings
Aug 31, 2026Zimo Yan, Yifan Li, Hao Li +4Representation Identifiability Positional Encoding
National University of Defense Technology, Changsha, China · Wuhan University, Wuhan, China
Higher-Dimensional Rotary Position Embedding
Aug 30, 2026Yixing Li, Ruobing Xie, Yudong Zhang +3Rotary Positional Embeddings Transformer Attention
The Chinese University of Hong Kong · Tencent Hunyuan · Tsinghua University +1

Oct 22, 2026J/K move · Enter open · S save

Wavenumber-Aware Deep Learning for Robust Analysis of Raman Spectra

Jihwan Kim, Jaeseong Heo, Yujin Kim, Sang Bin Yoon+6

Department of Chemistry, Kookmin University, 77 Jeongneung-ro, Seongbuk-gu, Seoul, 02707, Republic of Korea. · BB21+, Department of Chemistry, Pukyong National University, Busan, 48513, Republic of Korea. · Department of Chemical Engineering, Myongji University, Yongin, 17058, Republic of Korea.+4

Neural Network Robustness Positional Encoding Raman Spectroscopy Spectral Methods

Open paper AskPDF arXiv

Machine-learning models for spectroscopy usually treat a spectrum as a fixed-grid intensity vector, leaving the physical coordinate implicit in the array index. Such models are tied to a single sampling grid, spectral range and resolution, which limits their use when measurement conditions change. Here we represent each spectral point as a pair of intensity and wavenumber and compute Transformer positional encodings from the wavenumber itself. On Raman spectra of three bacterial species and their synthetic mixtures, this representation remained robust to noise, sparser sampling and partial spectral coverage across convolutional, residual and Transformer networks. In controlled comparisons, wavenumber-based encoding outperformed index-based and learned encodings under partial observation and retained its performance on sampling grids never seen during training, where index-based encoding degraded. These findings were reproduced on a public 30-isolate bacterial benchmark. Explicit physical coordinates offer a simple route to robust spectral analysis under variable measurement conditions.