Overview

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

CardsList

129 papers12 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

  1. Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs

    Nov 11, 2026Gaurav Sarkar, Syed Affan DaimiLong-Context Language ModelingDiffusion Language Models

  2. Disentangling Multimodal Rotary Positional Embedding in Vision-Language Models

    Nov 9, 2026Zhaoyi Liu, Haichuan Zhang, Xiaochuan Gong +5Positional EncodingPosition Bias in VLMs

  3. CABrain: A Correlation-Aware Brain Foundation Model for EEG Decoding

    Nov 9, 2026SuGil Choi, Mooseop Kim, Chi Yoon JeongEEG Foundation ModelsEEG Decoding

  4. Query-Guided RoPE Modulation for Lost-in-the-Middle Mitigation in Long-Context LLMs

    Nov 6, 2026Peiyu Zhang, Heng Ping, Shinibali Bhattacharyya +6Lost-in-the-Middle EffectLong-Context Language Model Inference

  5. Positional Encoding is All You Need for Scalable Equivariance Constraint Relaxation

    Nov 3, 2026Hagay Michaeli, Haggai Maron, Daniel SoudryEquivariant Neural NetworksNeural Network Optimization

  6. ProAlign: Progressive Positional and Prototype-Guided Alignment for Aerial-Ground Person Re-Identification

    Nov 3, 2026Qiao Li, Jing Chen, Cong Wu +3Person Re-IdentificationCross-Modal Alignment

  7. A Positional Realizability Theory of Length Generalization in Single-Head Attention

    Oct 30, 2026Jiasong Han, Yunkai Chai, Jintao WangLength GeneralizationPositional Encoding

  8. Structured Time-Frequency Representation for Channel Prediction

    Oct 25, 2026Ning Yang, Yuming Xian, Tong Li +2Channel State PredictionPositional Encoding

  9. POD-RoPE: Rotary Position Encoding over Physical Observation Distributions for Embodied AI

    Oct 24, 2026Xiaodi Huang, Zilin Wang, Jinpeng Liu +3Positional EncodingRotary Positional Embeddings

  10. Wavenumber-Aware Deep Learning for Robust Analysis of Raman Spectra

    Oct 22, 2026Jihwan Kim, Jaeseong Heo, Yujin Kim +7Neural Network RobustnessPositional Encoding

  11. HRoPE: Homography Relative Positional Encoding

    Oct 20, 2026Huachen Gao, Yang Yang, Xiaoqian Liang +6Positional EncodingRelative Positional Encoding

  12. Copying Under Repetition: From Prefix Matching to Position Hashing

    Oct 19, 2026Anxin Guo, Tianlong Huang, Chenxiao Yang +1TransformerRepetition in Language Generation

  13. Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting

    Oct 14, 2026Xinjie Lan, Yunbo Mei, Leheng Zhou +1Long-Term Time Series ForecastingNon-Stationary Time Series Forecasting

  14. RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can

    Oct 9, 2026Yi-Jen Shih, Puyuan Peng, Haohe Liu +2Full-Duplex Spoken Dialogue SystemsPositional Encoding

  15. GLIMPSE: Global-Local Indexing with Multi-Precision Spectral Encoding for Explorable Video World Models

    Oct 9, 2026Zhufeng Xu, Henglei Lv, Bailin Deng +5Video World ModelsMemory-Augmented Video Understanding

  16. Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

    Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language ModelingHybrid Attention

  17. LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

    Oct 6, 2026Shikhar Srivastava, Christopher KananTransformerPositional Encoding

  18. WavePrune: One period is often enough for RoPE

    Oct 3, 2026Guancheng Du, Luotian Huang, Shaowen Wang +2Rotary Positional EmbeddingsPositional Encoding

  19. Path Laplacian Encodings for Directed Graphs

    Oct 3, 2026Lydia Mezrag, Semih Cantürk, Michael Perlmutter +2Graph Positional EncodingPositional Encoding

  20. Feedforward Novel View Synthesis for Heterogeneous Cameras

    Oct 2, 2026Meng Wei, Cheng Zhang, Boying Li +4Novel View SynthesisPositional Encoding

  21. Higher-Order Positional Encodings for Graph Representation Learning

    Oct 1, 2026Caleb Stam, Aagrim Hoysal, Sanjukta KrishnagopalGraph Representation LearningPositional Encoding

  22. How Local Mixing Encodes Relative Position in Global NoPE Attention

    Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated AttentionPositional Encoding

  23. Pattern Formation in Transformers

    Sep 29, 2026Erkan Turan, Gaspard Abel, Maks OvsjanikovTransformerTransformer Attention

  24. DirectUV: Image-Conditioned UV Texture Generation with Surface-Aware Positional Encoding

    Sep 28, 2026Jiantao Lin, Yingjie Xu, Mingzhi Sheng +3Latent Diffusion ModelsPositional Encoding

  25. RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings

    Sep 28, 2026Jarod Lévy, Mathurin Videau, Jad Yehya +3Rotary Positional EmbeddingsLong-Context Language Modeling