Overview

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

CardsList

129 papers12 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

  1. Parabolic Positional Encoding: A Vision-Centric, Principled, Extrapolatable, and General Approach

    Feb 1, 2026Christoffer Koo Øhrstrøm, Rafael I. Cabral Muchacho, Yifei Dong +4Positional Encoding

  2. RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

    Jan 21, 2026Yu Wu, Minsik Jeon, Jen-Hao Rick Chang +2Multi-View 3D ReconstructionNovel View Synthesis

  3. SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

    Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language ModelsVLMs for Autonomous Driving

  4. EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

    Dec 11, 2025Chao Gong, Depeng Wang, Zhipeng Wei +3Rotary Positional EmbeddingsMultimodal Large Language Models

  5. CineScale: Tuning-Free High-Resolution Video Generation

    Aug 21, 2025Gordon Chen, Haonan Qiu, Ning Yu +3Rotary Positional EmbeddingsVideo Diffusion Models

  6. Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings

    May 19, 2025Adrien Lagesse, Marc LelargeGraph Neural NetworksSynthetic Benchmark Generation