Overview

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

CardsList

129 papers12 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

  1. MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception

    Sep 27, 2026Yuhao Li, Louie Hong Yao, Tianyi Shi +4Spatiotemporal ReasoningPositional Encoding

  2. Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates

    Sep 26, 2026Ahmad Bdeir, Niels LandwehrVision TransformerHyperbolic Representation Learning

  3. Enhancing Transformer Representations of Symbolic ODE Expressions

    Sep 21, 2026Xiyue Fan, Adam Prugel-Bennett, Stuart E. MiddletonTransformerPositional Encoding

  4. G-ray: Ray-Level Relative Geometric Position Encoding in Multi-View Vision Transformers under Camera Heterogeneity

    Sep 14, 2026Shuo Zhang, Xin Su, Wei Wang +8Rotary Positional Embeddings3D ViTs

  5. Degraded but Not Entirely Ineffective: PE-Based Deformable Graph Neural Networks

    Sep 12, 2026Jinhua Wu, Xinliang ZhangGraph Neural NetworksPositional Encoding

  6. Distance generalization in transformers: why bother with positional encoding?

    Sep 10, 2026Daniel Henrik Nevermann, Claudius GrosOOD GeneralizationPositional Encoding

  7. Physics-guided deep metric learning with continuous time embeddings for open-world radar pulse de-interleaving

    Sep 8, 2026Vikas Agnihotri, Jasleen KaurMetric LearningPhysics-Informed ML

  8. MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation

    Sep 1, 2026Zhijian Qiao, Xinjiang Wang, Jiajie Chen +5Rotary Positional EmbeddingsCamera-Controlled Video Generation

  9. Converse and Collision-Based Achievability for Node Localization with Hybrid Distance-Spectral Graph Positional Encodings

    Aug 31, 2026Zimo Yan, Yifan Li, Hao Li +4Representation IdentifiabilityPositional Encoding

  10. Higher-Dimensional Rotary Position Embedding

    Aug 30, 2026Yixing Li, Ruobing Xie, Yudong Zhang +3Rotary Positional EmbeddingsTransformer Attention

  11. Real-Valued Hyperdimensional Sequence Representations with Hadamard Product Binding and Shift Equivariance

    Aug 28, 2026Kenny Schlegel, Dmitri A. Rachkovskij, Denis Kleyko +3Positional EncodingHyperdimensional Computing

  12. Loopy: Seamless Video Loop Generation via Anchored Looping Shift of Positional Embedding

    Aug 24, 2026Haotian Dong, Wenjing Wang, Chen Li +3Video GenerationVideo Diffusion Models

  13. Decoupled Temporal Encoding for Generative Recommendation

    Aug 17, 2026Pengfei Jia, Jingjian Wang, Jingmao Li +2Recommender SystemsGenerative Recommendation

  14. Do Geometry-Aware Positional Encodings Help Transformers in Spatial Imperfect-Information Games?

    Aug 15, 2026Wenji FuPositional EncodingImperfect-Information Games

  15. Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

    Aug 6, 2026Haris Riaz, Hyungji Kim, Mihai SurdeanuLanguage Model PretrainingPositional Encoding

  16. When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

    Aug 4, 2026Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2Language Model PretrainingLong-Context Retrieval

  17. MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models

    Aug 4, 2026Tong Ling, Hang Lei, Feng Xiao +5Rotary Positional EmbeddingsMasked Diffusion Models

  18. Local-Frame Rotary Attention for Antibody-Specific Epitope Prediction

    Aug 2, 2026Chuanliu Fan, Junjie Wu, Nan Yu +1Rotary Positional EmbeddingsPositional Encoding

  19. Enhancing Transformer-based Routing by Encoding Distance via Relative Positional Encoding

    Jul 21, 2026Leyre Encío, Daniel Fuertes, Carlos R. del-Blanco +1Relative Positional EncodingCombinatorial Optimization

  20. An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers

    Jul 20, 2026Cheng Huan, Hongwei YuanLost-in-the-Middle EffectTransformer Attention

  21. Enhancing the KidSat Model: Integrating Geographical Encoding and Data Quality Assessment for Childhood Poverty Prediction

    Jul 9, 2026Hou Hin Ip, Ka Nam Lam, Joshua Man Yu Ng +4Geospatial Representation LearningRemote Sensing