Positional Encoding

Follow this topic Ask about this topic

Recent momentum

-77%

3 papers in the last 28 days · 0.1% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

1 new paper

A weekly snapshot of new work published in Positional Encoding.

Period ending 2026-09-07

1 new paper

A weekly snapshot of new work published in Positional Encoding.

73 papers

Latest in Positional Encoding

Open your feed →

Distance generalization in transformers: why bother with positional encoding?
Sep 11, 2026Daniel Henrik Nevermann, Claudius GrosPositional Encoding Context Length
RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Video Diffusion Models Rotary Position Embedding
Converse and Collision-Based Achievability for Node Localization with Hybrid Distance-Spectral Graph Positional Encodings
Aug 31, 2026Zimo Yan, Yifan Li, Hao Li +4Positional Encoding Graph Laplacians
ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Models
Aug 10, 2026Róisín LuoPositional Encoding Sparsegpt
Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
Aug 9, 2026Jiguo LiRotary Position Embedding Positional Encoding
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
Aug 6, 2026Haris Riaz, Hyungji Kim, Mihai SurdeanuSyntactic Structure Positional Encoding
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Aug 4, 2026Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2Positional Encoding Degradation
MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Aug 4, 2026Tong Ling, Hang Lei, Feng Xiao +5Masked Diffusion Language Models Diffusion Language Models
Learning to Tessellate: Point Cloud Generation via Recursive Spectral Partitioning
Aug 3, 2026Monan Sun, Bangzhen Liu, Huaidong Zhang +1Point Clouds Polygons
SemPIC: Learning Semantic Position-Independent KV Caches
Jul 30, 2026Hui Xie, Peng Xiao, Yutong Deng +3Key-Value Cache Cache
Latent Stability Analysis of Malware Representations Under Feature-Space Perturbations
Jul 27, 2026Bamidele Ajayi, Ken McGarryMalware Families Evasion
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Jul 23, 2026Ji Ho BaeRotary Position Embedding Positional Encoding
Hash-QNeRF: Multiresolution Hash Encoding for Quantum Neural Radiance Fields
Jul 23, 2026Digonto Biswas, Tana Ballove, Anjan Bandyopadhyay +2Quantum Neural Networks Neural Radiance Fields
AI-Driven Surrogate Models for Predicting Electrode-Scale Discharge Behavior in Lithium-Ion Batteries
Jul 22, 2026Mengda Xing, Jean-Marie Lagniez, Alejandro FrancoLithium-Ion Batteries Electrochemical Models
Enhancing Transformer-based Routing by Encoding Distance via Relative Positional Encoding
Jul 21, 2026Leyre Encío, Daniel Fuertes, Carlos R. del-Blanco +1Multiplex Graph Transformers Positional Encoding
Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention
Jul 21, 2026Subham Singh, Ashutosh Mishra, Subha RautPositional Encoding Absolute Pose
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Jul 17, 2026Haodong Wen, Yiran Zhang, Yingfa Chen +1Protein Language Model Positional Encoding
PE-Field 4D: Video Generation Models as Canvas
Jul 17, 2026Yunpeng Bai, Haoxiang Li, Qixing HuangVideo Diffusion Models Camera Control
What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
Jul 14, 2026Farrukh RahmanVideo Temporal Grounding Recent Vision-Language Models
Eigenbasis-Independent Learnable Spectral Positional Encodings for Directed Graphs via Hermitian Block Krylov Subspaces
Jul 8, 2026Jiaqing Xie, Yuxin WangGraph Laplacians Positional Encoding
MemDefrag: Latent Memory Defragmentation for Large Language Models
Jul 7, 2026Ruiyi Yan, Zhuoyuan Mao, Yiwen GuoLarge Language Model Memory Memforest
Geometry of Ordinal Representations in Language Models
Jul 5, 2026Saksham Bassi, Sharvi TomarProtein Language Model Geometric Reasoning
Observable- and Positional-Encoding-Dependent Symmetry Readout from Neural Network Weights
Jul 3, 2026Naoya Chiba, Satoshi Sugiyama, Yuki UranishiSymmetry Model Weights
HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding
Jul 1, 2026Ji Ha Jang, Hayeon Kim, Chulwon Lee +2Contrastive Language-Image Pre-Training Efficient Long-Context Inference
DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers
Jun 30, 2026Shun Kenney, Teppei SuzukiPositional Encoding Multimodal Transformers
RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation
Jun 25, 2026Minghao Yin, Jiahao Lu, Wenbo Hu +3Video Diffusion Models Positional Encoding
Communicability-Inspired Positional Encoding (CIPE)
Jun 24, 2026Yipeng Zhang, Zhongtian Sun, Pietro Liò +1Multiplex Graph Transformers Positional Encoding
Randomized YaRN Improves Length Generalization for Long-Context Reasoning
Jun 22, 2026Manas Mehta, Fangcong Yin, Greg DurrettEfficient Long-Context Inference LLM Reasoning Methods
Robustness of Similarity-based Positional Encoding Under Rotations: Theoretical Analysis and Experimental Validation
Jun 16, 2026Andrea Santomauro, Luigi Portinale, Giorgio LeonardiPositional Encoding Rotation
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh +2Positional Encoding Equivalence
DreamX-World 1.0: A General-Purpose Interactive World Model
Jun 15, 2026DreamX Team, Yancheng Bai, Rui Chen +20Cinematic Video Generation World Models
ParkingTransformer: LLM-Enhanced End-to-End Trajectory Planning for Autonomous Parking
Jun 12, 2026Hauteng Wu, Xu Li, Dong Kong +4Autonomous Parking Autonomous Driving
Understanding Truncated Positional Encodings for Graph Neural Networks
Jun 11, 2026James Flora, Mitchell Black, Weng-Keen Wong +1Graph Neural Networks Positional Encoding
Positional Encoding in the Context of Memristor-Based Analog Computation for Automatic Speech Recognition
Jun 11, 2026Benedikt Hilmes, Nick Rossenbach, Ralf SchlüterIn-Memory Computing Ternary Memristive Logic
LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding
Jun 3, 2026Haocheng Xia, Mihir Pamnani, Hanxi Fang +2Dynamic Sparse Attention LLM Inference Efficiency
Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
Jun 1, 2026Qian Li, Xinyu Mao, Shang-Hua TengPositional Encoding Transformer Encoder
Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
May 31, 2026Athanasios ZerisPositional Encoding Rotary Position Embedding
Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
May 30, 2026Petr NyomaEfficient Long-Context Inference State Space Models
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
May 28, 2026Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1Self-Attention Mechanism Positional Encoding
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
May 28, 2026Pierre-Antoine Lequeu, Camille Barboule, Benjamin PiwowarskiPositional Encoding Transformer Encoder
Positional Encodings Anchor Spatial Structure in Vision Transformers: A Geometric Perspective on Robustness
May 28, 2026Mahmoud MannesPositional Encoding Vision Transformer
Benchmarking Positional Encoding Strategies for Transformer-Based EEG Foundation Models
May 28, 2026Ayse Betul Yuce, Sebastian StoberElectroencephalography Foundation Models Positional Encoding
Periodic RoPE for Infinite Context LLMs
May 27, 2026Simin HuoEfficient Long-Context Inference Context Length
LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation
May 26, 2026Gabriele Cesa, Thomas Hehn, Aleix Torres-Camps +4Parallel Decoding and Training Positional Encoding
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisTransformer Attention Positional Encoding
Building an Adversarial Malware Dataset by Family and Type: Generation, Evasion, and Poisoning Evaluation
May 25, 2026David Košťál, Martin JurečekMalware Classification Evasion
Weierstrass Positional Encoding for Vision Transformers
May 20, 2026Zhihang Xin, Rui Wang, Xitong Hu +1Positional Encoding Vision Transformer
MetaBackdoor: Exploiting Positional Encoding as a Backdoor Attack Surface in LLMs
May 14, 2026Rui Wen, Mark Russinovich, Andrew Paverd +2Backdoor Attacks Language Model Evasion Attacks
CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation
May 13, 2026Seonghyun Jin, Youngmin Kim, Sunwoo Park +1Camera Control Camera-Conditioned Video Generation
Rethinking Positional Encoding for Neural Vehicle Routing
May 12, 2026Chuanbo Hua, Federico Berto, Andre Hottung +8Vehicle Routing Problem Positional Encoding
A Proprioceptive-Only Benchmark for Quadruped State Estimation: ATE, RPE, and Runtime Trade-offs Between Filters and Smoothers
May 12, 2026Ylenia Nisticò, João Carlos Virgolino Soares, Joan Solà +1State Estimation Invariant Extended Kalman Filter
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
May 11, 2026Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +2Positional Encoding Transformer Attention
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Christopher Irwin +2Rotary Position Embedding Positional Encoding
Game-Theoretic Modeling of Heterogeneous Investor Interactions for Stock Price Forecasting
May 11, 2026Yong Zhang, Xinxiao Wu, Yunde Jia +1Matching Markets Financial Strategy Research
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
May 10, 2026Daniel Wolfson, Tal WagnerPositional Encoding Efficient Long-Context Inference
Prompt-Free and Efficient SAM2 Adaptation for Biomedical Semantic Segmentation via Dual Adapters
May 7, 2026Hinako Mitsuoka, Kazuhiro HottaMedical Image Segmentation Sam2
Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
May 5, 2026Yaobo ZhangPositional Encoding Rotation
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
May 5, 2026Jie Ou, Jinyu Guo, Shiyao Guo +5In-Context Learning Few-Shot Learning
Decouple and Cache: KV Cache Construction for Streaming Video Understanding
May 3, 2026Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener +1Streaming Video Key-Value Cache
Spiking Sequence Machines and Transformers
May 1, 2026Joy BosePositional Encoding Sequence Modeling

Sep 11, 2026 · cs.CLJ/K move · Enter open · S save

Distance generalization in transformers: why bother with positional encoding?

Daniel Henrik Nevermann, Claudius Gros

Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such as RoPE and ALiBi improve distance resolution relative to no positional encoding (NoPE)? (B) How does data diversity, the number of inter-token distances seen in training, affect performance? (C) When is distance transfer learning positive or negative? We present a thorough investigation, finding that it is paramount to improve our understanding of the underlying mechanisms.

Positional Encoding Context Length Distance

Open paper Ask about it PDF