Overview

Positional encoding supplies Transformers with information about where tokens occur, helping attention distinguish order, distance, and structure. Common approaches include learned or sinusoidal absolute embeddings, relative attention biases such as ALiBi, and rotary position embeddings (RoPE); newer methods encode geometry, time, or graph structure for images, videos, point clouds, and networks. Work also explores extending context length, reducing encoding and attention costs, and adapting positional signals to the input or task. A central tension is that methods that work well within training lengths may not generalize reliably to longer contexts, while gains in retrieval or spatial reasoning can come with trade-offs in other tasks.

Latest papers

129 papers12 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

Feb 1, 2026

Parabolic Positional Encoding: A Vision-Centric, Principled, Extrapolatable, and General Approach

We propose Parabolic Positional Encoding (PaPE), a parabola-based positional encoding for vision modalities in attention-based architectures. Given a set of vision tokens-such as from videos, event camera streams, images, or point clouds-our objective is to encode their positions while accounting for the characteristics of vision modalities. Prior works have largely extended positional encodings from 1D-sequences in language to nD-structures in vision, but only with partial account of vision characteristics. We address this gap by designing PaPE from principles distilled from prior work: translation invariance, rotation invariance (PaPE-RI), distance decay, directionality, and context awareness. Extrapolation experiments on ImageNet-1K show how PaPE extrapolates remarkably well, improving in absolute terms by up to 10.5 over the next-best encoding. Generality experiments on 8 datasets across 4 modalities show that PaPE is a general vision positional encoding, as PaPE matches the best baseline on 5 datasets and exceeds all on 2 datasets. The code is available at https://github.com/DTU-PAS/parabolic-position-encoding.
Jan 21, 2026

RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).
Dec 11, 2025

SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

End-to-end autonomous driving methods built on vision language models (VLMs) have undergone rapid development driven by their universal visual understanding and strong reasoning capabilities obtained from the large-scale pretraining. However, we find that current VLMs struggle to understand fine-grained 3D spatial relationships which is a fundamental requirement for systems interacting with the physical world. To address this issue, we propose SpaceDrive, a spatial-aware VLM-based driving framework that treats spatial information as explicit positional encodings (PEs) instead of textual digit tokens, enabling joint reasoning over semantic and spatial representations. SpaceDrive employs a universal positional encoder to all 3D coordinates derived from multi-view depth estimation, historical ego-states, and text prompts. These 3D PEs are first superimposed to augment the corresponding 2D visual tokens. Meanwhile, they serve as a task-agnostic coordinate representation, replacing the digit-wise numerical tokens as both inputs and outputs for the VLM. This mechanism enables the model to better index specific visual semantics in spatial reasoning and directly regress trajectory coordinates rather than generating digit-by-digit, thereby enhancing planning accuracy. Extensive experiments validate that SpaceDrive achieves state-of-the-art open-loop performance on the nuScenes dataset and the second-best Driving Score of 78.02 on the Bench2Drive closed-loop benchmark over existing VLM-based methods. Code is available at: https://github.com/zhenghao2519/SpaceDrive.
Dec 11, 2025

EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

Audio-Visual Large Language Models (AV-LLMs) face prohibitive computational costs of processing massive, redundant audio-visual tokens. Existing unimodal compression techniques fail to capture the heterogeneous and mutually influential information density of joint audio-visual signals. Furthermore, we identify a fundamental and overlooked theoretical bottleneck in sparse token reduction: positional aliasing. We demonstrate that aggressive sparse sampling on standard position-encoded sequences violates the Nyquist limit relative to the effective token interval, causing phase-wrapping collisions that corrupt temporal monotonicity. To address this, we introduce EchoingPixels, a framework for aliasing-resistant joint token reduction. Our Cross-Modal Semantic Sieve performs extractive selection on the synergistic audio-visual stream, dynamically allocating budgets based on joint-modality saliency rather than fixed per-modality ratios. To resolve positional aliasing, we derive Sync-RoPE, a spectral low-pass filter for Rotary Positional Embeddings that adapts encoding bandwidth to the sparse sampling rate, preserving monotonic temporal relationships in the reduced stream. Experiments show that EchoingPixels achieves performance comparable to full models using only 5-20% of original tokens, validating theoretically grounded sparse learning as a robust solution for efficient AV-LLMs. Code is available at https://github.com/CharlesGong12/EchoingPixels.
Aug 21, 2025

CineScale: Tuning-Free High-Resolution Video Generation

Video diffusion models have achieved remarkable progress in recent years, yet generating high-resolution videos remain a fundamental challenge. Most video generators are trained at limited spatial resolutions due to the scarcity of high-resolution 4K video data and the prohibitive computational cost of large-scale training on such data. Most video diffusion models are trained on 720p videos and are therefore effectively limited to generating videos at similar resolutions during inference. To address this gap, we propose CineScale. CineScale, to the best of our knowledge, is the first tuning-free inference framework enabling pretrained video diffusion models to generate high-quality videos at resolutions far beyond those seen during training. Our key observation is that generation quality degrades at higher resolutions because positional encodings shift beyond their training distribution, producing blurred details and structurally incoherent videos. To address this gap, we introduce Adaptively Rectified RoPE. Our extensive experiments show that CineScale enables pretrained diffusion models, despite never being trained on high-resolution data, to generate high-fidelity 4K video without any fine-tuning, improving local detail and sharpness while preserving temporal coherence. This demonstrates that high-resolution generation capabilities can be unlocked purely at inference time.
May 27, 2025

ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models

Currently, a prevalent approach for enhancing Vision-Language Models (VLMs) performance is to encode both the high-resolution version and the thumbnail of an image simultaneously. While effective, this method generates a large number of image tokens. When combined with the widely used Rotary Position Embedding (RoPE), its long-term decay property hinders the interaction between high-resolution tokens and thumbnail tokens, as well as between text and image. To address these issues, we propose ID-Align, which alleviates these problems by reordering position IDs. In this method, high-resolution tokens inherit IDs from their corresponding thumbnail token while constraining the overexpansion of positional indices. Our experiments conducted within the LLaVA-Next framework demonstrate that ID-Align achieves significant improvements, including a 6.09% enhancement on MMBench's relation reasoning tasks and notable gains across multiple benchmarks. Our code is available at the following link: https://github.com/zooblastlbz/ID-Align.
May 19, 2025

Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings

We propose a novel benchmarking methodology for graph neural networks (GNNs) based on the graph alignment problem, a combinatorial optimization task that generalizes graph isomorphism by aligning two unlabeled graphs to maximize overlapping edges. We frame this problem as a self-supervised learning task and present several methods to generate graph alignment datasets using synthetic random graphs and real-world graph datasets from multiple domains. For a given graph dataset, we generate a family of graph alignment datasets with increasing difficulty, allowing us to rank the performance of various architectures. Our experiments prove that there is an optimal task difficulty for having a statistically relevant ranking of different models and that, even on a structure-only task, anisotropic models perform better compared to isotropic ones. To further prove that our synthetic task capture meaningful information, we show its effectiveness for self-supervised GNN pre-training: the learned node embeddings can be leveraged as positional encodings by transformers for graph regression or can be used to reconstruct the full structure of the graph with 98%98\% accuracy. To support reproducibility and further research, we provide an open-source Python package to generate graph alignment datasets and benchmark new GNN architectures. The source code is available at https://github.com/adrien-lagesse/graph-alignment-benchmark.
Nov 26, 2024

Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis

Unified decoder-only transformers have shown promise for multimodal generation, yet how sequence design affects synchronization across modalities with heterogeneous sampling rates remains underexplored. We investigate this through video-text-to-speech (VTTS) synthesis, a controlled task requiring fine-grained temporal alignment between sparse text, video, and continuous speech. Using Visatronic, a unified decoder-only transformer trained on VoxCeleb2, we examine complementary modality contributions, positional encoding strategies, the effects of modality ordering on in-domain performance and cross-domain transfer, and phoneme-level timing errors. Global sequential indexing, which assigns position IDs in concatenation order, and co-temporal ordered indexing, which assigns identical IDs to temporally corresponding video and speech tokens, achieve comparable in-domain synchronization. Under domain shift, however, synchronization depends on the interaction between token ordering and position-ID design. Text-first co-temporal indexing transfers most robustly among the evaluated variants without requiring explicit timestamp metadata. Text and video provide complementary signals: text supports intelligibility, while video contributes temporal cues and emotional expressiveness. Modality ordering also introduces a trade-off: video-first ordering achieves stronger in-domain performance, whereas text-first ordering generalizes more robustly among the timestamp-informed prefix variants. Training on diverse, in-the-wild data is associated with more transferable behavior, although isolating its contribution requires targeted ablations. We introduce TimeSync, a phoneme-level synchronization metric that reveals temporal misalignments overlooked by frame-level metrics. Together, these findings establish VTTS as a testbed for understanding temporal synchronization in unified multimodal decoders.
Oct 30, 2023

HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding

In Transformer-based architectures, unmasked self-attention without positional information is permutation-equivariant with respect to the input sequence's tokens. To impose sequential order, token positions are typically encoded using a scheme with either fixed or learnable parameters. We introduce Hyperbolic Positional Encoding (HyPE), a method that utilizes hyperbolic functions' properties to encode tokens' relative positions. This approach biases the attention mechanism without the necessity of storing the O(L2)O(L^2) values of the bias tensor, with LL being the length of the input sequence. HyPE leverages preliminary concatenation operations and matrix multiplications, encoding relative distances by indirectly incorporating biases into the softmax computation. This design supports gradient backpropagation through the encoding and can be used with FlashAttention-2 under the width and scaling conditions below. We analytically demonstrate that, by careful hyperparameter selection, HyPE can approximate the causal attention bias of ALiBi \citep{press2021train} on a bounded range; model generalization beyond the pretraining length remains to be evaluated. The experimental evaluation of HyPE is proposed as a direction for future research.