Position Extrapolation

Follow this topic Ask about this topic

Latest papers

8 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Learning to Extrapolate in Depth with Looped Transformers
Oct 10, 2026Huitao Yang, Jianhao Huang, Zixuan Wang +1Position Extrapolation Recurrent Transformers
University of California, Los Angeles · Princeton University
Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers
Sep 17, 2026Aras Kavuncu, Muhammad Burhan HafezMemory-Augmented Neural Networks Recurrent Transformers
University of Southampton, Electronics and Computer Science
Coupled-cluster molecular properties across the main group that extrapolate beyond training size
Aug 18, 2026Wenhao He, Xu Chen, Noah Song +10Quantum Chemistry Equivariant Neural Networks
Center for Computational Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Department of Chemistry, Emory University, Atlanta, GA 30322, USA · Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA +2
SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers
May 21, 2026Javad Rajabi, Kimia Shaban, Koorosh Roohi +2Rotary Positional Embeddings Diffusion Model Guidance
University of Toronto · Vector Institute
NeuroLiDAR: Adaptive Frame Rate Depth Sensing via Neuromorphic Event-LiDAR Fusion
May 16, 2026Darshana Rathnayake, Dulanga Weerakoon, Meera Radhakrishnan +1Depth Estimation Camera-LiDAR Fusion
Singapore Management University, Singapore · Singapore-MIT Alliance for Research and Technology Centre, Singapore · University of Technology Sydney, Australia
Decouple and Cache: KV Cache Construction for Streaming Video Understanding
May 3, 2026Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener +1Streaming Video Understanding KV Caching
National University of Singapore.
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
Apr 9, 2026Harsh Kohli, Srinivasan Parthasarathy, Huan Sun +1Position Extrapolation Implicit Reasoning in Language Models
The Ohio State University
Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders
Mar 15, 2026Jiaming Chu, Tao Wang, Lei JinPosition Extrapolation Variational Autoencoders
School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China

Oct 10, 2026J/K move · Enter open · S save

Learning to Extrapolate in Depth with Looped Transformers

Huitao Yang, Jianhao Huang, Zixuan Wang, Baharan Mirzasoleiman

University of California, Los Angeles · Princeton University

Position Extrapolation Recurrent Transformers Graph Transformer

Open paper AskPDF arXiv

Looped transformers reuse a single block across iterations, raising a natural question: how far can a block trained at one depth be safely extrapolated to greater depths? We study this question for all-pairs graph connectivity, where harder graphs require longer-range information propagation. We show that under exchangeability and symmetric positive initialization, population training at depth K converges to an update rule governed by a single effective parameter that balances graph propagation against fitting structure unresolved at training depth. This trade-off creates a finite extrapolation window: too few test-time loops provide insufficient propagation, while too many eventually create false connections. We derive explicit endpoints for this window and bound the graph hardness solvable beyond training as a function of training depth and hardness. In particular, a test depth chosen solely from the training distribution can guarantee recovery on substantially harder graphs, with a certified depth gain that grows exponentially as training depth outpaces training hardness. Experiments validate the predicted windows and show how compute budget and initialization affect the optimal training depth.