Length Generalization

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 32

All topics

TraceRelay: Attention-Aligned Recurrence over Rolling Traces
Oct 8, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungSequence Modeling Persistent Memory for Language Models
College of Pharmacy, Chungnam National University · Department of Computer Science & Engineering, Chungnam National University
HySPE: Positional Encoding via Symplectic Dual Shears
Oct 7, 2026Zhongping JiPositional Encoding Length Generalization
Exact-Solution Volume and Length Generalization in Transformers
Oct 6, 2026Yijia Jessica Zhu, David ChiangTransformer Expressivity Transformer
University of Notre Dame
Recurrent Looped Transformer
Oct 6, 2026Yifan Zhang, Jichen Feng, Shihan QinLength Generalization Recurrent Transformers
Princeton University · University of Pennsylvania
Extreme Length Generalization in a Compact Recurrent Architecture for One-Shot Exploration
Oct 1, 2026Izen Thornton, Aaron Shey, William SuMemory-Augmented Neural Networks Recurrent Neural Networks
FRANK Autonomous Systems Inc., frankautonomous.com · University of California, Berkeley
Universality and Generalization of Causal Transformers Across Context Lengths
Sep 28, 2026Takashi Furuya, Maarten V. de Hoop, Gabriel PeyréNeural Network Generalization Transformer Expressivity
Doshisha University, RIKEN AIP · Rice University · CNRS, ENS, PSL Université
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length Generalization Delta-Rule Linear Attention
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
Sep 14, 2026Vaibhav Singh, Pierre-André Noël, Torsten Scholak +2Blockwise Diffusion Language Models Diffusion Language Model Inference
Mila · Concordia University · ServiceNow Research
Learning Length-Extrapolatable Recurrent Models
Sep 8, 2026Hanwen JiangBackpropagation Temporal Credit Assignment
Adobe Research
Length Generalization for Transformers via Compression
Sep 8, 2026Georg Zetzsche, Hongjian Jiang, Andy Yang +4Transformer Expressivity Transformer
Max Planck Institute for Software Systems (MPI-SWS) · RPTU Kaiserslautern-Landau · University of Notre Dame
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
Aug 31, 2026Takuya Ito, Ruchir Puri, Murray Campbell +1Transformer Expressivity Length Generalization
IBM Research
Can Video World Models Track Unobserved World States?
Aug 31, 2026Joonghyuk Shin, Yicong Hong, Jaesik Park +1Action-Conditioned World Models Length Generalization
Seoul National University · Roblox
Algebraic Decomposition Theory for Transformer Length Generalization
Aug 13, 2026Andy Yang, Blerta Veseli, Corentin Barloy +5Transformer Length Generalization
University of Notre Dame · 2Saarland University · 3Ruhr University Bochum +4
Out-of-Length Scene Text Recognition: A Two-Axis Diagnosis and a Training-Free Fix
Jul 25, 2026Zobeir Raisi, John ZelekScene Text Recognition Length Generalization
aElectrical Engineering Department, Chabahar Maritime University, Chabahar, Iran · bDepartment of Systems Design Engineering, University of Waterloo, Waterloo, N2L 3G1, ON, Canada
When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Jul 22, 2026Tong Zhang, Junhao Hu, Yun Peng +1Inference-Time Scaling Length Generalization
Fudan University · Peking University
Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention
Jul 21, 2026Subham Singh, Ashutosh Mishra, Subha RautSoftmax Attention Rotary Positional Embeddings
Department of Mathematics and Statistics, Mississippi State University · Independent Researcher
On Locality and Length Generalization in Visual Reasoning
Jul 10, 2026Pulkit Madan, Sanjay Haresh, Reza Ebrahimi +3Visual Reasoning Length Generalization
Qualcomm AI Research*
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
Jul 8, 2026Xinyi Wu, Siyuan Liu, Ali JadbabaieRotary Positional Embeddings Long-Context Language Modeling
MIT IDSS · IIIS, Tsinghua University
Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
Jul 1, 2026Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1In-Context Retrieval Length Generalization
UC Berkeley · UT Austin
Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping
Jun 29, 2026Hsun-Yu Kuo, El Mahdi Chayti, Patrik Reizinger +2Early Stopping OOD Generalization
EPFL, Lausanne, Switzerland · Max Planck Institute for Intelligent Systems & ELLIS Institute, Tübingen, Germany
ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory
Jun 23, 2026Habibullah AkbarMemory-Augmented Language Models Long-Context Language Modeling
Kreasof AI Jakarta, Indonesia
Why Do Accumulated Transformations Extrapolate?
Jun 23, 2026Mahesh GodavartiRotary Positional Embeddings Length Generalization
A Carrot, Inc.
Randomized YaRN Improves Length Generalization for Long-Context Reasoning
Jun 22, 2026Manas Mehta, Fangcong Yin, Greg DurrettLong-Context Language Modeling LLM Fine-Tuning
New York University
POLARIS: Guiding Small Models to Write Long Stories
Jun 2, 2026Rishanth Rajendhran, Jenna Russell, Mohit Iyyer +1Length Generalization LLM Post-Training
University of Maryland · Google DeepMind
ReFLEX: Length-Generalizable CSI Denoising for MIMO-OFDM via Relative-Frequency Bias
May 29, 2026Zhibin Zhang, Robert Potekhin, Ziwei Wan +2Transformer Length Generalization
Moscow Institute of Physics and Technology (State University), Moscow, Russia · Yangtze Delta Region Academy, Beijing Institute of Technology (Jiaxing), Jiaxing 314019, China · School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China
Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
May 29, 2026Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3Rotary Positional Embeddings Attention Head Analysis
CENIA & Faculty of Mathematics UC Santiago, Chile · CENIA Santiago, Chile · IMC UC & CENIA Santiago, Chile
Length Generalization with Log-Depth Recurrent Units
May 25, 2026Charles Pert, Dalal Alrajeh, Alessandra RussoRecurrent Neural Networks Length Generalization
Department of Computing Imperial College London
On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length
May 4, 2026Sunghwan Kim, Junhee Cho, Beong-woo Kwak +6Long-Horizon Planning Length Generalization
Department of Artificial Intelligence, Yonsei University · †Work done during internship at Microsoft Research. · 2Microsoft Research.
Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
Apr 28, 2026Oliver Kraus, Yash Sarrof, Yuekun Yao +2Transformer Expressivity CoT Reasoning
1Saarland University · 2Ohio State University
Generalization in LLM Problem Solving: The Case of the Shortest Path
Apr 16, 2026Yao Tong, Jiayuan Ye, Anastasia Borovykh +1Inference-Time Scaling Length Generalization
National University of Singapore · Capital Fund Management · Google Research

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

TraceRelay: Attention-Aligned Recurrence over Rolling Traces

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

College of Pharmacy, Chungnam National University · Department of Computer Science & Engineering, Chungnam National University

Sequence Modeling Persistent Memory for Language Models Recurrent Transformers Attention Mechanisms Length Generalization

Open paper AskPDF arXiv

We present TraceRelay, an attention-aligned recurrent architecture that distributes persistent representations over a rolling sequence of low-dimensional traces. Local right looking attention forms increments from lower-layer representations; delivery is delayed until all attended inputs are in the causal past. A fixed additive phase recurrence accumulates the delayed increments, and left-looking attention reads the resulting residual augmented stream. A stride-wise prefix sum supports parallel prefill and bounded-buffer continuation. We study 36 small-model runs on Equal Repeats, bounded Dyck closing-type prediction, and causal Most-Freq generation, using three seeds per setting. At trained length 256, Equal Repeats models with recurrent phase inheritance reach 98.81-99.69% accuracy versus 50.73-51.63% for separately trained variants without inheritance, despite the latter receiving more updates. Accuracy drops sharply at lengths beyond the training range. At the longest evaluated lengths, models with more dimensions in the middle layer's recurrent traces perform better on Dyck (76.34% versus 55.49% close accuracy at length 4096), whereas models with fewer trace dimensions perform better on five-symbol Most-Freq (70.74% versus 55.60% exact generation at length 1024). These contrasting cases motivate further study of how the size of recurrent representations should be chosen for different tasks, without establishing a general rule across tasks or model configurations.

Momentum

Related topics

Latest papers 32