Length Generalization

Ask about this topic

Overview

Length generalization asks whether a model can extend a learned rule beyond the sequence lengths seen during training, which matters for reliable reasoning and long-form processing. Approaches include recurrent and state-space models, looped Transformers, relative positional encodings such as RoPE, and training methods like length curricula and regularization. Tests range from regular-language recognition, parity, and arithmetic to planning, long-context retrieval, and multimodal generation. Results depend strongly on task structure and whether a model learns reusable state updates rather than length-specific shortcuts, so no single architecture or positional method generalizes reliably across settings.

Latest papers

CardsList

47 papers12 papers in the last four weeks, up 71% on the four weeks before. 0.1% of all new papers.

  1. Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones

    Oct 26, 2026Marius Henry, Michael Rizvi-Martel, Michael HahnLanguage Model GeneralizationGeneralization Bounds

    LaCoCo, Saarland University, Saarland Informatics Campus · Zuse School ELIZA · Mila & Université de Montréal

  2. PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking

    Oct 20, 2026Pan Wang, Yihao Hu, Guangyuan Dong +8Length GeneralizationRecurrent Language Models

    Ant International · National University of Singapore

  3. Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes

    Oct 18, 2026Khashayar Gatmiry, Nathan W. Henry, Chloe H. Su +5Length GeneralizationTransformer Training Dynamics

    UC Berkeley & Apple · UC Berkeley · Kempner Institute, Harvard University +5

  4. Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech

    Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimLong-Form TTS SynthesisTTS Synthesis

    Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea

  5. Fitting Every Hint Is Not Enough: Premature Floods in Recurrent Algorithmic Reasoners

    Oct 10, 2026Amin Najafgholizadeh, Mohsen AfsharchiNeural Algorithmic ReasoningGraph Neural Networks

    University of Zanjan

  6. TraceRelay: Attention-Aligned Recurrence over Rolling Traces

    Oct 8, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungSequence ModelingPersistent Memory for Language Models

    College of Pharmacy, Chungnam National University · Department of Computer Science & Engineering, Chungnam National University

  7. HySPE: Positional Encoding via Symplectic Dual Shears

    Oct 7, 2026Zhongping JiPositional EncodingLength Generalization

  8. Exact-Solution Volume and Length Generalization in Transformers

    Oct 6, 2026Yijia Jessica Zhu, David ChiangTransformer ExpressivityTransformer

    University of Notre Dame

  9. Recurrent Looped Transformer

    Oct 6, 2026Yifan Zhang, Jichen Feng, Shihan Qin +2Length GeneralizationRecurrent Transformers

    Princeton University · University of Pennsylvania

  10. Length Generalization Needs Proper Regularization

    Oct 3, 2026Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1Length GeneralizationLanguage Model Generalization

    Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ELLIS Unit Lisbon +1

  11. Extreme Length Generalization in a Compact Recurrent Architecture for One-Shot Exploration

    Oct 1, 2026Izen Thornton, Aaron Shey, William SuMemory-Augmented Neural NetworksRecurrent Neural Networks

    FRANK Autonomous Systems Inc., frankautonomous.com · University of California, Berkeley

  12. Adapting Linear-Time Architectures for Tabular In-Context Learning

    Sep 28, 2026David Schnurr, Felix Sarnthein, Thomas Hofmann +1Tabular ICLTabular Foundation Models

    ETH Zürich · ELLIS Institute Tübingen · MPI-IS +1

  13. Universality and Generalization of Causal Transformers Across Context Lengths

    Sep 28, 2026Takashi Furuya, Maarten V. de Hoop, Gabriel PeyréNeural Network GeneralizationTransformer Expressivity

    Doshisha University, RIKEN AIP · Rice University · CNRS, ENS, PSL Université

  14. Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers

    Sep 27, 2026Jia Liang, Xi Jin, Liangming PanLength GeneralizationRecurrent Transformers

    Institute for Computational and Mathematical Engineering, Stanford University · MOE Key Laboratory of Computational Linguistics, School of Computer Science, Peking University

  15. Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

    Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length GeneralizationDelta-Rule Linear Attention

    ♡University of Freiburg · ♢Microsoft Research · •ELLIS Institute Tübingen +9

  16. Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

    Sep 9, 2026Vaibhav Singh, Pierre-André Noël, Torsten Scholak +2Blockwise Diffusion Language ModelsDiffusion Language Model Inference

    Mila · Concordia University · ServiceNow Research

  17. Learning Length-Extrapolatable Recurrent Models

    Sep 8, 2026Hanwen JiangBackpropagationTemporal Credit Assignment

    Adobe Research

  18. Length Generalization for Transformers via Compression

    Sep 8, 2026Georg Zetzsche, Hongjian Jiang, Andy Yang +4Transformer ExpressivityTransformer

    Max Planck Institute for Software Systems (MPI-SWS) · RPTU Kaiserslautern-Landau · University of Notre Dame

  19. Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

    Aug 31, 2026Takuya Ito, Ruchir Puri, Murray Campbell +1Transformer ExpressivityLength Generalization

    IBM Research

  20. Can Video World Models Track Unobserved World States?

    Aug 31, 2026Joonghyuk Shin, Yicong Hong, Jaesik Park +1Action-Conditioned World ModelsLength Generalization

    Seoul National University · Roblox

  21. On the Representational Geometry of Dynamic Programs

    Aug 25, 2026Richard F. M. Lim, Ruriko YoshidaDynamic ProgrammingRepresentation Geometry

    Departments of Mathematics and Computer Science, Bowdoin College, Brunswick, Maine · Department of Operations Research, Naval Postgraduate School, Monterey, California · Department of Mathematical Sciences, Aalborg University, Denmark

  22. Algebraic Decomposition Theory for Transformer Length Generalization

    Aug 13, 2026Andy Yang, Blerta Veseli, Corentin Barloy +5TransformerLength Generalization

    University of Notre Dame · 2Saarland University · 3Ruhr University Bochum +4

  23. Out-of-Length Scene Text Recognition: A Two-Axis Diagnosis and a Training-Free Fix

    Jul 25, 2026Zobeir Raisi, John ZelekScene Text RecognitionLength Generalization

    aElectrical Engineering Department, Chabahar Maritime University, Chabahar, Iran · bDepartment of Systems Design Engineering, University of Waterloo, Waterloo, N2L 3G1, ON, Canada

  24. When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

    Jul 22, 2026Tong Zhang, Junhao Hu, Yun Peng +1Inference-Time ScalingLength Generalization

    Fudan University · Peking University

  25. Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

    Jul 21, 2026Subham Singh, Ashutosh Mishra, Subha RautSoftmax AttentionRotary Positional Embeddings

    Department of Mathematics and Statistics, Mississippi State University · Independent Researcher

  26. On Locality and Length Generalization in Visual Reasoning

    Jul 10, 2026Pulkit Madan, Sanjay Haresh, Reza Ebrahimi +3Visual ReasoningLength Generalization

    Qualcomm AI Research*

  27. How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

    Jul 8, 2026Xinyi Wu, Siyuan Liu, Ali JadbabaieRotary Positional EmbeddingsLong-Context Language Modeling

    MIT IDSS · IIIS, Tsinghua University

  28. Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

    Jul 1, 2026Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1In-Context RetrievalLength Generalization

    UC Berkeley · UT Austin

  29. Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping

    Jun 29, 2026Hsun-Yu Kuo, El Mahdi Chayti, Patrik Reizinger +2Early StoppingOOD Generalization

    EPFL, Lausanne, Switzerland · Max Planck Institute for Intelligent Systems & ELLIS Institute, Tübingen, Germany

  30. ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory

    Jun 23, 2026Habibullah AkbarHybrid AttentionLong-Context Language Modeling

    Kreasof AI Jakarta, Indonesia