Overview

Length generalization asks whether a model can extend a learned rule beyond the sequence lengths seen during training, which matters for reliable reasoning and long-form processing. Approaches include recurrent and state-space models, looped Transformers, relative positional encodings such as RoPE, and training methods like length curricula and regularization. Tests range from regular-language recognition, parity, and arithmetic to planning, long-context retrieval, and multimodal generation. Results depend strongly on task structure and whether a model learns reusable state updates rather than length-specific shortcuts, so no single architecture or positional method generalizes reliably across settings.

Latest papers

CardsList

52 papers6 papers in the last four weeks, down 54% on the four weeks before. 0.0% of all new papers.

  1. Differentiable Exact Learning

    Nov 16, 2026Hristo Papazov, Francesco D'Angelo, Nicolas FlammarionLength Generalization

  2. How Finite Brains Generalize Without Bound: Spatial Memory as a Universal Computer

    Nov 5, 2026Yuhang He, Si WuLength GeneralizationRecurrent Neural Networks

  3. Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization

    Nov 3, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungRecurrent Neural NetworksLength Generalization

  4. How Does Chain-of-Thought Supervision Teach Models to Reason? A Theoretical Analysis

    Nov 1, 2026Kaito Takanami, Cengiz PehlevanError Propagation in LLM InferenceCoT Supervision

  5. A Positional Realizability Theory of Length Generalization in Single-Head Attention

    Oct 30, 2026Jiasong Han, Yunkai Chai, Jintao WangLength GeneralizationPositional Encoding

  6. Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones

    Oct 26, 2026Marius Henry, Michael Rizvi-Martel, Michael HahnLanguage Model GeneralizationGeneralization Bounds

  7. PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking

    Oct 20, 2026Pan Wang, Yihao Hu, Guangyuan Dong +8Length GeneralizationRecurrent Language Models

  8. Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes

    Oct 18, 2026Khashayar Gatmiry, Nathan W. Henry, Chloe H. Su +5Length GeneralizationTransformer Training Dynamics

  9. Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech

    Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimLong-Form TTS SynthesisTTS Synthesis

  10. TraceRelay: Attention-Aligned Recurrence over Rolling Traces

    Oct 8, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungSequence ModelingPersistent Memory for Language Models

  11. Exact-Solution Volume and Length Generalization in Transformers

    Oct 6, 2026Yijia Jessica Zhu, David ChiangTransformer ExpressivityTransformer

  12. Recurrent Looped Transformer

    Oct 6, 2026Yifan Zhang, Jichen Feng, Shihan Qin +2Length GeneralizationRecurrent Transformers

  13. Length Generalization Needs Proper Regularization

    Oct 3, 2026Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1Length GeneralizationLanguage Model Generalization

  14. Extreme Length Generalization in a Compact Recurrent Architecture for One-Shot Exploration

    Oct 1, 2026Izen Thornton, Aaron Shey, William SuMemory-Augmented Neural NetworksRecurrent Neural Networks

  15. Adapting Linear-Time Architectures for Tabular In-Context Learning

    Sep 28, 2026David Schnurr, Felix Sarnthein, Thomas Hofmann +1Tabular ICLTabular Foundation Models

  16. Universality and Generalization of Causal Transformers Across Context Lengths

    Sep 28, 2026Takashi Furuya, Maarten V. de Hoop, Gabriel PeyréNeural Network GeneralizationTransformer Expressivity

  17. Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers

    Sep 27, 2026Jia Liang, Xi Jin, Liangming PanLength GeneralizationRecurrent Transformers

  18. Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

    Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length GeneralizationDelta-Rule Linear Attention

  19. Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

    Sep 9, 2026Vaibhav Singh, Pierre-André Noël, Torsten Scholak +2Blockwise Diffusion Language ModelsDiffusion Language Model Inference

  20. Length Generalization for Transformers via Compression

    Sep 8, 2026Georg Zetzsche, Hongjian Jiang, Andy Yang +4Transformer ExpressivityTransformer

  21. Can Video World Models Track Unobserved World States?

    Aug 31, 2026Joonghyuk Shin, Yicong Hong, Jaesik Park +1Action-Conditioned World ModelsLength Generalization

  22. On the Representational Geometry of Dynamic Programs

    Aug 25, 2026Richard F. M. Lim, Ruriko YoshidaDynamic ProgrammingRepresentation Geometry

  23. Algebraic Decomposition Theory for Transformer Length Generalization

    Aug 13, 2026Andy Yang, Blerta Veseli, Corentin Barloy +5TransformerLength Generalization

  24. Out-of-Length Scene Text Recognition: A Two-Axis Diagnosis and a Training-Free Fix

    Jul 25, 2026Zobeir Raisi, John ZelekScene Text RecognitionLength Generalization

  25. Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

    Jul 21, 2026Subham Singh, Ashutosh Mishra, Subha RautSoftmax AttentionRotary Positional Embeddings