Length Generalization
Overview
Length generalization asks whether a model can extend a learned rule beyond the sequence lengths seen during training, which matters for reliable reasoning and long-form processing. Approaches include recurrent and state-space models, looped Transformers, relative positional encodings such as RoPE, and training methods like length curricula and regularization. Tests range from regular-language recognition, parity, and arithmetic to planning, long-context retrieval, and multimodal generation. Results depend strongly on task structure and whether a model learns reusable state updates rather than length-specific shortcuts, so no single architecture or positional method generalizes reliably across settings.
Latest papers
52 papers6 papers in the last four weeks, down 54% on the four weeks before. 0.0% of all new papers.