Overview

Long-context modeling aims to let models use information spread across lengthy text, conversations, documents, videos, and time series without losing relevant details. Approaches include sparse or hierarchical attention, recurrent state-space models, external and compressed memory, retrieval, and training methods that extend context or teach evidence use. Researchers evaluate these systems on tasks such as long-document question answering, multi-hop retrieval, dialogue memory, and long-video understanding, while also studying positional encodings and how performance changes with context length. A central tension is that larger windows do not guarantee reliable reasoning: models may miss distant evidence or be distracted by irrelevant material, while efficient methods must balance computation and memory against recall and accuracy.

Latest papers

CardsList

142 papers10 papers in the last four weeks, down 41% on the four weeks before. 0.1% of all new papers.

  1. TreeAgent: Structure-Aware Memory Agent for Processing Long Hierarchical Data

    Nov 10, 2026Wei Gu, Weiwei Qi, Jiajun Tang +5Structured Memory for LLM AgentsMemory-Augmented Language Model Agents

  2. Recursive Sparse Attention: Million-Token Length Extrapolation at Logarithmic Cost

    Nov 4, 2026Ruitian Wang, Xinyu Yang, Minshen Zhang +1Sparse AttentionLong-Context Language Modeling

  3. Sparse Text Diffusion under Dual Budgets: Context Connectivity and Expert Activation

    Oct 30, 2026Alexandros Christoforos, Chadbourne Davis, Paul RajDiffusion Language ModelsLong-Context Modeling

  4. Topology Survives Shuffling: Robust Long-Context Memory Access Prediction for Graph Analytics

    Oct 29, 2026Dongyan Sun, Neelesh Gupta, Rajgopal Kannan +1Long-Context ModelingDistribution Shift Robustness

  5. Unlimited OCR Works 2

    Oct 29, 2026Shu Wei, Jingjing Wu, Qunyi Xie +11Optical Character RecognitionLong-Context Modeling

  6. GTFormer: A Syntax-Free Graph Network Alternative to Transformers for 8192×\times Length Extrapolation in Linear Time

    Oct 27, 2026Elisabeth FinkLong-Context ModelingSequence Modeling

  7. Why Full-Duplex Speech Models Collapse in Long Conversations

    Oct 23, 2026Yeonju Kim, Bella Godiva, Se Jin Park +1Full-Duplex Spoken Dialogue SystemsLong-Context Modeling

  8. Weighted State Souping for Efficient Long-Context Retrieval

    Oct 19, 2026William L. Tong, Aryo Lotfi, Emmanuel Abbe +3State Space Language ModelsLong-Context Retrieval

  9. Joint Subspace Sparse Attention for Efficient Long-Context Decoding

    Oct 13, 2026Jiwon Song, Yoongon Kim, Jae-Joon KimLLM Inference AccelerationSparse Attention

  10. FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

    Oct 12, 2026Zekun Shi, Alastair James Arthur PriceLong-Context Language ModelingLong-Context Modeling

  11. History Replays Itself: Unleashing Long-Context Capability in Linear Attention

    Oct 11, 2026Li Nanbo, Firas Laakom, Yilan Zhang +10Long-Context ModelingMemory-Augmented Language Models

  12. In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation

    Oct 9, 2026Matteo Rufolo, Dario Piga, Filippo Pura +1System IdentificationIn-Context Learning

  13. Understanding Mamba-3 Through Oscillatory Dynamics

    Oct 9, 2026Shutian Liang, Yue SongSequence ModelingSelective SSMs

  14. Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

    Oct 9, 2026Vakul Raghavan, Utkarsh Grover, Mingyang Mao +2Long-Context ModelingMemory-Augmented Neural Networks

  15. Learning infinite context windows in recurrent architectures via spatial neural computing

    Oct 7, 2026Aleix Salvador-Pomarol, Arthur N. Montanari, Earl K. Miller +2Recurrent Neural NetworksLong-Context Modeling

  16. PHBA: Prefix-State Hybrid Block Attention

    Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context ModelingLinear Attention

  17. How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models

    Oct 2, 2026Amir FathiLong-Context ModelingSequence Modeling

  18. StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning

    Sep 30, 2026Naen Xu, Wanqing Cui, Yibo Hu +5Temporal Reasoning in Language ModelsRL for Language Model Reasoning

  19. ReMem: Rethinking Perception and Memory in Long-Context Recommendation Agents

    Sep 29, 2026Haohao Qu, Yongcheng Jing, Chun Hin Chan +3Long-Context ModelingRecommender Systems

  20. SMat-Attention: Structured Long-Context Sequence Modeling

    Sep 28, 2026Emile Anand, Abdullah Ateyeh, Archer Wang +1Long-Context ModelingLinear Attention

  21. Universality and Generalization of Causal Transformers Across Context Lengths

    Sep 28, 2026Takashi Furuya, Maarten V. de Hoop, Gabriel PeyréNeural Network GeneralizationTransformer Expressivity

  22. Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding

    Sep 25, 2026Zhaoyuan Xia, Qinghongbing Xie, Yung Xiang Hue +7Query-Focused SummarizationLong-Context Reasoning in Language Models

  23. When Can Agents Forget Their Reasoning? ICLR for Long-Horizon Agent Context Compression

    Sep 24, 2026Mingxuan Wang, Fei Luo, Bo Wang +6Long-Context ModelingLong-Horizon LLM Agents