Long-Context Modeling

Ask about this topic

Overview

Long-context modeling aims to let models use information spread across lengthy text, conversations, documents, videos, and time series without losing relevant details. Approaches include sparse or hierarchical attention, recurrent state-space models, external and compressed memory, retrieval, and training methods that extend context or teach evidence use. Researchers evaluate these systems on tasks such as long-document question answering, multi-hop retrieval, dialogue memory, and long-video understanding, while also studying positional encodings and how performance changes with context length. A central tension is that larger windows do not guarantee reliable reasoning: models may miss distant evidence or be distracted by irrelevant material, while efficient methods must balance computation and memory against recall and accuracy.

Latest papers

CardsList

139 papers18 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

  1. Long-Context Encoder Models for Polish Language Understanding

    Mar 12, 2026Sławomir Dadas, Rafał Poświata, Marek Kozłowski +4Long-Context Language ModelingLong-Context Modeling

    National Information Processing Institute, Warsaw, Poland · PKO Bank Polski, Warsaw, Poland

  2. Why Attend to Everything? Focus is the Key

    Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention MechanismsEfficient Attention

    Sapient Intelligence · University of Alberta · Tsinghua University

  3. Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding

    Mar 11, 2026Lin Chen, Bolin Ni, Qi Yang +5Multimodal Large Language ModelsLong-Context Modeling

    1MAIS, Institute of Automation, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · 3Tencent +1

  4. Chow-Liu Ordering for Long-Context Reasoning in Chain-of-Agents

    Mar 10, 2026Naman Gupta, Vaibhav Singh, Arun Iyer +8Long-Context InferenceMulti-Agent LLM Systems

    Microsoft

  5. Automatic Cardiac Risk Management Classification using large-context Electronic Patients Health Records

    Mar 10, 2026Jacopo Vitale, David Della Morte, Luca Bacco +5Clinical Risk PredictionCardiovascular Risk Prediction

    Research Unit of Intelligent Health Technologies, Universit`a Campus Bio-Medico di Roma, Rome, Italy. · Central Diagnostic Laboratory, University Medical Center Utrecht, Utrecht, The Netherlands.

  6. Recursive Models for Long-Horizon Reasoning

    Mar 2, 2026Chenxiao Yang, Nathan Srebro, Zhiyuan LiRecursive Language ModelsLanguage Modeling

    Toyota Technological Institute at Chicago

  7. Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios

    Mar 1, 2026Nisrine Ait KhayiHybrid Transformer-State Space Model ArchitecturesText Summarization

    Independent Researcher · University of Memphis

  8. Towards Long-Form Spatio-Temporal Video Grounding

    Feb 26, 2026Xin Gu, Bing Fan, Jiali Yao +5Temporal Video GroundingLong-Context Modeling

    University of Chinese Academy of Sciences · 2UNT · Institute of Software Chinese Academy of Sciences +2

  9. SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning

    Feb 26, 2026Sanjay Kariyappa, G. Edward SuhKV-Cache CompressionLLM Inference Acceleration

    NVIDIA

  10. An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention

    Feb 25, 2026Madhusudan Ghosh, Rishabh GuptaLong-Context ModelingLong-Context Language Model Evaluation

    Indian Association for the Cultivation of Science, Kolkata, India · Bosch Research and Technology Centre, Bangalore, India

  11. Frayed RoPE and Long Inputs: A Geometric Perspective

    Feb 24, 2026Davis Wertheimer, Aozhong Zhang, Derrick Liu +2Rotary Positional EmbeddingsLong-Context Language Modeling

    IBM Research · University at Albany, SUNY

  12. Avey-B

    Feb 17, 2026Devang Acharya, Mohammad HammoudLanguage ModelingTransformer

    Avey AI

  13. How to Train Your Long-Context Visual Document Model

    Feb 16, 2026Austin VeselkaVisual Question AnsweringVision-Language Models

    LightOn

  14. Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts

    Dec 3, 2025Yijun LiaoLLM AlignmentDirect Preference Optimization

  15. Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

    Nov 4, 2025Amanda Bertsch, Adithya Pratapa, Teruko Mitamura +2LLM EvaluationLong-Context Language Model Evaluation

    Carnegie Mellon University, Language Technologies Institute

  16. Contextual Representation Flow Across Depth in State-Space and Transformer Architectures

    Oct 8, 2025Nhat M. Hoang, Do Xuan Long, Cong-Duy Nguyen +3State Space ModelsState Space Language Models

    Nanyang Technological University, Singapore · National University of Singapore

  17. PERK: Long-Context Reasoning as Test-Time Learning

    Jul 8, 2025Zeming Chen, Angelika Romanou, Gail Weiss +1Memory-Augmented Language ModelsLong-Context Modeling

    EPFL

  18. LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

    Feb 20, 2025Yansheng Mao, Yufei Xu, Jiaqi Li +5Long-Context QALLM Fine-Tuning

    Institute for Artificial Intelligence, Peking University · School of Electronics Engineering and Computer Science, Peking University · State Key Laboratory of General Artificial Intelligence, BIGAI

  19. RotRNN: Modelling Long Sequences with Rotations

    Jul 9, 2024Kai Biegun, Rares Dolga, Jake Cunningham +1Recurrent Neural NetworksLong-Context Modeling

    UCL AI Centre London, UK · UCL AI Centre & UiPath London, UK