Long-Context LLM Inference

LLM: Large Language Model

Ask about this topic

Overview

Long-context inference concerns serving language models over long documents, conversations, and agent histories, where attention computation and key-value (KV) cache storage can make prefill and generation expensive. Approaches include sparse attention and token selection, KV-cache compression and tiered storage, quantization, and speculative decoding; some systems also compare long prompts with compiling or retrieving information from external memory. Benchmarks such as LongBench and needle-in-a-haystack tests assess whether methods retain facts and support reasoning as context grows, but results depend on how evidence is distributed and what questions are asked. The central tension is reducing compute and memory without losing relevant evidence or increasing unsupported answers, alongside deciding when persistent memory is preferable to repeatedly processing a long context.

Latest papers

CardsList

14 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. Long-Context Drafting Does Not Require Long-Context Drafters

    Oct 14, 2026Aref Jafari, Vansh Bhatia, Matt Qin +5Speculative DecodingLLM Inference Acceleration

    Advanced Micro Devices Inc. (AMD) · University of Waterloo

  2. RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models

    Oct 9, 2026Ke Wan, Chen ChenRecurrent Language ModelsLLM Inference Acceleration

    Department of Computer Science, University of Virginia

  3. JustFit: Just-in-Time State Management for Local LLM Serving

    Sep 15, 2026Yuhua ChenLLM ServingOn-Device Language Model Inference

  4. Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache

    Aug 31, 2026Tong Yuan, Chengxi Liao, Zeyi WenMemory-Augmented Language ModelsKV Caching

    Data Science and Analytics Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou)

  5. KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

    Jul 27, 2026Shuo Wang, Fang Xi, Wenyuan Huang +2LLM ServingKV-Cache Management

    QiYuanLab Beijing, China

  6. CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

    Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3LLM Inference EfficiencyKV Caching

    Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany

  7. Inference Time Context Sparsity: Illusion or Opportunity?

    May 22, 2026Sahil Joshi, Prithvi Dixit, Agniva Chowdhury +5LLM Inference EfficiencyStructured Sparsity

    α Rice University · β UC Berkeley · γ IIT Bombay

  8. KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

    May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5KV-Cache OffloadingKV Caching

    Hong Kong University of Science and Technology, China · Xi’an Jiaotong University, China

  9. WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems

    May 8, 2026Juan M. HuertaKV CachingCounterexample-Guided Refinement

    Zinnia Tech Solutions 600 Steamboat Road Greenwich, CT 06830, USA

  10. From Speech to Profile: A Protocol-Driven LLM Agent for Psychological Profile Generation

    Apr 11, 2026Xingjian Yang, Yudong Yang, Zhixing Guo +3LLM Hallucination MitigationMemory-Augmented Language Model Agents

    School of Science&Engineering, South China University of Technology, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China · Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, China +1

  11. PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection

    Mar 23, 2026Hyoseok Park, Yeonsang ParkLLM Inference AccelerationLong-Context LLM Inference

    Department of Physics, Chungnam National University, Daejeon 34134, Republic of Korea

  12. Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents

    Mar 5, 2026Natchanon Pollertlam, Witchayut KornsuwannawitPersistent Memory for Language ModelsLong-Term Conversational Memory

    Bricks Technology

  13. Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference

    Jan 19, 2026Zimeng Wu, Donghao Wang, Chaozhe Jin +2LLM Inference AccelerationLong-Context LLM Inference

    School of Computer Science and Engineering, Beihang University, Beijing, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China

  14. Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

    Jan 5, 2026Amirali Ebrahimzadeh, Seyyed M. SaliliLong-Context RetrievalLLM Hallucination Mitigation

    Department of Electrical Engineering & Computer Science, University of Michigan, Ann Arbor, MI 48109, USA · Independent Researcher, Ann Arbor, MI 48106, USA