Long-Context Language Model Inference

Ask about this topic

Overview

Long-context language model inference studies how to process lengthy prompts and generate responses without excessive latency, memory use, or loss of relevant information. Approaches include sparse attention and block selection, KV-cache eviction, quantization and offloading, context compression and retrieval, and speculative decoding. Evaluations commonly use long-document question answering, multi-needle retrieval, and reasoning benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, alongside measures like time to first token and decoding throughput. The central tension is that aggressive reduction of computation and memory can miss important evidence, while practical speedups depend on hardware, workload, and whether selection overhead is included.

Latest papers

CardsList

227 papers35 papers in the last four weeks, up 21% on the four weeks before. 0.2% of all new papers.

  1. MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

    Oct 21, 2026Minsang Kim, Seungjun BaekKV-Cache CompressionLong-Context Language Model Inference

    Korea University · SK Telecom

  2. Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity

    Oct 20, 2026Sahil Joshi, Prithvi Dixit, Aryan Katiyar +5Sparse AttentionLong-Context Language Model Inference

    Department of Computer Science, Rice University, TX, USA. · Department of Electrical Engineering and Computer Sciences, UC Berkeley, CA, USA. · Department of Computer Science, Stanford University, CA, USA. +1

  3. TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

    Oct 19, 2026Liana Patel, Rishabh Ranjan, Ion Stoica +2Token PruningLLM Inference Acceleration

    Stanford University · UC Berkeley

  4. MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers

    Oct 18, 2026Sravanth Kodavanti, Manjunath ArvetiKV-Cache ManagementLong-Context Language Model Inference

    Samsung Research · Georgia Institute of Technology

  5. TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

    Oct 17, 2026Yu Wang, Zhu Wang, Ning Hu +4Memory-Efficient InferenceLong-Context Language Model Inference

    Northwestern Polytechnical University, No. 127 West Youyi Road, Xian, 710129, Shaanxi, China · Xi’an Aeronautics Computing Technique Research Institute, AVIC, No. 15 Jinye 2nd Road, Xi’an, 710065, Shaanxi, China · College of Computer Science and Technology, Harbin Engineering University, No. 145 Nantong Street, Harbin, 150001, Heilongjiang, China

  6. Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

    Oct 16, 2026Zhenxu Tian, Kebin Liu, Zhengwu Yang +8Long-Context Language Model InferenceLLM Inference Acceleration

    Soochow University · Baidu Inc.

  7. FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

    Oct 14, 2026Shuxuan Li, Jialiang Kang, Yanghao Zhou +2Speculative DecodingLong-Context Language Model Inference

    Sun Yat-sen University · Canva Research · Peking University

  8. Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

    Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM InferenceLLM Inference

    Office of CTO Dell Technologies Inc. Canada

  9. Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

    Oct 13, 2026Sarah Pan, Ali BehrouzLong-Context Language Model InferenceMemory-Augmented Language Models

    Massachusetts Institute of Technology · Cornell University

  10. SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

    Oct 13, 2026Jieyun Deng, Keming Hu, Yanghe Liu +1LLM Inference AccelerationOn-Device Language Model Inference

    Southwest Jiaotong University, Chengdu, China · Wuhan Institute of Technology, Wuhan, China · Southwest Petroleum University, Chengdu, China

  11. BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

    Oct 13, 2026Yaoyang Liu, Shuwei Shen, Jincheng Tang +5KV-Cache CompressionLow-Rank Compression

    Peking University · Tencent · BeiHang University

  12. SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba

    Oct 12, 2026Zhetong Zhang, Gayan K KulatillekeState Space Language ModelsLong-Context Language Model Inference

    School of Information Technology and Electrical Engineering The University of Queensland

  13. PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion

    Oct 12, 2026Sejun Park, Haocheng Xia, Yongjoo ParkMemory-Efficient InferenceLong-Context Language Model Inference

    Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA

  14. DP-MIND: Differentially private memory-guided Long context Inference

    Oct 9, 2026Zhihao Liu, Yitian Cheng, Shuang Liu +3Privacy-Preserving Language ModelsLong-Context Language Model Inference

    Zhejiang University · Ant Group · Institute of Science Tokyo +1

  15. Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog

    Oct 8, 2026Hadeel Al-Negheimish, Jasna Ilieva, Yoon KimLLM ReasoningLong-Horizon LLM Reasoning

    King Saud University · Massachusetts Institute of Technology

  16. Evaluating Trajectory Features for Routing Final-Layer Attention

    Oct 7, 2026Yupeng YaoAttention RoutingLLM Routing

  17. SPIN: Shadow Predictive Indexer for Sparse Attention

    Oct 6, 2026Yao Fu, Jiahan Chang, Ritchie Zhao +11LLM Inference AccelerationSparse Attention

  18. Hybrid Latent Attention for Looped Language Models

    Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model InferenceLLM Inference Acceleration

    Virginia Tech · Independent Researcher

  19. ReFold: Training-Free Reversible Inter-Turn Context Folding for Long-Horizon Agents

    Oct 6, 2026Yupeng Su, Jiayi Tian, Zheng Zhang +1LLM Inference EfficiencyLong-Context Language Model Inference

    University of California, Santa Barbara · Intel

  20. OVAL: Output-Aware Local Page Bases for KV Cache Retrieval

    Oct 5, 2026Ashkan Shahbazi, Chayne Thrash, Soheil KolouriLong-Context Language Model InferenceLLM Inference Acceleration

    College of Connected Computing Vanderbilt University Nashville, TN, USA

  21. DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

    Oct 5, 2026Zhe Wang, Jiakai Li, Yujia Sun +2Long-Context Language Model InferenceKV-Cache Eviction

    University of Electronic Science and Technology of China, Chengdu, China · Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province, Chengdu, China

  22. SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

    Oct 4, 2026Jeonghoon Park, Seongwoon Jo, Jongwon Lee +1Long-Context Language Model InferenceSpeculative Decoding

    UNIST

  23. Periscope: Extending Frozen Language Models Beyond Their Context Window

    Oct 2, 2026Mohamed Eltahir, Anas Obayd, Raed Rashid +5Long-Context Language Model InferenceLong-Context Reasoning in Language Models

    King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England

  24. HHR: Hierarchical Hash Retrieval for Efficient LLM Generation

    Oct 1, 2026Lianjun Liu, Tiantian Zheng, You Huang +5Long-Context Language Model InferenceLLM Inference Acceleration

    Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

  25. Role-aware Heuristic Episodic Attention for Conversational LLMs

    Oct 1, 2026Wanyang Hong, Zhaoning Zhang, Yi Chen +5Memory-Augmented Language ModelsLong-Context Language Model Inference

    National Key Laboratory of Parallel and Distributed Computing College of Computer Science and Technology National University of Defense Technology, Changsha, China.

  26. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

    Sep 30, 2026Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1Long-Context Language Model InferenceKV-Cache Management

    Amazon AGI

  27. Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

    Sep 30, 2026Sietse SchelpeLLM ServingLong-Context Language Model Inference

    Corbenic AI

  28. PatchKV: Weight-Space Compensation of KV Cache

    Sep 30, 2026Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi +2Long-Context Language Model InferenceLLM Inference Acceleration

    KAIST

  29. SparseEngine: Sparse-First Inference Engine

    Sep 30, 2026Jitai Hao, Quansheng Gu, Qiang Huang +1LLM InferenceLong-Context Language Model Inference

  30. LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

    Sep 29, 2026Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen +1Long-Context Language Model EvaluationLong-Context Language Model Inference

    University of Alberta · Alberta Machine Intelligence Institute (Amii)