Long-Context Language Model Inference

Overview

Long-context language model inference studies how to process lengthy prompts and generate responses without excessive latency, memory use, or loss of relevant information. Approaches include sparse attention and block selection, KV-cache eviction, quantization and offloading, context compression and retrieval, and speculative decoding. Evaluations commonly use long-document question answering, multi-needle retrieval, and reasoning benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, alongside measures like time to first token and decoding throughput. The central tension is that aggressive reduction of computation and memory can miss important evidence, while practical speedups depend on hardware, workload, and whether selection overhead is included.

Latest papers

CardsList

246 papers16 papers in the last four weeks, down 64% on the four weeks before. 0.1% of all new papers.

  1. Measuring and Charging for Retained Context in Long-Context LLM Services

    Nov 18, 2026Yi Lu, Qianshi Lin, Yumeng Yang +1LLM ServingLLM Context Management

  2. Evaluating Large Language Models for Scientific Paper Search: Retrieval Success and Inference Latency on Mac Studio and NVIDIA A100

    Nov 13, 2026Frederic Andriantseheno Rakoto, Yannis Bendi-Ouis, Xavier HinautLLM EvaluationScientific Document Retrieval

  3. Verify Before You Fetch: Reliable KV Cache Compression for LLM Inference

    Nov 8, 2026Devanshu Jha, Gaurav Kumar, Agniva Chowdhury +2KV-Cache CompressionLong-Context Language Model Inference

  4. Retrofitting Unbounded Context onto Stock Language Models

    Nov 6, 2026Mark OskinLong-Context Language Model InferenceLanguage Modeling

  5. Query-Guided RoPE Modulation for Lost-in-the-Middle Mitigation in Long-Context LLMs

    Nov 6, 2026Peiyu Zhang, Heng Ping, Shinibali Bhattacharyya +6Lost-in-the-Middle EffectLong-Context Language Model Inference

  6. FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving

    Nov 5, 2026Xuan Wang, Zhuanhao Wu, Tingzhu Fang +17KV-Cache OffloadingLLM Serving

  7. Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

    Nov 4, 2026João Eduardo Batista, Emil Vatai, Mohamed WahibLLM Inference AccelerationLong-Context Language Model Inference

  8. EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies

    Nov 1, 2026Yuhao Liu, Yuehao Li, Linghe Zhang +4KV Cache SelectionLong-Context Language Model Inference

  9. Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity

    Oct 20, 2026Sahil Joshi, Prithvi Dixit, Aryan Katiyar +5Sparse AttentionLong-Context Language Model Inference

  10. TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

    Oct 19, 2026Liana Patel, Rishabh Ranjan, Ion Stoica +2Token PruningLLM Inference Acceleration

  11. MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers

    Oct 18, 2026Sravanth Kodavanti, Manjunath ArvetiLanguage Model Inference CachingKV-Cache Management

  12. TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

    Oct 17, 2026Yu Wang, Zhu Wang, Ning Hu +4Memory-Efficient InferenceLong-Context Language Model Inference

  13. FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

    Oct 14, 2026Shuxuan Li, Jialiang Kang, Yanghao Zhou +2Speculative DecodingLong-Context Language Model Inference

  14. Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

    Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM InferenceLLM Inference

  15. SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

    Oct 13, 2026Jieyun Deng, Keming Hu, Yanghe Liu +1LLM Inference AccelerationOn-Device Language Model Inference

  16. BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

    Oct 13, 2026Yaoyang Liu, Shuwei Shen, Jincheng Tang +5KV-Cache CompressionLow-Rank Compression

  17. SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba

    Oct 12, 2026Zhetong Zhang, Gayan K KulatillekeState Space Language ModelsLong-Context Language Model Inference

  18. PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion

    Oct 12, 2026Sejun Park, Haocheng Xia, Yongjoo ParkMemory-Efficient InferenceLong-Context Language Model Inference

  19. DP-MIND: Differentially private memory-guided Long context Inference

    Oct 9, 2026Zhihao Liu, Yitian Cheng, Shuang Liu +3Privacy-Preserving Language ModelsLong-Context Language Model Inference

  20. Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog

    Oct 8, 2026Hadeel Al-Negheimish, Jasna Ilieva, Yoon KimDeductive Reasoning in Language ModelsLong-Horizon LLM Reasoning