Long-Context Language Model Inference

Overview

Long-context language model inference studies how to process lengthy prompts and generate responses without excessive latency, memory use, or loss of relevant information. Approaches include sparse attention and block selection, KV-cache eviction, quantization and offloading, context compression and retrieval, and speculative decoding. Evaluations commonly use long-document question answering, multi-needle retrieval, and reasoning benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, alongside measures like time to first token and decoding throughput. The central tension is that aggressive reduction of computation and memory can miss important evidence, while practical speedups depend on hardware, workload, and whether selection overhead is included.

Latest papers

CardsList

246 papers16 papers in the last four weeks, down 64% on the four weeks before. 0.1% of all new papers.

  1. Block Sparse Flash Attention

    Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU AccelerationSelf-Attention

  2. BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models

    Nov 7, 2025Chandra Vamsi Krishna Alla, Harish Naidu Gaddam, Manohar Kommi +1Memory-Efficient OptimizationLLM Compression

  3. Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

    Oct 21, 2025Siyuan Yan, Guo-Qing Jiang, Yuchen Zhang +4Long-Context Language Model InferenceLLM Inference Acceleration

  4. CompLLM: Compression for Long Context Q&A

    Sep 23, 2025Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran +1LLM CompressionLong-Context Language Model Inference

  5. Coherence boosting: When your pretrained language model is not paying enough attention

    Oct 15, 2021Esmeralda S. Whitammer, Zhen Wang, Nebojsa JojicLong-Context Language Model InferenceLong-Context Language Modeling