Long-Context Language Model Inference

Follow this topic Ask about this topic

Long-context language model inference studies how to process lengthy prompts and generate responses without excessive latency, memory use, or loss of relevant information. Approaches include sparse attention and block selection, KV-cache eviction, quantization and offloading, context compression and retrieval, and speculative decoding. Evaluations commonly use long-document question answering, multi-needle retrieval, and reasoning benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, alongside measures like time to first token and decoding throughput. The central tension is that aggressive reduction of computation and memory can miss important evidence, while practical speedups depend on hardware, workload, and whether selection overhead is included.

Latest papers

227 papers35 papers in the last four weeks, up 21% on the four weeks before. 0.2% of all new papers.

Stacked from One: Multi-Scale Self-Injection for Context Window Extension
Mar 5, 2026Wei Han, Pan Zhou, Soujanya Poria +1Long-Context Language Model Inference Long-Context Language Modeling
Singapore University of Technology and Design (SUTD), Singapore · Singapore Management University (SMU), Singapore · Nanyang Technological University (NTU), Singapore +1
MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling
Mar 3, 2026Jinwoong Kim, Sangjin ParkHybrid Transformer-State Space Model Architectures Long-Context Language Modeling
Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea
VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling
Mar 3, 2026Chen GuanzhongLong-Context Language Model Inference LLM Inference Acceleration
School of Computer Science and Technology, Tongji University
Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models
Mar 3, 2026Shubhangi Upasani, Ravi Shanker Raju, Bo Li +5Prompt Compression Long-Context Language Model Inference
SambaNova AI
Multi-Head Low-Rank Attention
Mar 2, 2026Songtao Liu, Hongwu Peng, Zhiwei Zhang +2Long-Context Language Model Inference LLM Inference Acceleration
The Pennsylvania State University · University of Connecticut · Carnegie Mellon University +1
DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models
Feb 25, 2026Xi Ye, Wuwei Zhang, Fangcong Yin +2Long-Context Reasoning in Language Models Language Model Decoding
Princeton Language and Intelligence, Princeton University · Department of Computer Science, New York University
CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference
Feb 24, 2026Chao Fei, Guozhong Li, Chenxi Liu +1Long-Context Language Model Inference LLM Inference Acceleration
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China
KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem
Feb 23, 2026Seongjin Cha, Gyuwan Kim, Dongsu Han +2LLM Inference Acceleration Speculative Decoding
School of Electrical Engineering, KAIST, Daejeon, South Korea · Department of Computer Science, University of California, Santa Barbara, CA, USA
ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
Feb 19, 2026Jianlong Lei, Shashikant IlagerKV-Cache Management Long-Context Language Model Inference
Multiscale Networked Systems (MNS) Group, Informatics Institute University of Amsterdam, The Netherlands
CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
Feb 17, 2026Bradley McDanel, Steven Li, Harshit KhaitanCross-Layer Attention LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
Learning to Evict from Key-Value Cache
Feb 10, 2026Luca Moschella, Laura Manduchi, Ozan SenerKV Caching Long-Context Language Model Inference
Apple
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-Attention Long-Context Language Model Inference
Microsoft Research India.
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models
Nov 7, 2025Chandra Vamsi Krishna Alla, Harish Naidu Gaddam, Manohar Kommi +1Memory-Efficient Optimization LLM Compression
University of Texas at Arlington
CompLLM: Compression for Long Context Q&A
Sep 23, 2025Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran +1LLM Compression Long-Context Language Model Inference
Amazon · Center For Research in Computer Vision, University of Central Florida
Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals
Feb 3, 2025Roy Xie, Junlin Wang, Paul Rosu +4LLM Context Management Long-Context Language Model Inference
Duke · Rice · JHU +1

Mar 5, 2026J/K move · Enter open · S save

Stacked from One: Multi-Scale Self-Injection for Context Window Extension

Wei Han, Pan Zhou, Soujanya Poria, Shuicheng Yan

Singapore University of Technology and Design (SUTD), Singapore · Singapore Management University (SMU), Singapore · Nanyang Technological University (NTU), Singapore+1

Long-Context Language Model Inference Long-Context Language Modeling LLM Context Management Context Compression LLM Inference Acceleration

Open paper AskPDF arXiv

The limited context window of contemporary large language models (LLMs) remains a primary bottleneck for their broader application across diverse domains. Although continual pre-training on long-context data offers a straightforward solution, it incurs prohibitive data acquisition and computational costs. To address this challenge, we propose~\modelname, a novel framework based on multi-grained context compression and query-aware information acquisition. SharedLLM comprises two stacked short-context LLMs: a lower model serving as a compressor and an upper model acting as a decoder. The lower model compresses long inputs into compact, multi-grained representations, which are then forwarded to the upper model for context-aware processing. To maximize efficiency, this information transfer occurs exclusively at the lowest layers, bypassing lengthy forward passes and redundant cross-attention operations. This entire process, wherein the upper and lower models are derived from the same underlying LLM layers, is termed~self-injection. To support this architecture, a specialized tree-based data structure enables the efficient encoding and query-aware retrieval of contextual information. Despite being trained on sequences of only 8K tokens, \modelname~~effectively generalizes to inputs exceeding 128K tokens. Across a comprehensive suite of long-context modeling and understanding benchmarks, \modelname~~achieves performance superior or comparable to strong baselines, striking an optimal balance between efficiency and accuracy. Furthermore, these design choices allow \modelname~to substantially reduce the memory footprint and yield notable inference speedups (

2\times

over streaming and

3\times

over encoder-decoder architectures).