Long-Context Language Model Inference

Follow this topic Ask about this topic

Long-context language model inference studies how to process lengthy prompts and generate responses without excessive latency, memory use, or loss of relevant information. Approaches include sparse attention and block selection, KV-cache eviction, quantization and offloading, context compression and retrieval, and speculative decoding. Evaluations commonly use long-document question answering, multi-needle retrieval, and reasoning benchmarks such as LongBench, RULER, and Needle-in-a-Haystack, alongside measures like time to first token and decoding throughput. The central tension is that aggressive reduction of computation and memory can miss important evidence, while practical speedups depend on hardware, workload, and whether selection overhead is included.

Latest papers

227 papers35 papers in the last four weeks, up 21% on the four weeks before. 0.2% of all new papers.

SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
Apr 24, 2026Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16Long-Context Language Model Inference Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences · The Hong Kong Polytechnic University · Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model +1
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
Apr 22, 2026Enshuai Zhou, Yifan Hao, Chao Wang +7KV Caching Long-Context Language Model Inference
University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Simplified Sparse Attention via Gist Tokens
Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxSelf-Attention Long-Context Language Model Inference
Stanford University
Absorber LLM: Harnessing Causal Synchronization for Test-Time Training
Apr 22, 2026Zhixin Zhang, Shabo Zhang, Chengcan Wu +2Memory-Augmented Language Models Long-Context Language Model Inference
1Peking University.
DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing
Apr 21, 2026Jinyu Guo, Zhihan Zhang, Jiehui Xie +7Long-Context Language Model Inference LLM Inference Acceleration
School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China · Bangladesh University of Engineering and Technology +2
AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantization
Apr 20, 2026Kosuke Matsushima, Yasuyuki Okoshi, Masato Motomura +1Compute-in-Memory Long-Context Language Model Inference
Institute of Science Tokyo†
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Self-Attention Long-Context Language Model Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University
Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
Apr 18, 2026Sai VegasenaAI Accelerator Inference Self-Attention
Ensue
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
ART: Attention Run-time Termination for Efficient Large Language Model Decoding
Apr 15, 2026Chen Qiu, Guozhong Li, Cristian McGee +2Long-Context Language Model Inference LLM Inference Acceleration
King Abdullah University of Science and Technology · University of Central Florida
IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
Apr 12, 2026Yuzhen Mao, Qitong Wang, Martin Ester +1KV-Cache Management LLM Inference Acceleration
Simon Fraser University, BC, Canada · Harvard University, Cambridge, USA
AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
Apr 9, 2026Yuxuan Hu, Jianchao Tan, Jiaqi Zhang +7LLM Inference Acceleration Long-Context Language Model Inference
School of Information, Renmin University of China, Beijing, China · Meituan, Beijing, China
SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs
Apr 9, 2026Jie Sun, Yu Liu, Lu Han +9Long-Context Language Model Inference LLM Inference Acceleration
University of Science and Technology of China · Shanghai Innovation Institute · Nanjing University +5
Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-Attention Long-Context Language Model Inference
School of Computer Science and Technology, Soochow University · Baidu Inc, China
StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference
Apr 8, 2026Zhirui Chen, Peiyang Liu, Ling ShaoKV-Cache Compression Long-Context Language Model Inference
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, China · National Engineering Research Center for Software Engineering, Peking University
In-Place Test-Time Training
Apr 7, 2026Guhao Feng, Shengjie Luo, Kai Hua +4Test-Time Training Continual Learning for LLMs
ByteDance Seed · Peking University
MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation
Mar 31, 2026Jinghan Yao, Sam Adé Jacobs, Walid Krichene +2LLM Inference Acceleration Efficient Attention
The Ohio State University · Microsoft, WA, USA · Anyscale, CA, USA
CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference
Mar 30, 2026Chuxu Song, Zhencan Peng, Jiuqi Wei +1LLM Inference Acceleration Long-Context Language Model Inference
Department of Computer Science, Rutgers University · Oceanbase, Ant Group.
ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
Mar 28, 2026Qiuyang Zhang, Kai Zhou, Ding Tang +5KV-Cache Offloading LLM Inference Acceleration
Huazhong University of Science and Technology, Wuhan, China · Huawei Technologies, Hefei, China · Research Center for High Efficiency Computing Infrastructure, Zhejiang Lab, Hangzhou, China
TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
Mar 27, 2026Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology, Shenzhen · Guangzhou University
The $\mathbf{Y}$ -Combinator for LLMs: Solving Long-Context Rot with $λ$ -Calculus
Mar 20, 2026Amartya Roy, Rasul Tutunov, Xiaotong Ji +2Long-Context Reasoning in Language Models Long-Context Language Model Inference
The School of Interdisciplinary Research, Indian Institute of Technology (IIT) Delhi, Robert Bosch GmbH, India · Huawei Noah’s Ark Lab · Huawei Noah’s Ark +1
BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection
Mar 20, 2026Zhengpei Hu, Kai Li, Dapeng Fu +4LLM Context Compression Prompt Compression
School of Computer Technology and Application, Qinghai University · Tsinghua University · Ant Group Security and Intelligence Laboratory (SIL)
MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Mar 16, 2026Hanxian Huang, Igor Fedorov, Andrey Gromov +14On-Device Language Model Inference LLM Inference Acceleration
Meta AI
Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability
Mar 12, 2026Xingyu Xie, Zhaochen Yu, Yue Liao +3LLM Inference Acceleration Long-Context Language Model Inference
National University of Singapore · ByteDance
Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries
Mar 12, 2026Zhenxu Tian, Yi Su, Juntao Li +1KV-Cache Compression KV-Cache Eviction
School of Computer Science and Technology, Soochow University, China
LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Mar 11, 2026Jinwoo Ahn, Ingyu Seong, Akhil Kedia +4KV-Cache Eviction Long-Context Language Model Inference
Samsung Research
The $qs$ Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference
Mar 9, 2026Vignesh Adhinarayanan, Nuwan JayasenaMixture-of-Experts Inference Long-Context Language Model Inference
AMD Research and Advanced Development (RAD).
LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing
Mar 9, 2026Dongfang Li, Zixuan Liu, Gang Lin +2KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology · University of Electronic Science and Technology of China
Recursive Language Models Meet Uncertainty: The Surprising Effectiveness of Self-Reflective Program Search for Long Context
Mar 7, 2026Keivan Alizadeh, Parshin Shojaee, Minsik Cho +1Long-Context Language Model Inference Recursive Language Models
Apple
FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling
Mar 6, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +3Long-Context Language Model Inference LLM Inference Acceleration
MAIS&NLPR, CASIA · UCAS · WeChat, Tencent

Apr 24, 2026J/K move · Enter open · S save

SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference

Yuqi Pan, Jinghao Zhuang, Yupeng Feng, Fangzhi Zhong+15

Institute of Automation, Chinese Academy of Sciences · The Hong Kong Polytechnic University · Beijing Key Laboratory of Brain-Inspired General Intelligence Large Model+1

Long-Context Language Model Inference Spiking Neural Networks Sparse Attention Neuromorphic Computing

Open paper AskPDF arXiv

Scaling context length is reshaping large-model development, yet full-attention Transformers suffer from prohibitive computation and inference bottlenecks at long sequences. A key challenge is to design foundation models that maintain performance and long-context efficiency with minimal training overhead. We introduce SpikingBrain2.0 (SpB2.0), a 5B model that advances both architecture and training efficiency of its predecessor. Our contributions are two-fold. (1) Architectural Innovation: We propose Dual-Space Sparse Attention (DSSA), an inter-layer hybrid of Sparse Softmax Attention (MoBA) and Sparse Linear Attention (SSE), achieving an improved performance-efficiency trade-off for long-context modeling. SpB2.0 further supports dual quantization paths: INT8-Spiking coding enables sparse event-driven computation, while FP8 coding accelerates inference on modern GPUs. (2) Enhanced Training Strategy: We develop an optimized Transformer-to-Hybrid (T2H) pipeline with dual conversion paths for LLMs and VLMs using curated open-source data. Empirically, SpB2.0-5B and SpB2.0-VL-5B recover most of the base Transformer (Qwen3-4B) capability with under 7k A100 GPU hours. SpB2.0 achieves a 10.13x TTFT speedup at 4M context and supports over 10M tokens on 8 A100 GPUs under vLLM, where full-attention models exceed memory limits. It also demonstrates strong cross-platform compatibility, enabling FP8 GPU inference (2.52x speedup at 250k) and efficient neuromorphic execution (64.31% sparsity, with 70.6% and 46.5% area and power reduction at 500MHz). Overall, SpikingBrain2.0 provides a practical pathway for lightweight, multimodal, spiking foundation models, highlighting the potential of combining brain-inspired mechanisms with efficient architectures for resource-constrained and edge scenarios.