Long-Context Language Model Inference

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 275% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 166

All topics

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Self-Attention Long-Context Language Model Inference
Nanjing University · Alibaba Group
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Self-Attention Long-Context Language Model Inference
The Hong Kong University of Science and Technology · UC Berkeley
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-Attention KV Caching
Meta FAIR · MICS, CentraleSupélec
SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference
May 13, 2026Anay Chauhan, Gurucharan Marthi Krishna Kumar, Arion Das +4KV Caching Long-Context Language Model Inference
1Synopsys, India · 2McGill University, Canada · 3IIIT Ranchi, India +4
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Long-Context Language Model Inference Speculative Decoding
Northwestern University · GE Aerospace · fal +1
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Softmax Attention Self-Attention
Apple
ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Pruning KV Caching
Shanghai Jiao Tong University
ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
May 9, 2026Yongqi An, Chang Lu, Kuan Zhu +5KV Caching Long-Context Language Model Inference
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China +2
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
May 8, 2026Junkai Zhang, Hang Guo, Luca Benini +1KV Caching Long-Context Language Model Inference
ETH Zurich · Tsinghua University
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6Long-Context Language Model Inference LLM Inference Acceleration
AWS Agentic AI
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Self-Attention Long-Context Language Model Inference
Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
May 8, 2026Tho Mai, Joo-Young KimKV Caching Long-Context Language Model Inference
KAIST Daejeon, South Korea
UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
May 7, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +2Long-Context Language Model Inference LLM Inference Acceleration
1MAIS&NLPR, CASIA · 2UCAS · 3WeChat, Tencent
Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
May 7, 2026Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji +2Self-Attention Long-Context Language Model Inference
Graduate School of Data Science Seoul National University Seoul, Republic of Korea
When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
May 7, 2026Ruijie Zhang, Haozhe Liang, Da Chang +4KV Caching Long-Context Language Model Inference
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
May 5, 2026Jie Ou, Jinyu Guo, Shiyao Guo +5Long-Context Language Model Inference KV-Cache Management
School of Information and Software Engineering, University of Electronic Science and Technology of China
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5GPU Acceleration Self-Attention
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
May 3, 2026Nikita Agrawal, Ruben MayerLLM Serving KV Caching
University of Bayreuth
Context Recycling for Long-Horizon LLM Inference
May 1, 2026Derek ThomasLLM Inference Efficiency Long-Context Language Model Inference
Independent Researcher · contextforge
Budget-Aware Routing for Long Clinical Text
May 1, 2026Khizar Qureshi, Geoffrey Martin, Yifan PengSubmodular Optimization Long-Context Language Model Inference
MIT, Cambridge, MA · Cornell University, Ithaca, NY · Weill Cornell Medicine, New York, NY
Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM Serving KV-Cache Offloading
University of Virginia · Microsoft Research · Georgia Institute of Technology +1
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
Apr 28, 2026Jiaming Yang, Chenwei Tang, Liangli Zhen +1KV Caching Long-Context Language Model Inference
College of Computer Science, Sichuan University, Chengdu, China · Institute of High Performance Computing, A*STAR, Singapore
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Apr 27, 2026Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh +7Long-Context Language Model Inference Long-Context Language Modeling
1AMD
DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
Apr 27, 2026Zahra Dehghanighobadi, Asja FischerLLM Pruning KV Caching
1Ruhr University Bochum · 2UAR Research Center for Trustworthy Data Science and Security
Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Self-Attention Long-Context Language Model Inference
Fudan University
Kwai Summary Attention Technical Report
Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Self-Attention KV Caching
Kuaishou
GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads
Apr 27, 2026Minjie Hua, Ning Wang, Peijun Yang +2LLM Inference Efficiency LLM Serving
China Unicom
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors
Apr 25, 2026Yitian Zhou, Chaoning Zhang, Jiaquan Zhang +6LLM Compression Long-Context Language Model Inference
School of Computer Science and Engineering University of Electronic Science and Technology of China · School of Information and Software Engineering, University of Electronic Science and Technology of China · Department of Computer Science and Engineering, Kyung Hee University +3

May 16, 2026, cs.CLJ/K move · Enter open · S save

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li+5

Nanjing University · Alibaba Group

Self-Attention Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention Multi-Head Attention

Open paper AskPDF arXiv

Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-

p

selection more suitable than fixed top-

k

sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36

\times

prefill speedup at 1M context and about a 2.01

\times

decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.

Momentum

Related topics

Latest papers 166