LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
May 24, 2026Yubo Li, Yidi MiaoKV Caching Long-Context Language Model Inference
Carnegie Mellon University
EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture
May 22, 2026Bowen Duan, Cong Guo, Chiyue Wei +8Vector Quantization LLM Inference Acceleration
Duke University, USA
When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
May 22, 2026Pu Li, Jiawen Qi, Qinyu ChenAI Accelerator Inference On-Device Language Model Inference
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands.
A Simple Plug-in for Improving Eviction-Based KV Cache Compression
May 22, 2026Yuping Lin, Jiayuan Ding, Yue Xing +3KV Caching Memory-Efficient Optimization
Michigan State University · Hippocratic AI
FD-RAG: Federated Dual-System Retrieval-Augmented Generation
May 22, 2026Tianhao Gao, Kai Yang, Yiyang LiRetrieval-Augmented Generation Memory-Augmented Language Models
School of Computer Science and Technology, Tongji University
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1
Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference
May 21, 2026An Xuan NguyenKV Caching Memory-Efficient Optimization
Codepawl Ho Chi Minh City, Vietnam
Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression
May 21, 2026Wei Luo, Yi Huang, Songchen Ma +3KV Caching KV-Cache Eviction
Guangdong Institute of Intelligence Science and Technology · University of Macau · Chinese University of Hong Kong, Shenzhen +1
Skill Weaving: Efficient LLM Improvement via Modular Skillpacks
May 21, 2026Zhuo Li, Guodong Du, Zesheng Shi +5LLM Compression LLM Inference Acceleration
Harbin Institute of Technology, Shenzhen, China · The Hong Kong Polytechnic University · Huawei Technologies Co., Ltd. +1
SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents
May 21, 2026Mehrdad Saberi, Keivan Rezaei, Soheil FeiziMulti-Hop QA Tool-Augmented Language Model Agents
University of Maryland, College Park
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoSelf-Attention KV Caching
Instituto Superior Técnico, Universidade de Lisboa. · ELLIS Unit Lisbon. · INESC-ID. +1
Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
May 20, 2026Reese Levine, Rithik Sharma, Nikhil Jain +5GPU Acceleration LLM Inference
UC Santa Cruz · Microsoft Research
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
May 19, 2026Haiquan Lu, Zigeng Chen, Gongfan Fang +2LLM Quantization Long-Context Language Model Inference
National University of Singapore
Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
May 19, 2026Yuhao Shen, Tianyu Liu, Xinyi Hu +9Speculative Decoding LLM Inference Acceleration
1Zhejiang University · 2Qwen Applications Business Group of Alibaba · University of Science and Technology of China
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
May 19, 2026Yaojie Zhang, Jianuo Huang, Junlong Ke +5Parallel Speculative Decoding Speculative Decoding
EPIC Lab, SJTU · School of Software Engineering, HUST · UESTC +3
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
May 19, 2026Zunhai Su, Rui Yang, Chao Zhang +12KV Caching Rotation-Based Quantization
Tsinghua University · Meituan LongCat Team · The University of Hong Kong +3
MMoA: An AI-Agent framework with recurrence for Memoried Mixure-of-Agent
May 18, 2026Rui ChuLLM Inference Acceleration Multi-Agent Systems
Post-Trained MoE Can Skip Half Experts via Self-Distillation
May 18, 2026Xingtai Lv, Li Sheng, Kaiyan Zhang +12Mixture of Experts Communication-Efficient Mixture-of-Experts
Tsinghua University · Shanghai AI Lab · WeChat AI +1
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1
Context Memorization for Efficient Long Context Generation
May 18, 2026Yasuyuki Okoshi, Hao Mark Chen, Guanxi Lu +3Memory-Augmented Language Models Long-Context Language Model Inference
Institute of Science Tokyo, Japan · Imperial College London, UK
KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5KV-Cache Offloading KV Caching
Hong Kong University of Science and Technology, China · Xi’an Jiaotong University, China
Predictive Prefetching for Retrieval-Augmented Generation
May 18, 2026Wuyang Zhang, Shichao PeiRetrieval-Augmented Generation LLM Inference Acceleration
Department of Computer Science, University of Massachusetts Boston, Boston, United States of America.
CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
May 18, 2026Muyoung Son, Yi Chen, Seungjae Yoo +2Expert Offloading GPU Acceleration
KAIST · Daejeon, Republic of Korea
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
May 18, 2026Zhongzhu Zhou, Donglin Zhuang, Jisen Li +4Rotation-Based Quantization LLM Inference Acceleration
Together AI · University of Sydney · University of Illinois Urbana-Champaign
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7KV-Cache Offloading KV Caching
University of Chicago · Tensormesh Inc. · Samsung Semiconductor +1
Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
May 17, 2026Kangkang Sun, Jianhua Li, Xiuzhen Chen +2Sequential Decision Making Optimal Stopping
Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, China
TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
May 16, 2026Hanzhang Shen, Haoran Wu, Yiren Zhao +1Mixed-Precision Quantization LLM Inference Acceleration
University of Cambridge · Imperial College London
Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference
May 16, 2026Mengtian Yang, Zhekun Zhang, Mingheng Wu +3Language Model Performance Prediction LLM Inference
ByteDance Seed · University of Texas at Austin
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Self-Attention Long-Context Language Model Inference
Nanjing University · Alibaba Group
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1Self-Attention KV-Cache Management
Seoul National University

May 24, 2026J/K move · Enter open · S save

CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

Yubo Li, Yidi Miao

Carnegie Mellon University

KV Caching Long-Context Language Model Inference KV-Cache Eviction LLM Inference Acceleration KV-Cache Pruning

Open paper AskPDF arXiv

Long-horizon LLM inference turns the key--value (KV) cache into the dominant GPU memory consumer and makes per-token attention increasingly expensive. Many common eviction policies use static recency windows or historical attention, leaving unused a signal computed on every decoding step: the model's current uncertainty. We introduce CONF-KV, a KV-cache manager that converts the next-token distribution into a scalar confidence score and uses it to choose the per-step cache budget, retaining more context when the model is uncertain and pruning aggressively when it is confident. Within each budget, tokens are ranked by a composite of accumulated attention mass and recency, while a protected recent window preserves local coherence. We combine the policy with blockwise online-softmax attention, mixed FP16/INT8 storage, and a pyramidal per-layer budget variant. Across four model families and generated lengths up to 4K, CONF-KV stays near the footprint of a fixed 512-token sliding window while remaining within 1.5--2.1 perplexity points of full KV. On Needle-in-a-Haystack up to 32K tokens, CONF-KV reaches 91.4% retrieval accuracy versus 53.8% for sliding windows and 80.6% for H2O; on 75 VisualWebArena tasks it retains 95.3% of full-KV success at 2.8 times lower peak memory.