LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache
Apr 7, 2026Shao Wang, Rui Ren, Lin GuiLLM Serving LLM Inference Acceleration
Shanghai Jiao Tong University, Shanghai, China
STDec: Spatio-Temporal Stability Guided Decoding for dLLMs
Apr 7, 2026Yuzhe Chen, Jiale Cao, Xuyang Liu +3Diffusion Language Model Decoding Diffusion Language Models
1Tianjin University · 2Sichuan University · 3Chongqing University
BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
Apr 7, 2026Abbas Ghaddar, Ivan Kobyzev, Boxing Chen +1Hybrid Attention LLM Inference Acceleration
Huawei Noah’s Ark Lab, Montreal Research Center, Canada
Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion
Apr 7, 2026Zhen Cheng, Hao-Bo Yang, Wan-Yi Huang +1LLM Inference Acceleration
China Merchants Bank Artificial Intelligence Laboratory
See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs
Apr 7, 2026Yicheng Ji, Jun Zhang, Jinpeng Chen +4Speculative Decoding LLM Inference Acceleration
ZJU · BUPT
Residual-Mass Accounting for Partial-KV Decoding
Apr 7, 2026Yasuto Hoshi, Daisuke Miyashita, Jun DeguchiSoftmax Attention Long-Context Language Modeling
Kioxia Corporation
Multi-Drafter Speculative Decoding with Alignment Feedback
Apr 7, 2026Taehyeon Kim, Hojung Jung, Se-Young YunSpeculative Decoding LLM Inference Acceleration
LG AI Research · KAIST AI
Rethinking Model Efficiency: Multi-Agent Inference with Large Models
Apr 6, 2026Sixun Dong, Juhua Hu, Steven Li +2Efficient VLM Inference LLM Inference Acceleration
Independent Researcher · University of Washington · Meta Reality Labs
TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Apr 6, 2026Weian Mao, Xi Lin, Wei Huang +5Long-Context Reasoning in Language Models KV-Cache Compression
MIT · ZJU · NVIDIA
Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache Quantization On-Device Language Model Inference
Clemson University, USA
BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design
Apr 5, 2026Yifu Ding, Xianglong Liu, Shenghao Jin +2Binary Quantization Low-Bit Quantization
School of Computer Science and Engineering, Beihang University, Beijing, China. · Institute of Artificial Intelligence, Beihang University, Beijing, China. · Department of Automation, Tsinghua University, Beijing, China.
Diagonal-Tiled Mixed-Precision Attention for Efficient Low-Bit MXFP Inference
Apr 5, 2026Yifu Ding, Xinhao Zhang, Jinyang GuoLLM Inference Acceleration GPU Kernel Optimization
State Key Laboratory of Complex & Critical Software Environment (SKLCCSE), Beihang University · School of Computer Science and Engineering, Beihang University · Beijing Jiaotong University +1
Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Apr 5, 2026Yongchang Hao, Lili MouSpeculative Decoding LLM Inference Acceleration
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Canada CIFAR AI Chair
Researchers waste 80% of LLM annotation costs by classifying one text at a time
Apr 4, 2026Christian Pipal, Eva-Maria Vogel, Morgan Wack +1LLM-Assisted Annotation Large Language Model-Based Text Classification
Department of Communication and Media Research (IKMZ), University of Zurich
LightThinker++: From Reasoning Compression to Memory Management
Apr 4, 2026Yuqi Zhu, Jintian Zhang, Zhenjie Wan +7Memory Management for LLM Agents Efficient Language Model Reasoning
Zhejiang University · Ant Group
JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency
Apr 3, 2026Aichen Cai, Anmeng Zhang, Anyu Li +66Language Modeling Mixture-of-Experts Language Models
Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
Apr 3, 2026Cornelius Kummer, Lena Jurkschat, Michael Färber +1Prompt Compression LLM Inference Efficiency
ScaDS.AI Dresden/Leipzig, CIDS, TU Dresden
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Apr 3, 2026Qingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1
AXELRAM: Quantize Once, Never Dequantize
Apr 3, 2026Yasushi NishidaAttention Quantization KV-Cache Quantization
Axelidea Inc.
Fast NF4 Dequantization Kernels for Large Language Model Inference
Apr 2, 2026Xiangbo Qi, Chaoyi Jiang, Murali AnnavaramLLM Inference Acceleration GPU Kernel Optimization
University of Southern California Los Angeles, USA
Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding
Apr 2, 2026Tao Jin, Phuong Minh Nguyen, Naoya InoueSpeculative Decoding LLM Inference Acceleration
Japan Advanced Institute of Science and Technology (JAIST)
DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
Apr 2, 2026Wanqian Li, Jintao Peng, Zongfei Jing +7LLM Inference Acceleration Distributed Inference
NVIDIA
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation
Apr 1, 2026Yuhang Li, Donghyun Lee, Ruokai Yin +1Low-Rank Approximation LLM Compression
Yale University · University of Sounthern California
SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding
Apr 1, 2026Yijun Lin, Jinhao Sheng, Qingyue Cai +1Speculative Decoding LLM Inference Acceleration
Center for Applied Statistics and School of Statistics, Renmin University of China · School of Intelligent Medicine, China Medical University · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing
Adaptive Parallel Monte Carlo Tree Search for Efficient Test-time Compute Scaling
Apr 1, 2026Hongbeen Kim, Juhyun Lee, Sanghyeon Lee +2Monte Carlo Tree Search LLM Inference Acceleration
School of Computing, Korea Advanced Institute of Science and Technology, Daejeon, South Korea.
Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
Apr 1, 2026Zhensu Sun, Zhihao Lin, Zhi Chen +4LLM Inference Acceleration Pipeline Parallelism
Singapore Management University · Beihang University
MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation
Mar 31, 2026Jinghan Yao, Sam Adé Jacobs, Walid Krichene +2LLM Inference Acceleration Efficient Attention
The Ohio State University · Microsoft, WA, USA · Anyscale, CA, USA
SISA: A Scale-In Systolic Array for GEMM Acceleration
Mar 31, 2026Luigi Altamura, Alessio Cicero, Mateo Vázquez Maceiras +2LLM Inference Acceleration
Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden
Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
Mar 30, 2026Zifan He, Rui Ma, Yizhou Sun +1LLM Inference LLM Inference Acceleration
Department of Computer Science, University of California, Los Angeles · Microsoft Research Asia

Apr 7, 2026J/K move · Enter open · S save

ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache

Shao Wang, Rui Ren, Lin Gui

Shanghai Jiao Tong University, Shanghai, China

LLM Serving LLM Inference Acceleration Disaggregated LLM Serving KV-Cache Reuse

Open paper AskPDF arXiv

The serving paradigm of large language models (LLMs) is rapidly shifting towards complex multi-agent workflows where specialized agents collaborate over massive shared contexts. While Low-Rank Adaptation (LoRA) enables the efficient co-hosting of these specialized agents on a single base model, it introduces a critical memory footprint bottleneck during serving. Specifically, unique LoRA activations cause Key-Value (KV) cache divergence across agents, rendering traditional prefix caching ineffective for shared contexts. This forces redundant KV cache maintenance, rapidly saturating GPU capacity and degrading throughput. To address this challenge, we introduce ForkKV, a serving system for multi-LoRA agent workflows centered around a novel memory management paradigm in OS: fork with copy-on-write (CoW). By exploiting the structural properties of LoRA, ForkKV physically decouples the KV cache into a massive shared component (analogous to the parent process's memory pages) and lightweight agent-specific components (the child process's pages). To support this mechanism, we propose a DualRadixTree architecture that allows newly forked agents to inherit the massive shared cache and apply CoW semantics for their lightweight unique cache. Furthermore, to guarantee efficient execution, we design ResidualAttention, a specialized kernel that reconstructs the disaggregated KV cache directly within on-chip SRAM. Comprehensive evaluations across diverse language models and practical datasets of different tasks demonstrate that ForkKV achieves up to 3.0x the throughput of state-of-the-art multi-LoRA serving systems with a negligible impact on generation quality.