LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Nov 18, 2025Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia +5Mixture-of-Experts Inference LLM Inference Acceleration
Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China
CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization
Nov 7, 2025Ziqian Bi, Yinzhi Wang, Tianyang Wang +6Efficient Language Model Reasoning LLM Inference Acceleration
1Purdue University, USA · 2Baruch College, City University of New York, USA · 3The Ohio State University, USA +6
BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models
Nov 7, 2025Chandra Vamsi Krishna Alla, Harish Naidu Gaddam, Manohar Kommi +1Memory-Efficient Optimization LLM Compression
University of Texas at Arlington
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
Accelerating Attention with Basis Decomposition
Oct 2, 2025Jialin ZhaoSelf-Attention Transformer Attention
Department of Computer Science Tsinghua University
Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression
Oct 2, 2025Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad +3RL for Language Model Reasoning LLM Inference Acceleration
UNC Chapel Hill · The University of Texas at Austin · Microsoft Research
PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
Sep 27, 2025Younes Hourri, Mohammad Mozaffari, Maryam Mehri DehnaviLLM Pruning LLM Compression
Department of Computer Science University of Toronto
PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoning
Sep 17, 2025Jingyuan Fan, Purui Liu, Hengbo Xiao +4Numerical Reasoning in Language Models LLM Inference Acceleration
Peking University · Tsinghua University
FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3GPU Kernel Optimization GPU Acceleration
1. The Hong Kong University of Science and Technology · 2. Carnegie Mellon University
From "Aha Moments" to Controllable Thinking: Toward Meta-Cognitive Reasoning in Large Reasoning Models via Decoupled Reasoning and Control
Aug 6, 2025Rui Ha, Rui Pu, Chaozhuo Li +2RL for Language Model Reasoning LLM Inference Acceleration
Beijing University of Posts and Telecommunications, China · 2Chongqing University of Posts and Telecommunications, China
DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
Jul 25, 2025Jiawen Qi, Jiayu Liu, Chang Gao +2On-Device Language Model Inference LLM Inference Acceleration
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, 2300 RA Leiden, The Netherlands · Department of Microelectronics, Delft University of Technology, 2628 CD Delft, The Netherlands
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
Jun 24, 2025Shuchen Xue, Tianyu Xie, Tianyang Hu +5Masked Diffusion Models Decoder-Only Language Models
University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · School of Mathematical Sciences, Peking University +2
RCStat: A Statistical Framework for using Relative Contextualization in Transformers
Jun 24, 2025Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena +1Attention Mechanisms Transformer Interpretability
Adobe Research, India
Less Is More: Reducing Token Counts Without Compromising Performance
Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference Acceleration Token Efficiency
Graduate School of Data Science, Seoul National University · Department of Computer Science, Seoul National University
FPTQuant: Function-Preserving Transforms for LLM Quantization
Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM Quantization LLM Inference Acceleration
Qualcomm AI Research, Amsterdam, Netherlands
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
May 23, 2025Donghyun Son, Euntae Choi, Sungjoo YooKV Caching Vector Quantization
Seoul National University
Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
May 22, 2025Runpeng Yu, Xinyin Ma, Xinchao WangMultimodal Large Language Models LLM Inference Acceleration
National University of Singapore
Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoning
May 13, 2025Ren ZhuangGradient-Based Attribution LLM Pruning
School of Information Science and Technology, Hangzhou Normal University
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta
Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model Inference LLM Inference
MBZUAI · UESTC · City University of Hong Kong +1
FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference
May 7, 2024Runheng Liu, Xingchen Xiao, Heyan Huang +2Knowledge Augmentation for Language Models LLM Inference Acceleration
School of Computer Science and Technology, Beijing Institute of Technology

Nov 18, 2025J/K move · Enter open · S save

Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia, Xinkai Wang+4

Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China

Mixture-of-Experts Inference LLM Inference Acceleration Expert Offloading Memory-Efficient Inference

Open paper AskPDF arXiv

Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at https://github.com/wwwwwf/specter_asplos.