LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning
Mar 9, 2026Juming Xiong, Kevin Guo, Congning Ni +7CoT Reasoning Efficient Language Model Reasoning
1Vanderbilt University · 2Vanderbilt University Medical Center · 3Intuit AI Research
ConFu: Contemplate the Future for Better Speculative Sampling
Mar 9, 2026Zongyue Qin, Raghavv Goel, Mukul Gagrani +3Speculative Decoding LLM Inference Acceleration
University of California Los Angeles, United States · Qualcomm AI Research, United States.
LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing
Mar 9, 2026Dongfang Li, Zixuan Liu, Gang Lin +2KV-Cache Management Long-Context Language Model Inference
Harbin Institute of Technology · University of Electronic Science and Technology of China
EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs
Mar 9, 2026Chang Han, Yijie Hu, Jingling LiuSpeculative Decoding LLM Inference Acceleration
College of Computer Science, Chongqing University
Deterministic Differentiable Structured Pruning for Large Language Models
Mar 9, 2026Weiyu Huang, Pengle Zhang, Xiaolu Zhang +3LLM Pruning Structured Pruning
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Ant Group, Beijing, China
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI
ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs
Mar 8, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +1Efficient Neural Network Inference LLM Inference Acceleration
Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China
DualSpec: Accelerating Deep Research Agents via Dual-Process Action Speculation
Mar 8, 2026Shuzhang Zhong, Baotong Lu, Qi Chen +3Deep Research Agents LLM Inference Acceleration
Peking University, Beijing, China · Microsoft Research · Microsoft
RedParrot: Accelerating NL-to-DSL for Business Analytics via Query Semantic Caching
Mar 7, 2026Tong Wang, Yongqin Xu, Jianfeng Zhang +6Enterprise Analytics LLM Inference Acceleration
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Xiaohongshu
Stem: Rethinking Causal Information Flow in Sparse Attention
Mar 6, 2026Lin Niu, Xin Luo, Linchuan Xie +4Causal Attention Sparse Attention
Tencent · School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230026, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, 215123, China +5
FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling
Mar 6, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +3Long-Context Language Model Inference LLM Inference Acceleration
MAIS&NLPR, CASIA · UCAS · WeChat, Tencent
A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA
Mar 6, 2026Neelesh Gupta, Peter Wang, Rajgopal Kannan +1LLM Inference Acceleration FPGA-Based Neural Network Acceleration
University of Southern California, USA · DEVCOM Army Research Office, USA
Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference
Mar 6, 2026Ramchand KumaresanNeural Network Hardware Acceleration On-Device Training
MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens
Mar 6, 2026Yu Chen, Runkai Chen, Sheng Yi +9Memory-Augmented Language Models Long-Context Language Modeling
Evermind · Shanda Group · Peking University
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Mar 5, 2026Xin Teng, Canyu Zhang, Shaoyi Zheng +3Retrieval-Augmented Generation KV Caching
New York University · New York University Shanghai · Duke University +1
Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs
Mar 5, 2026Jihao Xin, Tian Lyu, Qilong Pan +2Model Compression LLM Compression
KAUST · HUMAIN
SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity
Mar 5, 2026Hanyong Shao, Yingbo Hao, Ting Song +10LLM Inference Acceleration Structured Sparsity
Peking University · Microsoft Research · South China University of Technology +2
Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
Mar 5, 2026Ofir Ben ShohamLLM Inference Acceleration Speculative Decoding
Intuit
Beyond Linear LLM Invocation: An Efficient and Effective Semantic Filter Paradigm
Mar 5, 2026Nan Hou, Kangfei Zhao, Jiadong Xie +1LLM Inference Acceleration Semantic Search
The Chinese University of Hong Kong · Beijing Institute of Technology · HKUST (Guangzhou)
Stacked from One: Multi-Scale Self-Injection for Context Window Extension
Mar 5, 2026Wei Han, Pan Zhou, Soujanya Poria +1Long-Context Language Model Inference Long-Context Language Modeling
Singapore University of Technology and Design (SUTD), Singapore · Singapore Management University (SMU), Singapore · Nanyang Technological University (NTU), Singapore +1
Linear Predictability of Attention Heads in Large Language Models
Mar 4, 2026Khalid Shaikh, Asmit Kumar Singh, Rebecca Christopher Dsouza +1Attention Head Analysis LLM Inference Acceleration
Georgia Institute of Technology · Santa Clara University
Speculative Speculative Decoding
Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative Decoding LLM Inference Acceleration
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
An experimental study of KV cache reuse strategies in chunk-level caching systems
Mar 3, 2026Samuel Cestola, Tianxiang Xia, Zheng Weiyan +2KV-Cache Reuse LLM Inference Acceleration
Huawei, Switzerland. · Huawei, China.
VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling
Mar 3, 2026Chen GuanzhongLong-Context Language Model Inference LLM Inference Acceleration
School of Computer Science and Technology, Tongji University
Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models
Mar 3, 2026Shubhangi Upasani, Ravi Shanker Raju, Bo Li +5Prompt Compression Long-Context Language Model Inference
SambaNova AI
SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
Mar 3, 2026Sunghyeon Woo, Ahreum Seo, Jaegwang Lee +6Disaggregated LLM Serving LLM Inference Acceleration
NAVER Cloud
GPUTOK: GPU Accelerated Byte Level BPE Tokenization
Mar 3, 2026Venu Gopal Kadamba, Kanishkha JaisankarSubword Tokenization LLM Inference Acceleration
M.S. in Data Science, New York University
CUCo: An Agentic Framework for Compute and Communication Co-design
Mar 2, 2026Yoga Sri Varshan Varadharajan, Bodun Hu, Saurabh Agarwal +1GPU Kernel Optimization LLM Inference Acceleration
UT Austin
Multi-Head Low-Rank Attention
Mar 2, 2026Songtao Liu, Hongwu Peng, Zhiwei Zhang +2Long-Context Language Model Inference LLM Inference Acceleration
The Pennsylvania State University · University of Connecticut · Carnegie Mellon University +1
PonderLM-3: Adaptive Token-Wise Pondering with Differentiable Masking
Mar 2, 2026He Li, Feichen Song, Boyi Zeng +4Adaptive Computation Inference-Time Adaptation of Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai AI Laboratory · Shanghai Jiao Tong University +1

Mar 9, 2026J/K move · Enter open · S save

Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning

Juming Xiong, Kevin Guo, Congning Ni, Weixin Liu+6

1Vanderbilt University · 2Vanderbilt University Medical Center · 3Intuit AI Research

CoT Reasoning Efficient Language Model Reasoning LLM Inference Acceleration Adaptive Sampling LLM Uncertainty Estimation LLM Reasoning+1

Open paper AskPDF arXiv

Large language models (LLMs) can achieve strong reasoning performance through chain-of-thought (CoT) reasoning, yet they often generate unnecessarily long reasoning paths that incur high inference cost. Self-consistency-based approaches push accuracy higher still, but they require sampling and aggregating multiple reasoning trajectories, leading to substantial computational overhead. In this paper, we introduce a confidence-aware selective sampling framework that, at inference time, analyzes a single reasoning trajectory to adaptively determine whether to rely on that trajectory alone or trigger multi-path sampling. The framework uses trajectory-level numeric features and sentence-level linguistic features extracted from reasoning states to guide selective multi-path reasoning. We train it on MedQA and evaluate it in-domain on MedQA and under calibration-only transfer on MathQA, MedMCQA, and MMLU, without further fine-tuning. Experimental results show that the proposed framework maintains comparable performance to full and efficient multi-path reasoning baselines, with accuracy changes of

-0.41 \pm 0.58

and

-0.31 \pm 0.58

percentage points, respectively, while reducing token usage by

71.7 \pm 5.0%

and

36.6 \pm 9.1%

. These findings demonstrate that reasoning trajectories contain rich signals for uncertainty estimation, enabling a simple, transferable mechanism to balance accuracy and efficiency in LLM reasoning.