LLM Inference Acceleration

LLM: Large Language Model

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets
Oct 29, 2026Bole Ma, Ayesha Afzal, Jan Eitzinger +2Speculative Decoding Energy-Aware LLM Inference
Erlangen National High Performance Computing Center, Erlangen, Germany
Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
Oct 29, 2026Dmytro Babarytskyi, Oleksandr BabarytskyiRecurrent Language Models Hybrid Attention
Kharkiv National University of Radio Electronics Kharkiv, Ukraine · Military Law Institute Yaroslav Mudryi National Law University Kharkiv, Ukraine
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
Oct 28, 2026Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh +1Speculative Decoding LLM Inference Acceleration
Cohere
TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding
Oct 28, 2026Abhishek Malvankar, Paul Castro, Nick Mitchell +1LLM Inference Acceleration Speculative Decoding
IBM Research.
Causal Attention on CPU: A Cache-Aware and Load-Balanced Scheduling Kernel for Prefill
Oct 28, 2026Kwun Fung Lau, Frost Mitchell, Kanya Mo +1LLM Inference Acceleration Causal Attention
Intel Corp., USA.
Multi-Token Prediction Done Right
Oct 28, 2026Jayden TeohMulti-Token Prediction Language Model Pretraining
MIT
GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
Oct 28, 2026Yipin Guo, Arun M George, Siddharth JoshiKV-Cache Compression Memory-Efficient Inference
University of Notre Dame
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI
Oct 28, 2026NVIDIA, :, Aastha Jhunjhunwala +556LLM Agent Training Mixture-of-Experts Language Models
Merge and Branch: Efficient Tree-Structured Supervised Fine-Tuning of LLMs for Recommendation
Oct 27, 2026Yunqin Zhu, Yao XieLarge Language Model-Based Recommendation LLM Fine-Tuning
Georgia Institute of Technology, Atlanta, GA, USA
FAD-KV: Adaptive Dictionary KV Caching for Efficient Reasoning Decoding
Oct 27, 2026Yuxuan Zhu, David H. Yang, Parisa Shiravani +5KV-Cache Compression LLM Inference Acceleration
Rensselaer Polytechnic Institute · IBM Research
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels
Oct 27, 2026Utkarsh KanwatKV-Cache Compression Memory-Efficient Attention
Independent Researcher
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Recurrent Transformers Efficient Transformer Inference
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
CRIM: Composable and Reusable Item Memory for Efficient Personalized Retrieval
Oct 26, 2026Haolin Li, Haipeng Zhang, Mang Li +5Information Retrieval Memory-Augmented Language Models
Tsinghua University, Beijing, China · Alibaba International Digital Commerce Group, Beijing, China
Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs
Oct 26, 2026Tianqi Chen, Xunchao Li, Linchuan Xie +4Memory-Augmented Language Models LLM Quantization
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent
End-to-End MXFP8 Reinforcement Learning on Blackwell: What Matching Buys, When It Pays Off, and How It Fails
Oct 26, 2026Wengee ZhangRL Fine-Tuning Mixed-Precision Quantization
SPARTA: A Sparse Ternary LLM Accelerator with Adaptive Attention and Unified Datapath for Long-Context Edge Inference
Oct 26, 2026Yanzhen Zhou, Zhengyu Han, Qunyou Liu +3AI Accelerator Inference LLM Inference Acceleration
Zhejiang University · Veloxis AI
The Physical Cost of Context: Quality-Constrained Resource Accounting for Coding Agents
Oct 26, 2026Zhongming Yu, Yitong Ma, Hengjia Yu +4LLM Agent Efficiency LLM Context Management
University of California, San Diego · University of Massachusetts Amherst
T2D: Target-to-Draft Model Pruning & Adaptation for Efficient Speculative Decoding
Oct 25, 2026Shahriar Kabir Nahin, Xiao Ye, Muhao Chen +2Speculative Decoding LLM Inference Acceleration
Bellini College of AI, Cybersecurity, and Computing, University of South Florida · School of Computing and Augmented Intelligence, Arizona State University · Department of Computer Science, University of California, Davis
Distributed LLM Inference via Communication-Efficient Speculative Decoding
Oct 25, 2026Joohyun Shin, Byeonghyeon Park, Keumseo Ryum +1Speculative Decoding LLM Inference Acceleration
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving
Oct 24, 2026YuXuan Lin, KeQiang Wu, Kingsum ChowLLM Serving LLM Inference
Ningbo Global Innovation Center, Zhejiang University, Ningbo, China
One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting
Oct 24, 2026Yuxiao Chen, Changxiao CaiSpeculative Decoding LLM Inference Acceleration
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA
Multiple Importance Sampling for Guided Speculative Inference
Oct 23, 2026Joseph Rowan, Ashish J. KhistiLLM Inference Acceleration Inference-Time Guidance
Department of Electrical and Computer Engineering, University of Toronto
NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding
Oct 23, 2026Nilesh Prasad Pandey, Jangseon Park, Jason Kong +2Speculative Decoding LLM Inference Acceleration
University of California San Diego La Jolla, CA, USA · West Virginia University Morgantown, WV, USA
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
Oct 23, 2026Yudong Zhang, Wei Ding, Kaili Zheng +2Efficient Language Model Inference LLM Inference Acceleration
Tsinghua University · Baidu Inc.
Do Halting Heads Really Save Compute in Looped Language Models?
Oct 23, 2026Tsung-Han Wu, Minsuk Heo, Kenny TanLLM Inference Acceleration Layer Skipping in Language Model Decoding
Apple
BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization
Oct 23, 2026David Jin, Beshr IslamBouli, Tarushii Goel +2LLM Quantization Post-Training Quantization
Massachusetts Institute of Technology
Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference
Oct 22, 2026Tianxiang Xia, Kai Lion, Niao He +3LLM Compression LLM Inference Acceleration
ETH Zürich, Zürich · Max Planck Institute for Intelligent Systems, Tübingen · Vienna University of Economics and Business, Vienna +1
BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing
Oct 22, 2026Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick +1KV-Cache Reuse Memory-Efficient Inference
University of California, San Diego · Together AI
Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation
Oct 22, 2026Bin Hu, Yingbo XiaoMultimodal Large Language Models KV-Cache Compression
Tsinghua University · The Chinese University of Hong Kong, Shenzhen
ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding
Oct 22, 2026Liang He, SiYuan Ma, Qishi Zhan +5Speculative Decoding LLM Inference Acceleration
Tongji University · Nanyang Technological University · Marquette University +4

Oct 29, 2026J/K move · Enter open · S save

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler+1

Erlangen National High Performance Computing Center, Erlangen, Germany

Speculative Decoding Energy-Aware LLM Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.