LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

  1. AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth

    Mar 2, 2026Shixiang Song, He Li, Zitong Wang +6Recurrent Language ModelsInference-Time Adaptation of Language Models

    LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Sun Yat-sen University +2

  2. Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

    Mar 2, 2026Jiebin Zhang, Zhenghan Yu, Liang Wang +8Speculative DecodingLLM Inference Acceleration

    National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia

  3. Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification

    Mar 2, 2026Guang Huang, Zeyi WenLLM Inference AccelerationSpeculative Decoding

    HKUST(GZ), Guangzhou, China · HKUST, Hong Kong, China

  4. Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention

    Mar 1, 2026Mengqi Liao, Lu Wang, Chaoyun Zhang +6LLM Inference AccelerationLLM Inference Scheduling

    School of Computer Science and Technology, Beijing Jiaotong University · Microsoft · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence

  5. TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading

    Mar 1, 2026Yudong Pan, Yintao He, Tianhua Han +7Mixture-of-Experts InferenceLLM Inference Acceleration

    SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  6. KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging

    Mar 1, 2026Lianjun Liu, Hongli An, Weiqi Yan +4KV-Cache CompressionLLM Inference Acceleration

    School of Information and Communication Engineering, Hainan University, Haikou, China · School of Cyberspace Security, Hainan University, Haikou, China · MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China +1

  7. RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

    Feb 28, 2026Yingsheng Geng, Yuchong Gao, Weihong Wu +2LLM Inference AccelerationKV-Cache Reuse

    Beijing University of Posts and Telecommunications, Beijing, China · Tsinghua University, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China +1

  8. ICaRus: Identical Cache Reuse for Efficient Multi Model Inference

    Feb 27, 2026Sunghyeon Woo, Jaeeun Kil, Hoseung Kim +9LLM Inference AccelerationMemory-Efficient Inference

    NAVER Cloud

  9. Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study

    Feb 27, 2026Athos GeorgiouLLM Inference EfficiencyLLM Inference

  10. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

    Feb 27, 2026Alexander Samarin, Sergei Krutikov, Anton Shevtsov +3Speculative DecodingLLM Inference Acceleration

    Nebius, Amsterdam, Netherlands

  11. KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planning

    Feb 27, 2026Zebin Yang, Tong Xie, Baotong Lu +3Agent Memory ManagementMemory-Augmented Language Models

    Institute for Artificial Intelligence · School of Integrated Circuits · Microsoft Research +2

  12. InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

    Feb 26, 2026Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. GrossHardware-Aware QuantizationKV-Cache Quantization

    Department of Electrical and Computer Engineering McGill University Montreal, QC, Canada

  13. Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching

    Feb 26, 2026Hiroki Matsutani, Naoki Matsuda, Naoto SugiuraLLM Inference AccelerationOn-Device Language Model Inference

    Dept. of ICS, Keio University, 3-14-1 Hiyoshi, Kohoku-ku, Yokohama, Japan

  14. Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators

    Feb 26, 2026Zhengyang Su, Isay Katsman, Yueqi Wang +10Constrained DecodingGenerative Retrieval

    Google Mountain View, CA, USA · Yale University New Haven, CT, USA

  15. SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning

    Feb 26, 2026Sanjay Kariyappa, G. Edward SuhKV-Cache CompressionLLM Inference Acceleration

    NVIDIA

  16. S2O: Early Stopping for Sparse Attention via Online Permutation

    Feb 26, 2026Yu Zhang, Songwei Liu, Chenqian Yan +4LLM Inference AccelerationSparse Attention

    ByteDance · Xiamen University

  17. Ruyi2 Technical Report

    Feb 26, 2026Huan Song, Shuyu Tian, Junyi Hao +5LLM Inference AccelerationAdaptive Computation

    Institute of Artificial Intelligence (TeleAI), China Telecom

  18. Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference

    Feb 25, 2026Bo-Wei Chen, Chung-Chi Chen, An-Zi YenCost-Aware InferenceLLM Routing

    Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Artificial Intelligence Research Center, AIST, Japan

  19. Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple

    Feb 25, 2026Amirhossein Bozorgkhoo, Igor MolybogSpeculative DecodingLLM Inference Acceleration

    Independent Researcher · University of Hawai’i at Manoa, Honolulu, HI, USA.

  20. Scaling State-Space Models on Multiple GPUs with Tensor Parallelism

    Feb 24, 2026Anurag Dutt, Nimit Shah, Hazem Masarani +1LLM Inference AccelerationTensor Parallelism

    Stony Brook University

  21. MineDraft: A Framework for Batch Parallel Speculative Decoding

    Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative DecodingLLM Inference Acceleration

    College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1

  22. Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning

    Feb 24, 2026Sanket Badhe, Deep ShahEfficient Language Model ReasoningLanguage Model Distillation

    Google Mountain View, California, USA

  23. CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference

    Feb 24, 2026Chao Fei, Guozhong Li, Chenxi Liu +1Long-Context Language Model InferenceLLM Inference Acceleration

    King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China

  24. KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

    Feb 23, 2026Seongjin Cha, Gyuwan Kim, Dongsu Han +2LLM Inference AccelerationSpeculative Decoding

    School of Electrical Engineering, KAIST, Daejeon, South Korea · Department of Computer Science, University of California, Santa Barbara, CA, USA

  25. RPU -- A Reasoning Processing Unit

    Feb 20, 2026Matthew Adiletta, Gu-Yeon Wei, David BrooksAI Accelerator InferenceLLM Inference Efficiency

    Harvard University

  26. SPQ: An Ensemble Technique for Large Language Model Compression

    Feb 20, 2026Jiamin Yao, Eren GultepeLLM CompressionLLM Quantization

    Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA

  27. RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference

    Feb 20, 2026Xiuying Wei, Caglar GulcehreLLM Inference AccelerationSparse Attention

    CLAIRE lab at EPFL, Lausanne, Switzerland.

  28. Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding

    Feb 19, 2026Rahul Thomas, Teo Kitanovski, Micah Goldblum +1Speculative DecodingLLM Inference Acceleration

    Ritual · Columbia University · Vanderbilt University

  29. Greedy Multi-Path Block Verification for Faster Decoding in Speculative Sampling

    Feb 18, 2026Rahul Thomas, David Hidary, Arka PalSpeculative DecodingLLM Inference Acceleration

    Ritual · Columbia University

  30. ActTail: Global Activation Sparsity in Large Language Models

    Feb 18, 2026Wenwen Hou, Xinyuan Song, Wenqian Weng +1Activation SparsityLLM Inference Acceleration

    The Hong Kong University of Science and Technology(Guangzhou) · Emory University · ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems