LLM Inference Acceleration

LLM: Large Language Model

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,239 papers153 papers in the last four weeks, down 27% on the four weeks before. 1.2% of all new papers.

  1. GPUTOK: GPU Accelerated Byte Level BPE Tokenization

    Mar 3, 2026Venu Gopal Kadamba, Kanishkha JaisankarSubword TokenizationLLM Inference Acceleration

  2. CUCo: An Agentic Framework for Compute and Communication Co-design

    Mar 2, 2026Yoga Sri Varshan Varadharajan, Bodun Hu, Saurabh Agarwal +1GPU Kernel OptimizationLLM Inference Acceleration

  3. Multi-Head Low-Rank Attention

    Mar 2, 2026Songtao Liu, Hongwu Peng, Zhiwei Zhang +2Long-Context Language Model InferenceLLM Inference Acceleration

  4. PonderLM-3: Adaptive Token-Wise Pondering with Differentiable Masking

    Mar 2, 2026He Li, Feichen Song, Boyi Zeng +4Adaptive ComputationInference-Time Adaptation of Language Models

  5. AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth

    Mar 2, 2026Shixiang Song, He Li, Zitong Wang +6Recurrent Language ModelsInference-Time Adaptation of Language Models

  6. Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

    Mar 2, 2026Jiebin Zhang, Zhenghan Yu, Liang Wang +8Speculative DecodingLLM Inference Acceleration

  7. Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention

    Mar 1, 2026Mengqi Liao, Lu Wang, Chaoyun Zhang +6LLM Inference AccelerationLLM Inference Scheduling

  8. TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading

    Mar 1, 2026Yudong Pan, Yintao He, Tianhua Han +7Mixture-of-Experts InferenceLLM Inference Acceleration

  9. KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging

    Mar 1, 2026Lianjun Liu, Hongli An, Weiqi Yan +4KV-Cache CompressionLLM Inference Acceleration

  10. RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

    Feb 28, 2026Yingsheng Geng, Yuchong Gao, Weihong Wu +2LLM Inference AccelerationKV-Cache Reuse

  11. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

    Feb 27, 2026Alexander Samarin, Sergei Krutikov, Anton Shevtsov +3Speculative DecodingLLM Inference Acceleration

  12. KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planning

    Feb 27, 2026Zebin Yang, Tong Xie, Baotong Lu +3Agent Memory ManagementMemory-Augmented Language Models

  13. InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

    Feb 26, 2026Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. GrossHardware-Aware QuantizationKV-Cache Quantization

  14. Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching

    Feb 26, 2026Hiroki Matsutani, Naoki Matsuda, Naoto SugiuraLLM Inference AccelerationOn-Device Language Model Inference

  15. Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators

    Feb 26, 2026Zhengyang Su, Isay Katsman, Yueqi Wang +10Constrained DecodingGenerative Retrieval

  16. S2O: Early Stopping for Sparse Attention via Online Permutation

    Feb 26, 2026Yu Zhang, Songwei Liu, Chenqian Yan +4LLM Inference AccelerationSparse Attention

  17. Ruyi2 Technical Report

    Feb 26, 2026Huan Song, Shuyu Tian, Junyi Hao +5LLM Inference AccelerationAdaptive Computation

  18. Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference

    Feb 25, 2026Bo-Wei Chen, Chung-Chi Chen, An-Zi YenCost-Aware InferenceLLM Routing

  19. Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple

    Feb 25, 2026Amirhossein Bozorgkhoo, Igor MolybogSpeculative DecodingLLM Inference Acceleration

  20. Scaling State-Space Models on Multiple GPUs with Tensor Parallelism

    Feb 24, 2026Anurag Dutt, Nimit Shah, Hazem Masarani +1LLM Inference AccelerationTensor Parallelism

  21. MineDraft: A Framework for Batch Parallel Speculative Decoding

    Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative DecodingLLM Inference Acceleration

  22. CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference

    Feb 24, 2026Chao Fei, Guozhong Li, Chenxi Liu +1Long-Context Language Model InferenceLLM Inference Acceleration

  23. KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

    Feb 23, 2026Seongjin Cha, Gyuwan Kim, Dongsu Han +2LLM Inference AccelerationSpeculative Decoding