LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

  1. KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs

    Apr 14, 2026Chuangtao Chen, Grace Li Zhang, Xunzhao Yin +3KV CachingKV-Cache Reuse

    Technical University of Munich Munich, Germany · Technical University of Darmstadt Darmstadt, Germany · Zhejiang University Hangzhou, China +1

  2. OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension

    Apr 14, 2026Zhiyuan Zhang, Yanzhao Li, Zhiqiang Zou +4LLM QuantizationNeural Network Quantization

    Huawei Technology

  3. BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation

    Apr 14, 2026Sihang Guo, Chenlin Zhou, Jiaqi Wang +3Spiking Language ModelsSpiking Neural Networks

    Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China

  4. Latent-Condensed Transformer for Efficient Long Context Modeling

    Apr 14, 2026Zeng You, Yaofo Chen, Qiuwu Chen +5Long-Context Language ModelingLLM Inference Acceleration

    South China University of Technology · Pengcheng Laboratory · AIGCode +2

  5. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

    Apr 14, 2026NVIDIA, :, Aakshita Chandiramani +544Hybrid Transformer-State Space Model ArchitecturesMixture-of-Experts Language Models

  6. SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding

    Apr 14, 2026Shaowen Chen, Zhicheng Liao, Hongwei WangSpeculative DecodingLLM Inference Acceleration

    Zhejiang University, Hangzhou, China.

  7. SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

    Apr 14, 2026Zhuofan Wen, Yang FengSpeculative DecodingLLM Inference Acceleration

    Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

  8. ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval

    Apr 13, 2026David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri +4Memory-Efficient InferenceEfficient Language Model Reasoning

    Rensselaer Polytechnic Institute · IBM Research

  9. LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deployment

    Apr 12, 2026Gregory MagarshakEfficient InferenceLLM Inference Acceleration

  10. IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs

    Apr 12, 2026Yuzhen Mao, Qitong Wang, Martin Ester +1KV-Cache ManagementLLM Inference Acceleration

    Simon Fraser University, BC, Canada · Harvard University, Cambridge, USA

  11. Spike-driven Large Language Model

    Apr 11, 2026Han Xu, Xuerui Qiu, Baiyu Chen +7Spiking Language ModelsSpiking Neural Networks

    Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China +7

  12. Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation

    Apr 11, 2026Tiancheng Hu, Jin Qin, Zheng Wang +10GPU AccelerationLLM Inference Acceleration

    SCS, Peking University, Beijing, China · SKLP, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences, Beijing, China +3

  13. SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding

    Apr 11, 2026Jehyeon Bang, Eunyeong Cho, Ranggi Hwang +2Mixture-of-Experts InferenceLLM Inference Acceleration

    KAIST · UNIST

  14. MEMENTO: Teaching LLMs to Manage Their Own Context

    Apr 10, 2026Vasilis Kontonis, Yuchen Zeng, Shivam Garg +7LLM Context ManagementEfficient Language Model Reasoning

    Microsoft Research

  15. EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

    Apr 10, 2026Shu-Hao Zhang, Le-Tong Huang, Xiang-Sheng Deng +5Quantization-Aware DistillationLLM Quantization

    National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China · Microsoft AI, Beijing 100080, China +1

  16. A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs

    Apr 10, 2026Chen Zhang, Yan Ding, Haotian Wang +3LLM Inference AccelerationLLM Inference Scheduling

    Hunan University Changsha, China

  17. SMART: When is it Actually Worth Expanding a Speculative Tree?

    Apr 9, 2026Lifu Wang, Pan ZhouInference-Time OptimizationGPU Acceleration

    Singapore Management University, Singapore

  18. AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

    Apr 9, 2026Yuxuan Hu, Jianchao Tan, Jiaqi Zhang +7LLM Inference AccelerationLong-Context Language Model Inference

    School of Information, Renmin University of China, Beijing, China · Meituan, Beijing, China

  19. SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs

    Apr 9, 2026Jie Sun, Yu Liu, Lu Han +9Long-Context Language Model InferenceLLM Inference Acceleration

    University of Science and Technology of China · Shanghai Innovation Institute · Nanjing University +5

  20. Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall

    Apr 9, 2026Yasong FanLLM Inference AccelerationAssociative Memory

    Independent Researcher

  21. Efficient and Effective Internal Memory Retrieval for LLM-Based Healthcare Prediction

    Apr 8, 2026Mingchen Li, Jiatan Huang, Zonghai Yao +1Memory-Augmented Language ModelsMedical LLMs

    University of Massachusetts, Amherst · University of Connecticut · University of Massachusetts, Lowell +1

  22. DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

    Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative DecodingLLM Inference Acceleration

    Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1

  23. MARS: Enabling Autoregressive Models Multi-Token Generation

    Apr 8, 2026Ziqi Jin, Lei Wang, Ziwei Luo +1Multi-Token PredictionLLM Inference Acceleration

    Nanyang Technological University · Singapore Management University · Uppsala University

  24. Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models

    Apr 8, 2026Bajian Xiang, Tingwei Guo, Xuan Chen +1Speech Language ModelsAudio Token Compression

    Beike Inc., Beijing, China

  25. MoBiE: Efficient Inference of Mixture of Binary Experts under Post-Training Quantization

    Apr 8, 2026Zhixiong Zhao, Zukang Xu, Zhixuan Chen +1Mixture-of-Experts InferenceMixture-of-Experts Quantization

    Houmo AI

  26. Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference

    Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-AttentionLong-Context Language Model Inference

    School of Computer Science and Technology, Soochow University · Baidu Inc, China

  27. StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference

    Apr 8, 2026Zhirui Chen, Peiyang Liu, Ling ShaoKV-Cache CompressionLong-Context Language Model Inference

    UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, China · National Engineering Research Center for Software Engineering, Peking University

  28. NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies

    Apr 8, 2026Zhiyang Chen, Daliang Xu, Yinyuan Zhang +3Speculative DecodingLLM Inference Acceleration

    Institute for Artificial Intelligence, Peking University, Beijing, China · State Key Laboratory of Networking and Switching Technology; School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China · Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; School of Computer Science, Peking University, Beijing, China

  29. AudioKV: KV Cache Eviction in Efficient Large Audio Language Models

    Apr 8, 2026Yuxuan Wang, Peize He, Xiyan Gui +5KV-Cache EvictionAudio-Language Models

    Xidian University, Xi’an, China · University of Electronic Science and Technology of China, Chengdu, China · Huazhong University of Science and Technology, Wuhan, China +4

  30. Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start

    Apr 8, 2026Xueshen Liu, Yongji Wu, Yuncheng Yao +3LLM ServingLLM Inference Acceleration

    University of Michigan · UC Berkeley · Duke