LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

  1. Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction

    Apr 29, 2026Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani +2LLM Inference AccelerationParallel Decoding

    Amazon.com, Inc. · McGill University · Mila - Quebec AI Institute +1

  2. AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving

    Apr 28, 2026Zhongkai Yu, Haotian Ye, Chenyang Zhou +9Compute-in-MemoryLLM Inference Acceleration

    University of California, San Diego · Columbia University · Yonsei University +2

  3. RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts

    Apr 28, 2026Vyom Sharma, Debajyoti DattaGPU Kernel OptimizationGPU Acceleration

    Hippocratic AI

  4. Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models

    Apr 28, 2026Ajmain Inqiad Alam, Palash Roy, Chanchal K. Roy +2Software EngineeringLLM Compression

    University of Saskatchewan, Canada

  5. Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2

    Apr 28, 2026Chirag ShindeModel CompressionLLM Quantization

    Independent Researcher

  6. AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices

    Apr 28, 2026Ma Zirui, Fan Zhihua, Li Wenxing +4Compute-in-MemoryOn-Device Language Model Inference

    State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  7. Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning

    Apr 27, 2026Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim +3LLM PruningLLM Inference Acceleration

    Neural Superintelligence Lab, MODULABS, Republic of Korea · University of Southern California, United States · Boston University, United States +2

  8. Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices

    Apr 27, 2026Zihuai Xu, Ruofei Hou, Yang Xu +3Retrieval-Augmented GenerationOn-Device Language Model Inference

    Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, 215123

  9. Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding

    Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Self-AttentionLong-Context Language Model Inference

    Fudan University

  10. AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agents

    Apr 27, 2026Hojoon Kim, Yuheng Wu, Thierry TambeLLM PlanningLLM Inference Acceleration

    Seoul National University, Seoul, South Korea · Stanford University, Stanford, California, USA

  11. Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

    Apr 27, 2026Amogh Sheth, Andrew Lin, Biruk Assefa +2LLM PruningSelf-Attention

    Edison Academy Magnet School · Massachusetts Institute of Technology · State University of New York College at Plattsburgh +2

  12. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

    Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language ModelsLong-Context Language Model Inference

  13. Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning

    Apr 26, 2026Zichuan Fu, Xian Wu, Guojing Li +7Efficient Language Model ReasoningLLM Inference Acceleration

    City University of Hong Kong · Tencent Jarvis Lab · Renmin University of China +1

  14. Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference

    Apr 25, 2026Divakar Kumar Yadav, Tian ZhaoLLM InferenceLLM Inference Acceleration

    Department of Computer Science University of Wisconsin-Milwaukee Milwaukee, WI, USA

  15. Large Language Models Decide Early and Explain Later

    Apr 24, 2026Ayan Datta, Zhixue Zhao, Bhuvanesh Verma +3Efficient Language Model ReasoningLLM Inference Acceleration

    IIIT Hyderabad · University of Sheffield, UK · Goethe University, Frankfurt am Main, Germany

  16. ResRank: Unifying Retrieval and Listwise Reranking via End-to-End Joint Training with Residual Passage Compression

    Apr 24, 2026Xiaojie Ke, Shuai Zhang, Liansheng Sun +6Learning to RankLLM Reranking

    Qwen Applications Business Group of Alibaba

  17. LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs

    Apr 23, 2026Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia +3Transformer AttentionLLM Inference Acceleration

    Openchip & Software Technologies, S.L.

  18. SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

    Apr 23, 2026Hongyao Liu, Liuqun Zhai, Junyi Wang +3On-Device Language Model InferenceKV-Cache Offloading

    Department of Computer Science, City University of Hong Kong, Hong Kong SAR.

  19. FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression

    Apr 22, 2026Ye Qiao, Yian Wang, Zhiheng Chen +2LLM CompressionGPU Acceleration

    University of California, Irvine Irvine, California, USA

  20. MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

    Apr 22, 2026Anurita DasMemory-Efficient OptimizationMemory-Efficient Inference

    Genovation Technological Solutions Pvt Ltd

  21. Simplified Sparse Attention via Gist Tokens

    Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxSelf-AttentionLong-Context Language Model Inference

    Stanford University

  22. FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels

    Apr 22, 2026Fei Zuo, Xiaoyan Xi, Quanyi Zeng +2Efficient Neural Network InferenceTernary Quantization

    BA TechWorks · Peking University

  23. Super Apriel: One Checkpoint, Many Speeds

    Apr 21, 2026SLAM Labs, :, Oleksiy Ostapenko +13LLM Inference AccelerationAttention Mechanisms

    SLAM Labs · ServiceNow Research · University of Toronto +4

  24. Micro Language Models Enable Instant Responses

    Apr 21, 2026Wen Cheng, Tuochao Chen, Karim Helwani +3On-Device Language Model InferenceSmall Language Models

    Paul G. Allen School of Computer Science & Engineering, University of Washington · Meta AI

  25. SimDiff: Depth Pruning via Similarity and Difference

    Apr 21, 2026Yuli Chen, Shuhao Zhang, Fanshen Meng +4LLM PruningLLM Inference Acceleration

  26. DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

    Apr 21, 2026Jinyu Guo, Zhihan Zhang, Jiehui Xie +7Long-Context Language Model InferenceLLM Inference Acceleration

    School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China · Bangladesh University of Engineering and Technology +2

  27. SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

    Apr 21, 2026Jinda Jia, Jisen Li, Zhongzhu Zhou +8LLM ServingKV Caching

    Together AI

  28. Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs

    Apr 20, 2026Afsara Benazir, Felix Xiaozhu LinAI Accelerator InferenceEnergy-Efficient ML

    University of Virginia Charlottesville, Virginia, USA

  29. River-LLM: Large Language Model Seamless Exit Based on KV Share

    Apr 20, 2026Yingtao Shen, An ZouDecoder-Only Language ModelsKV-Cache Management

    Shanghai Jiao Tong University Shanghai, China

  30. Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing

    Apr 20, 2026Ziyang LiuLLM Inference AccelerationParallel Decoding

    Independent Researcher