LLM Inference Acceleration

LLM: Large Language Model

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,239 papers153 papers in the last four weeks, down 27% on the four weeks before. 1.2% of all new papers.

  1. Sandwich Reasoning: An Answer-Reasoning-Answer Approach for Low-Latency Query Correction

    Jan 7, 2026Chen Zhang, Kepu Zhang, Jiatong Zhang +2Efficient Language Model ReasoningQuery Reformulation

  2. Entropy-Aware Token Rejection for Improving Speculative Decoding

    Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative DecodingLLM Inference Acceleration

  3. Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference

    Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-AttentionLong-Context Language Model Inference

  4. Batched Early-Exit LLM Inference Done Right with DREX

    Dec 17, 2025Xuting Liu, Daniel Alexander, Jiaheng Lu +3LLM Inference AccelerationEfficient Language Model Inference

  5. xGR: Efficient Generative Recommendation Serving at Scale

    Dec 12, 2025Qingxiao Sun, Tongxuan Liu, Shen Zhang +13LLM ServingLLM Inference Acceleration

  6. Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference

    Dec 12, 2025Siddharth Jayashankar, Joshua Kim, Michael B. Sullivan +2Homomorphic EncryptionGPU Kernel Optimization

  7. Block Sparse Flash Attention

    Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU AccelerationSelf-Attention

  8. Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement

    Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware InferenceSmall Language Models

  9. Data-Free Pruning of Self-Attention Layers in LLMs

    Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Inference AccelerationLLM Compression

  10. ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models

    Nov 24, 2025Long Lian, Sida Wang, Felix Juefei-Xu +7Parallel Reasoning in Language ModelsRL for Language Model Reasoning

  11. DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures

    Nov 19, 2025Peiming Yang, Sankeerth Durvasula, Ivan Fernandez +4Compute-in-MemoryLLM Inference Acceleration

  12. Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference

    Nov 19, 2025Kexin Chu, Dawei Xiang, Zixu Shen +3Mixed-Precision QuantizationGPU Acceleration

  13. Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

    Nov 18, 2025Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia +5Mixture-of-Experts InferenceLLM Inference Acceleration

  14. CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization

    Nov 7, 2025Ziqian Bi, Yinzhi Wang, Tianyang Wang +6Efficient Language Model ReasoningLLM Inference Acceleration

  15. BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models

    Nov 7, 2025Chandra Vamsi Krishna Alla, Harish Naidu Gaddam, Manohar Kommi +1Memory-Efficient OptimizationLLM Compression

  16. Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference

    Oct 21, 2025Siyuan Yan, Guo-Qing Jiang, Yuchen Zhang +4Long-Context Language Model InferenceLLM Inference Acceleration

  17. NOSA: Native and Offloadable Sparse Attention

    Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache OffloadingGPU Acceleration

  18. Accelerating Attention with Basis Decomposition

    Oct 2, 2025Jialin ZhaoSelf-AttentionTransformer Attention

  19. Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression

    Oct 2, 2025Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad +3RL for Language Model ReasoningLLM Inference Acceleration

  20. PATCH: Learnable Tile-level Hybrid Sparsity for LLMs

    Sep 27, 2025Younes Hourri, Mohammad Mozaffari, Maryam Mehri DehnaviLLM PruningLLM Compression

  21. FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

    Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3GPU Kernel OptimizationGPU Acceleration

  22. PrismServe: Efficient LLM Serving on Heterogeneous Accelerator Clusters

    Aug 9, 2025Jinhao Zhang, Zicheng Yan, Boyang Zhang +5LLM ServingLLM Inference Acceleration

  23. From "Aha Moments" to Controllable Thinking: Toward Meta-Cognitive Reasoning in Large Reasoning Models via Decoupled Reasoning and Control

    Aug 6, 2025Rui Ha, Rui Pu, Chaozhuo Li +2RL for Language Model ReasoningLLM Inference Acceleration

  24. DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

    Jul 25, 2025Jiawen Qi, Jiayu Liu, Chang Gao +2On-Device Language Model InferenceLLM Inference Acceleration

  25. Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

    Jun 24, 2025Shuchen Xue, Tianyu Xie, Tianyang Hu +5Masked Diffusion ModelsDecoder-Only Language Models

  26. RCStat: A Statistical Framework for using Relative Contextualization in Transformers

    Jun 24, 2025Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena +1Attention MechanismsTransformer Interpretability

  27. Less Is More: Reducing Token Counts Without Compromising Performance

    Jun 18, 2025Gyeongje Cho, Yeonkyoung So, Sangmin Lee +1LLM Inference AccelerationToken Efficiency

  28. FPTQuant: Function-Preserving Transforms for LLM Quantization

    Jun 5, 2025Boris van Breugel, Yelysei Bondarenko, Paul Whatmough +1LLM QuantizationLLM Inference Acceleration

  29. Highly Efficient and Effective LLMs with Multi-Boolean Architectures

    May 28, 2025Ba-Hien Tran, Van Minh NguyenLLM QuantizationBinary Neural Networks