LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,101 papers202 papers in the last four weeks, up 51% on the four weeks before. 1.2% of all new papers.

  1. AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

    Mar 30, 2026Haozhe Qi, Kevin Qu, Mahdi Rad +3Long-Video UnderstandingMultimodal Large Language Models

    Microsoft Spatial AI Lab · EPFL · ETH Zurich

  2. HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

    Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-AttentionLLM Inference Acceleration

    Xiaohongshu

  3. QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference

    Mar 30, 2026Jianxin Yan, Zeheng Qian, Wangze Ni +6LLM Inference AccelerationKV-Cache Management

    Zhejiang University, Hangzhou, China · The University of Sydney, Sydney, Australia · Ant Group, Shanghai, Cina +4

  4. CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference

    Mar 30, 2026Chuxu Song, Zhencan Peng, Jiuqi Wei +1LLM Inference AccelerationLong-Context Language Model Inference

    Department of Computer Science, Rutgers University · Oceanbase, Ant Group.

  5. ITQ3_S: High-Fidelity 3-bit LLM Inference via Interleaved Ternary Quantization with Rotation-Domain Smoothing

    Mar 30, 2026Edward J. YoonLLM QuantizationTernary Quantization

  6. Probabilistic Language Tries: A Unified Framework for Compression, Decision Policies, and Execution Reuse

    Mar 29, 2026Gregory MagarshakLossless Data CompressionLLM Inference Acceleration

  7. KVSculpt: KV Cache Compression as Distillation

    Mar 29, 2026Bo Jiang, Sian JinKV-Cache ManagementKV-Cache Compression

    Temple University

  8. RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication

    Mar 29, 2026Mohsen Dehghankar, Abolfazl AsudehLLM Inference Acceleration

    University of Illinois Chicago

  9. ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference

    Mar 28, 2026Qiuyang Zhang, Kai Zhou, Ding Tang +5KV-Cache OffloadingLLM Inference Acceleration

    Huazhong University of Science and Technology, Wuhan, China · Huawei Technologies, Hefei, China · Research Center for High Efficiency Computing Infrastructure, Zhejiang Lab, Hangzhou, China

  10. TAPS: Task Aware Proposal Distributions for Speculative Sampling

    Mar 27, 2026Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna +2Speculative DecodingLLM Inference Acceleration

    King Abdullah University of Science and Technology (KAUST) · American University of Beirut (AUB)

  11. Two-dimensional early exit optimisation of LLM inference

    Mar 27, 2026Jan Hůla, David Adamczyk, Tomáš Filip +2LLM Inference AccelerationEarly-Exit Neural Networks

    Institute for Research and Applications of Fuzzy Modelling, University of Ostrava, 70200 Ostrava, Czech Republic · Institute of Computer Science, Faculty of Philosophy and Science, Silesian University in Opava, 74601 Opava, Czech Republic

  12. LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling

    Mar 27, 2026Xiaoyu Fan, Huizhi Xie, Wei Zou +1TTS SynthesisZero-Shot TTS

    BRVoice Team, Bairong, Inc., China

  13. TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

    Mar 27, 2026Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4KV-Cache ManagementLong-Context Language Model Inference

    Harbin Institute of Technology, Shenzhen · Guangzhou University

  14. Accelerating PayPal's Commerce Agent with Speculative Decoding: An Empirical Study on EAGLE3 with Fine-Tuned Nemotron Models

    Mar 27, 2026Ally Qin, Jian Wan, Sarat Mudunuri +1LLM InferenceLLM Inference Acceleration

  15. GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

    Mar 26, 2026Selim An, Il hong Suh, Yeseong KimLLM QuantizationLLM Inference Acceleration

    Department of Artificial Intelligence, DGIST, Korea · COGA robotics, Korea · Department of Electrical Engineering, POSTECH, Korea

  16. Large Language Model as Token Compressor and Decompressor

    Mar 26, 2026Wenbing Li, Yiran Wang, Zikai Song +4LLM Context CompressionLLM Compression

  17. Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR

    Mar 25, 2026Haobo Xu, Sirui Chen, Ruizhong Qiu +5RL for Language Model ReasoningReinforcement Learning with Verifiable Rewards

    University of Illinois at Urbana-Champaign · Amazon

  18. Self-Distillation for Multi-Token Prediction

    Mar 25, 2026Guoliang Zhao, Ruobing Xie, An Wang +3Multi-Token PredictionLLM Inference Acceleration

    Large Language Model Department, Tencent

  19. The Diminishing Returns of Early-Exit Decoding in Modern LLMs

    Mar 24, 2026Rui Wei, Rui Du, Hanfei Yu +4Early-Exit Neural NetworksLLM Inference Acceleration

    Stevens Institute of Technology · Northeastern University · Stony Brook University

  20. SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

    Mar 24, 2026Haoyu Huang, Jinfa Huang, Zhongwei Wan +3Efficient Multimodal InferenceLLM Inference Acceleration

    Xiamen University, Xiamen, Fujian, China · University of Rochester, Rochester, NY, USA · The Ohio State University, Columbus, OH, USA

  21. Sparser, Faster, Lighter Transformer Language Models

    Mar 24, 2026Edoardo Cetin, Stefano Peluchetti, Emilio Castillo +3LLM Inference AccelerationSparse Neural Networks

    Sakana AI · NVIDIA

  22. APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs

    Mar 24, 2026Meriem Bouzouad, Yuan-Hao Chang, Jalil BoukhobzaHardware-Aware QuantizationLLM Quantization

    Lab-STICC, CNRS UMR 6285 , ENSTA, Institut Polytechnique de Paris, Brest, France · National Taiwan University, Taipei, Taiwan

  23. EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

    Mar 24, 2026Shiyu Ji, Yixuan Wang, Yijun Liu +2KV-Cache CompressionLLM Inference Acceleration

    Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China

  24. Caterpillar of Thoughts: The Optimal Test-Time Algorithm for Large Language Models

    Mar 24, 2026Amir Azarmehr, Soheil Behnezhad, Alma GhafariInference-Time OptimizationEfficient Language Model Reasoning

    Northeastern University

  25. ROM: Real-time Overthinking Mitigation via Streaming Detection and Intervention

    Mar 23, 2026Xinyan Wang, Xiaogeng Liu, Ming Pei +1Early Stopping for Language Model ReasoningOverthinking in Language Models

    University of Wisconsin–Madison · Johns Hopkins University

  26. PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection

    Mar 23, 2026Hyoseok Park, Yeonsang ParkLLM Inference AccelerationLong-Context LLM Inference

    Department of Physics, Chungnam National University, Daejeon 34134, Republic of Korea

  27. Cross-Family Speculative Decoding for Polish Language Models on AppleSilicon: An Empirical Evaluation of Bielik11B with UAG-Extended MLX-LM

    Mar 22, 2026Krzysztof FonalSpeculative DecodingLLM Inference Acceleration

    Wrocław University of Science and Technology, Poland

  28. TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference

    Mar 22, 2026Jaber Jaber, Osama JaberLLM InferenceLLM Inference Acceleration

    RightNow AI

  29. Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models

    Mar 22, 2026Andrew KirulutaLLM Inference AccelerationLLM Pruning

    UC Berkeley School of Information

  30. Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression

    Mar 21, 2026Ruijie Miao, Zhiming Wang, Wang Li +4KV-Cache ManagementKV-Cache Compression

    Peking University · ByteDance Seed