LLM Inference Acceleration

LLM: Large Language Model

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,239 papers153 papers in the last four weeks, down 27% on the four weeks before. 1.2% of all new papers.

  1. TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

    Mar 11, 2026Toshiaki Koike-Akino, Jing Liu, Ye WangLLM QuantizationNeural Network Quantization

  2. Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation

    Mar 10, 2026Luxi Lin, Zhihang Lin, Zhanpeng Zeng +5Speculative DecodingLLM Inference Acceleration

  3. ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

    Mar 10, 2026Xinyi Hu, Yuhao Shen, Baolin Zhang +6Speculative DecodingLLM Inference Acceleration

  4. FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation

    Mar 10, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM ServingOn-Device Language Model Inference

  5. Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning

    Mar 9, 2026Juming Xiong, Kevin Guo, Congning Ni +7CoT ReasoningEfficient Language Model Reasoning

  6. ConFu: Contemplate the Future for Better Speculative Sampling

    Mar 9, 2026Zongyue Qin, Raghavv Goel, Mukul Gagrani +3Speculative DecodingLLM Inference Acceleration

  7. LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing

    Mar 9, 2026Dongfang Li, Zixuan Liu, Gang Lin +2KV-Cache ManagementLong-Context Language Model Inference

  8. EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

    Mar 9, 2026Chang Han, Yijie Hu, Jingling LiuSpeculative DecodingLLM Inference Acceleration

  9. Deterministic Differentiable Structured Pruning for Large Language Models

    Mar 9, 2026Weiyu Huang, Pengle Zhang, Xiaolu Zhang +3LLM PruningStructured Pruning

  10. Speculating Experts Accelerates Inference for Mixture-of-Experts

    Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts InferenceLLM Inference Acceleration

  11. ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

    Mar 8, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +1Efficient Neural Network InferenceLLM Inference Acceleration

  12. DualSpec: Accelerating Deep Research Agents via Dual-Process Action Speculation

    Mar 8, 2026Shuzhang Zhong, Baotong Lu, Qi Chen +3Deep Research AgentsLLM Inference Acceleration

  13. RedParrot: Accelerating NL-to-DSL for Business Analytics via Query Semantic Caching

    Mar 7, 2026Tong Wang, Yongqin Xu, Jianfeng Zhang +6Enterprise AnalyticsLLM Inference Acceleration

  14. Stem: Rethinking Causal Information Flow in Sparse Attention

    Mar 6, 2026Lin Niu, Xin Luo, Linchuan Xie +4Causal AttentionSparse Attention

  15. A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA

    Mar 6, 2026Neelesh Gupta, Peter Wang, Rajgopal Kannan +1LLM Inference AccelerationFPGA-Based Neural Network Acceleration

  16. InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context

    Mar 5, 2026Xin Teng, Canyu Zhang, Shaoyi Zheng +3Retrieval-Augmented GenerationKV Caching

  17. SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity

    Mar 5, 2026Hanyong Shao, Yingbo Hao, Ting Song +10LLM Inference AccelerationStructured Sparsity

  18. Beyond Linear LLM Invocation: An Efficient and Effective Semantic Filter Paradigm

    Mar 5, 2026Nan Hou, Kangfei Zhao, Jiadong Xie +1LLM Inference AccelerationSemantic Search

  19. Stacked from One: Multi-Scale Self-Injection for Context Window Extension

    Mar 5, 2026Wei Han, Pan Zhou, Soujanya Poria +1Long-Context Language Model InferenceLong-Context Language Modeling

  20. Linear Predictability of Attention Heads in Large Language Models

    Mar 4, 2026Khalid Shaikh, Asmit Kumar Singh, Rebecca Christopher Dsouza +1Attention Head AnalysisLLM Inference Acceleration

  21. Speculative Speculative Decoding

    Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative DecodingLLM Inference Acceleration

  22. An experimental study of KV cache reuse strategies in chunk-level caching systems

    Mar 3, 2026Samuel Cestola, Tianxiang Xia, Zheng Weiyan +2Language Model Inference CachingKV-Cache Reuse