LLM Inference Acceleration

LLM: Large Language Model

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,239 papers153 papers in the last four weeks, down 27% on the four weeks before. 1.2% of all new papers.

  1. RPU -- A Reasoning Processing Unit

    Feb 20, 2026Matthew Adiletta, Gu-Yeon Wei, David BrooksAI Accelerator InferenceLLM Inference Efficiency

  2. SPQ: An Ensemble Technique for Large Language Model Compression

    Feb 20, 2026Jiamin Yao, Eren GultepeLLM CompressionLLM Quantization

  3. Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding

    Feb 19, 2026Rahul Thomas, Teo Kitanovski, Micah Goldblum +1Speculative DecodingLLM Inference Acceleration

  4. ActTail: Global Activation Sparsity in Large Language Models

    Feb 18, 2026Wenwen Hou, Xinyuan Song, Wenqian Weng +1Activation SparsityLLM Inference Acceleration

  5. Fast KV Compaction via Attention Matching

    Feb 18, 2026Adam Zweiger, Xinghong Fu, Han Guo +1KV-Cache ManagementLLM Context Compression

  6. CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill

    Feb 17, 2026Bradley McDanel, Steven Li, Harshit KhaitanCross-Layer AttentionLLM Inference Acceleration

  7. MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

    Feb 17, 2026Bradley McDanel, Steven Li, Sruthikesh Surineni +1Speculative DecodingLLM Inference Acceleration

  8. Mobility-Aware Cache Framework for Scalable LLM-Based Human Mobility Simulation

    Feb 17, 2026Hua Yan, Heng Tan, Yingxue Zhang +1Human Behavior SimulationLLM Inference Acceleration

  9. QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs

    Feb 11, 2026Kanghyun Noh, Jinheon Choi, Yulhwa KimLLM QuantizationLLM Compression

  10. Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference

    Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3KV Cache SelectionSelf-Attention

  11. ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference

    Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-AttentionKV Caching

  12. OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

    Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts InferenceLLM Inference Acceleration

  13. Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States

    Feb 3, 2026Ximing Dong, Shaowei Wang, Dayi Lin +2Speculative DecodingLLM Inference Acceleration

  14. MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models

    Feb 3, 2026Maximilian Kleinegger, Elvir Crnčević, Dan AlistarhLLM QuantizationLLM Inference Acceleration

  15. Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction

    Feb 2, 2026Yuanzhe Li, Jianing Deng, Jingtong Hu +3LLM Inference AccelerationMixed-Precision Quantization

  16. STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs

    Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4LLM Inference AccelerationLinear Attention

  17. ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure

    Feb 1, 2026Jie Deng, Shining Liang, Jun Li +2Efficient Language Model ReasoningLLM Inference Acceleration

  18. TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification

    Jan 30, 2026Haoyun Jiang, Junqi He, Feng Hong +9Speculative DecodingLLM Inference Acceleration

  19. Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning

    Jan 30, 2026Arvind Mahankali, Kaiyue Wen, Tengyu MaMulti-Agent LLM SystemsCoT Reasoning

  20. LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models

    Jan 29, 2026Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5LLM InferenceEfficient Neural Network Inference

  21. Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference

    Jan 19, 2026Zimeng Wu, Donghao Wang, Chaozhe Jin +2LLM Inference AccelerationLong-Context LLM Inference

  22. ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

    Jan 12, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +3LLM Quantization4-Bit Quantization

  23. JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence

    Jan 8, 2026Shengyin Sun, Yiming Li, Renxi Liu +5Speculative DecodingLLM Inference Acceleration