LLM Inference Acceleration

LLM: Large Language Model

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,239 papers153 papers in the last four weeks, down 27% on the four weeks before. 1.2% of all new papers.

  1. TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

    Nov 19, 2026Jinbin Luo, Xiaozhe Yao, Shang Yang +3Language Model Inference CachingLLM Serving

  2. HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

    Nov 19, 2026Ambroise Borbely, Mathieu Salzmann, Olivier VerscheureSparse AttentionMemory-Efficient Attention

  3. FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models

    Nov 19, 2026Zekun Wang, Yizhuo WangLanguage Model Inference CachingPrefix Caching

  4. DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention

    Nov 19, 2026Emre Uğur, Jiawen Qi, Qinyu ChenSparse AttentionLLM Inference Acceleration

  5. RoCoKV: Accelerating LLM Inference Serving through Uncertainty and Redundancy Aware KV Cache Compression

    Nov 19, 2026Payman Behnam, Elaheh Sobhani, Saad Amin +6KV-Cache CompressionLLM Inference Acceleration

  6. Reducing Inference Cost of Coding Agents with Timeout Cascades

    Nov 18, 2026Boxi Zhou, German Sviridov, Dmitrii Ustiugov +1AI Coding AgentsCost-Aware Inference

  7. SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding

    Nov 17, 2026Selin Yildirim, Ali Zeyrek, Deming ChenSpeculative DecodingLLM Inference Acceleration

  8. XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools

    Nov 17, 2026Yijun Sun, Xudong Liao, Zhuobin Huang +3KV-Cache OffloadingLLM Inference Acceleration

  9. Communicate the Change: Efficient Expert Parallelism via Activation Compression

    Nov 17, 2026Fengyuan Liu, Jay Gala, Siva Reddy +1Communication-Efficient Mixture-of-ExpertsMixture-of-Experts Inference

  10. DNeo: Your Parallel Draft Model Needs Better Target Feature Encoding

    Nov 16, 2026Jameson Sandler, Shane Bergsma, Mostafa Elhoushi +5Speculative DecodingLLM Inference Acceleration

  11. Latent Bridge: Asking for Help Without Words

    Nov 16, 2026Qichao Wang, Tingyu Qu, Pengcheng Wu +5Multi-LLM CollaborationLLM Inference Acceleration

  12. RoShift: Lazy Shift and Weight-aware Rotation for Activation Sparsity in LLM Inference

    Nov 16, 2026Geunsoo Song, Jiwon Seo, Hoeseok Yang +1LLM Inference EfficiencyActivation Sparsity

  13. GaugeRoPE: RoPE-Compatible Executable Latent Attention

    Nov 16, 2026Yuanhang Gao, Xiangrui Yang, Lingyuan Meng +2KV-Cache CompressionLLM Inference Acceleration

  14. CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

    Nov 15, 2026Byeongju Kim, Hoonki Lee, Byungjun Kim +4On-Device Language Model InferenceMixture-of-Experts Inference

  15. Multi-Branch Transformer

    Nov 15, 2026Gaotang Li, Lingjie Chen, Chunyuan Deng +6TransformerLanguage Model Scaling

  16. LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

    Nov 14, 2026Siqi Zhu, Suozhi Huang, Yi Zhao +1Speculative DecodingLLM Inference Acceleration

  17. WaferTetris: Efficient Freeform Mapping on Wafer-Scale Accelerators

    Nov 13, 2026Dayou Du, Congjie He, Andrew Boutros +3Neural Network Hardware AccelerationLLM Inference Acceleration

  18. The State Is Not the Bottleneck: A Paged-State Gated-DeltaNet Tile on a Tri-Mode Multicast Mesh, Bit-Exact from RTL to Silicon

    Nov 13, 2026Cong Xu, Ravi SankarLLM Inference AccelerationFPGA-Based Neural Network Acceleration