LLM Inference Acceleration

LLM: Large Language Model

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,239 papers153 papers in the last four weeks, down 27% on the four weeks before. 1.2% of all new papers.

  1. Adaptive GoGI-Skip: Coupling Goal-Gradient Importance with Dynamic Uncertainty for Efficient Reasoning

    May 13, 2025Ren ZhuangGradient-Based AttributionLLM Pruning

  2. The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

    Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer InferenceLLM Evaluation

  3. Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

    Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model InferenceLLM Inference

  4. Towards Optimal Multi-draft Speculative Decoding

    Feb 26, 2025Zhengmian Hu, Tong Zheng, Vignesh Viswanathan +5Speculative DecodingLLM Inference Acceleration

  5. FlashNorm: Fast Normalization for Transformers

    Jul 12, 2024Nils Graef, Filip Makraduli, Andrew Wasielewski +1Efficient Transformer InferenceLLM Inference Acceleration

  6. FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference

    May 7, 2024Runheng Liu, Xingchen Xiao, Heyan Huang +2Knowledge Augmentation for Language ModelsLLM Inference Acceleration