LLM Inference Acceleration

LLM: Large Language Model

Overview

This field studies how to make large language models answer faster and use less memory and computing power, especially during prompt processing and token-by-token generation. Common approaches include speculative decoding, which drafts and verifies tokens; sparse attention and KV-cache compression for long contexts; low-bit quantization; and adaptive computation that skips or reallocates work. Researchers test these methods on long-context and retrieval benchmarks such as LongBench and RULER, as well as reasoning and coding tasks, measuring both quality and serving metrics such as time to first token and throughput. A central challenge is turning theoretical reductions in computation or memory into real speedups on particular hardware without sacrificing accuracy or changing the model’s output distribution.

Latest papers

CardsList

1,233 papers153 papers in the last four weeks, down 27% on the four weeks before. 1.2% of all new papers.

  1. Towards Optimal Multi-draft Speculative Decoding

    Feb 26, 2025Zhengmian Hu, Tong Zheng, Vignesh Viswanathan +5Speculative DecodingLLM Inference Acceleration

  2. FlashNorm: Fast Normalization for Transformers

    Jul 12, 2024Nils Graef, Filip Makraduli, Andrew Wasielewski +1Efficient Transformer InferenceLLM Inference Acceleration

  3. FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference

    May 7, 2024Runheng Liu, Xingchen Xiao, Heyan Huang +2Knowledge Augmentation for Language ModelsLLM Inference Acceleration