Overview

Transformer inference covers the work needed to run trained models, from processing a prompt (prefill) to generating tokens one at a time (decoding). Common approaches reduce attention and KV-cache costs through sparsity, alternative attention mechanisms, compression, or cache reuse; others use quantization, parallel computation, adaptive depth, and specialized GPU, FPGA, or edge hardware. Researchers evaluate these methods on language-model quality, retrieval and reasoning benchmarks, latency, memory use, energy, and sometimes privacy guarantees. The central tension is reducing computation and memory without changing outputs or degrading task performance, while inconsistent hardware and evaluation assumptions make results difficult to compare.

Latest papers

CardsList

63 papers6 papers in the last four weeks, down 45% on the four weeks before. 0.0% of all new papers.

  1. Flexible Gravitational-Wave Parameter Estimation with Transformers

    Dec 2, 2025Annalena Kofler, Maximilian Dax, Stephen R. Green +6Transformer InferenceGravitational-Wave Astronomy

  2. The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

    Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer InferenceLLM Evaluation

  3. Numerical stability analysis of large language models

    Mar 13, 2025Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng +1Transformer InferenceTransformer Attention