Transformer Inference

Ask about this topic

Overview

Transformer inference covers the work needed to run trained models, from processing a prompt (prefill) to generating tokens one at a time (decoding). Common approaches reduce attention and KV-cache costs through sparsity, alternative attention mechanisms, compression, or cache reuse; others use quantization, parallel computation, adaptive depth, and specialized GPU, FPGA, or edge hardware. Researchers evaluate these methods on language-model quality, retrieval and reasoning benchmarks, latency, memory use, energy, and sometimes privacy guarantees. The central tension is reducing computation and memory without changing outputs or degrading task performance, while inconsistent hardware and evaluation assumptions make results difficult to compare.

Latest papers

CardsList

57 papers8 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. INTRA: Interleaved Non-contiguous Token spaRse Attention

    Oct 25, 2026Yuxuan Xia, Wenbo Lu, Shaoyi Zheng +1Sparse AttentionTransformer Inference

    New York University

  2. TRACE: Transformer Residual Activation Codebook Encoding for Communication-efficient Distributed Edge Inference

    Oct 15, 2026Muhammad Azlan Qazi, Alexandros Iosifidis, Qi ZhangEfficient Transformer InferenceTransformer Inference

    DIGIT and the Department of Electrical and Computer Engineering, Aarhus University, Denmark · the Data Science Research center, Tampere University, Finland

  3. When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

    Oct 14, 2026Wenwen Gong, Haotian Gu, Zhen Yang +6Tabular Foundation ModelsTransformer Inference

    China Agricultural University · Tsinghua University · Nankai University

  4. A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference

    Oct 14, 2026Souvik Das, Ariful Azad, Nitesh SaxenaTransformer InferencePrivacy-Preserving Language Models

    Texas A&M University, College Station, TX, USA

  5. Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

    Oct 9, 2026Weiqi Sun, Qianyue Wu, Wenjia Wang +1Attention ApproximationEfficient Attention

    The Chinese University of Hong Kong, Shenzhen · Beijing Normal University, Zhuhai · National University of Singapore

  6. EntroPrefill: Renyi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generation

    Oct 7, 2026Inbasekaran SToken PruningTransformer Pruning

    SRM Institute of Science and Technology, India

  7. Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2

    Oct 1, 2026Yohan Chatelain, Pablo de Oliveira CastroTransformer InferenceStochastic Rounding

    Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada · Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France

  8. The Geometry of Inference in Transformer Residual Streams

    Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer InterpretabilityTransformer Inference

    University of Luxembourg, Luxembourg · London Institute for Mathematical Sciences, London, UK

  9. The Extender: A Log-Structured Transformer

    Sep 26, 2026Jakob ErikssonTransformerTransformer Inference

    UIC · University of Illinois Chicago

  10. Memory as a cache: Exact context reuse and deletion by construction

    Sep 26, 2026Shengyao Wang, Jiang LiuKV CachingTransformer Inference

  11. DegreeSpar: Structured Degree Sparsity for Efficient Secure Transformer Inference

    Sep 26, 2026Yifei Cai, Zhuoran Li, Xiaozuo Shen +2Transformer InferencePrivacy-Preserving ML

    Iowa State University, Ames, IA, USA · University of Arizona, Tucson, AZ, USA

  12. Programs-of-Layers in LLMs through the Lens of Cortical Areas

    Sep 25, 2026Justus Westerhoff, Stephan Olbrich, Hatem Oraby +2Transformer InferenceAdaptive Computation

    Berliner Hochschule für Technik (BHT), Berlin, Germany · Humboldt-Universität zu Berlin, Berlin, Germany

  13. A Smaller Transformer in Your Transformer

    Sep 17, 2026Dhananjay Tomar, Marius Aasan, Andreas Kleppe +1Transformer InferenceVision Transformer

    Institute for Cancer Genetics and Informatics Oslo University Hospital Oslo, Norway · Department of Informatics University of Oslo Oslo, Norway · SFI Visual Intelligence UiT The Arctic University of Norway Tromsø, Norway

  14. MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators

    Sep 16, 2026Dong Liu, Yanxuan YuTransformer InferenceKV Caching

    Columbia University New York, NY, USA

  15. Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey

    Sep 1, 2026Arjan Blankestijn, Uraz Odyurt, Amirreza YousefzadehEfficient Transformer InferenceTransformer Inference

    Computer Architecture for Embedded Systems, University of Twente Enschede, The Netherlands · Faculty of Engineering Technology, University of Twente Enschede, The Netherlands

  16. CacheBridge: Efficient Cross-Model KV Cache Transfer

    Sep 1, 2026Xingyu Qu, Siyuan Lu, Zhiyu Chen +2Transformer InferenceKV Caching

    Westlake University. · †Work conducted during an internship at Westlake University. · Amazon. +1

  17. The Transformer Revolution, Part 1: Dynamic Processing through Output-Weight Interconnections

    Aug 4, 2026Marco Giunti, Fabrizia Giulia GaravagliaTransformer InferencePrompt Sensitivity

    ALOPHIS – Applied Logic Philosophy and History of Science Università di Cagliari · Dip. di Pedagogia, Psicologia, Filosofia Università di Cagliari

  18. At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

    Jul 28, 2026Bowen Wang, Chi Zhang, Diyou Shen +3Efficient Neural Network InferenceTransformer Inference

    *IIS, ETH Zürich, Switzerland · ‡Computing Systems Lab, Huawei, Switzerland · †DEI, Universit`a di Bologna, Italy

  19. Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs

    Jul 24, 2026Ilia Sobakinskikh, Paul Alexander BilokonTransformer InferenceFinancial Time Series

  20. Agree on the Model, Verify the Inference: GKR Protocols for HND-Based Transformer Inference

    Jul 23, 2026Xiaolong Liang, Juanjuan Li, Rui Qin +1Neural Network VerificationTransformer Inference

    State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

  21. MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel

    Jul 21, 2026Lenore Mulin, Gaetan HainsAttention MechanismsEfficient Transformer Inference

    University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil

  22. On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

    Jul 20, 2026Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5Dynamical SystemsKuramoto Model

    University of Wisconsin–Madison, Department of Statistics · University of Maryland, Department of Electrical and Computer Engineering · University of Warsaw, Institute of Applied Mathematics and Mechanics +2

  23. Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers

    Jul 15, 2026Joe LoganTransformer InferenceRecurrent Transformers

    Independent researcher

  24. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    Jul 6, 2026Ligong Han, Kai Xu, Hao Wang +3Efficient Transformer InferenceTransformer Inference

    MBZUAI IFM · Red Hat AI Innovation · MIT-IBM Watson AI Lab +3

  25. ELiTeFormer: An Efficient Transformer for FPGAs

    Jul 4, 2026Victor Agostinelli, Nicolas Bohm Agostini, Antonino TumeoLLM QuantizationTernary Quantization

    Oregon State University Corvallis, Oregon, USA · Pacific Northwest National Laboratory Richland, Washington, USA

  26. The risk of KV cache compression

    Jul 1, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Transformer InferenceKV Caching

    RWTH Aachen University · Stanford University

  27. The Context-Ready Transformer

    Jun 25, 2026Mahesh GodavartiEfficient Transformer InferenceTransformer Inference

    A Carrot, Inc

  28. Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

    Jun 20, 2026Xin GaoTransformer InferenceKV Caching

    Department of Mathematics and Statistics York University Toronto, M3J 1P3

  29. MemNovo: Look Back at the Spectrum for Balanced De Novo Peptide Sequencing from Mass Spectrometry

    Jun 10, 2026Dongxin Lyu, Jingbo Zhou, Hongxin Xiang +2Memory-Augmented Neural NetworksTransformer Inference

    Westlake University Hangzhou, Zhejiang, China · Hunan University Changsha, Hunan, China · Shanghai Artificial Intelligence Laboratory Shanghai, China +1

  30. Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior

    May 26, 2026Zeyi Huang, Xuehai He, LiLiang Ren +8Language Model PretrainingTransformer Inference

    Microsoft · University of Washington · University of Wisconsin-Madison