Efficient Transformer Inference

Ask about this topic

Overview

The field aims to make Transformer serving faster and cheaper, especially for autoregressive decoding and long contexts where attention, KV caches, memory movement, and hardware utilization are bottlenecks. Approaches include linear, sparse, windowed, or phase-aware attention; cache compression and sharing; quantization, pruning, and early exits; and optimized kernels, compilers, accelerators, and distributed or encrypted execution. Many methods adapt computation by token, layer, or workload, while others co-design model structure with hardware such as GPUs, CPUs, NPUs, and FPGAs. The central tension is turning theoretical savings into measured end-to-end speed and energy gains without unacceptable quality loss, since results depend on the model, workload, and hardware.

Latest papers

CardsList

122 papers15 papers in the last four weeks, up 7% on the four weeks before. 0.1% of all new papers.

  1. Evaluating CUDA Tile for AI Workloads on Hopper and Blackwell GPUs

    Apr 25, 2026Divakar Kumar Yadav, Tian Zhao, Deepak KumarEfficient Transformer InferenceGPU Kernel Optimization

    University of Wisconsin–Milwaukee Milwaukee, WI, USA · Illinois Institute of Technology Chicago, IL, USA

  2. The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

    Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language ModelingSelf-Attention

    Harvard University

  3. Decoding Text Spans for Efficient and Accurate Named-Entity Recognition

    Apr 22, 2026Andrea Maracani, Savas Ozkan, Junyi Zhu +2Efficient Transformer InferenceNamed Entity Recognition

    Samsung Research UK, Staines-upon-Thames, United Kingdom

  4. EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessment

    Apr 21, 2026Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira +4Efficient Transformer InferenceEfficient Inference

    Fraunhofer Institute for Computer Graphics Research IGD, Darmstadt, Germany · Department of Computer Science, TU Darmstadt, Germany

  5. Depth Adaptive Efficient Visual Autoregressive Modeling

    Apr 19, 2026Chunliang Li, Tianze Cao, Sanyuan ZhaoEfficient Transformer InferenceVisual Autoregressive Models

    Beijing Institute of Technology · Beijing Institute of Technology, Zhuhai

  6. Adaptive Computation Depth via Learned Token Routing in Transformers

    Apr 18, 2026Ahmed Abdelmuniem Abdalla MohammedEfficient Transformer InferenceTransformer

    Independent Researcher

  7. SecureRouter: Encrypted Routing for Efficient Secure Inference

    Apr 16, 2026Yukuan Zhang, Mengxin Zheng, Qian LouEfficient Transformer InferenceSecure Multi-Party Computation

    University of Central Florida · Orlando, Florida, USA

  8. Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU

    Apr 16, 2026Jevin Jiang, Ying Chen, Blake A. Hechtman +2Efficient Transformer InferenceAI Accelerator Inference

    Google

  9. Dispatch-Aware Ragged Attention for Pruned Vision Transformers

    Apr 16, 2026Seifeldin Abdellatif, Ahmad AlmasriEfficient Transformer InferenceVisual Attention

    College of Engineering Al Ain University Abu Dhabi, United Arab Emirates

  10. Forge-UGC: FX optimization and register-graph engine for universal graph compiler

    Apr 14, 2026Satyam Kumar, Saurabh JhaCompiler OptimizationAI Accelerator Inference

  11. Sustainable Transformer Neural Network Acceleration with Stochastic Photonic Computing

    Apr 10, 2026S. Afifi, O. Alo, I. Thakkar +1Efficient Transformer InferenceTransformer Inference

    Department of Electrical and Computer Engineering, Colorado State University, Fort Collins, CO, USA · Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY, USA

  12. ELT: Elastic Looped Transformers for Visual Generation

    Apr 10, 2026Sahil Goyal, Swayam Agrawal, Gautham Govind Anil +3Image GenerationTransformer

  13. Efficient Matrix Implementation for Rotary Position Embedding

    Apr 10, 2026Chen Minqi, Zhongqi Yue, Shihao Zhang +5Rotary Positional EmbeddingsEfficient Transformer Inference

    Huawei Technologies · Nanyang Technological University

  14. Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration

    Apr 8, 2026Md Zesun Ahmed Mia, Jiahui Duan, Kai Ni +1Compute-in-MemoryEfficient Transformer Inference

    The Pennsylvania State University, Department of Electrical Engineering, University Park, PA, USA · University of Notre Dame, Department of Electrical Engineering, Notre Dame, IN, USA

  15. AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems

    Apr 3, 2026Zhaoting Gong, Ran Ran, Fan Yao +1Efficient Transformer InferenceHomomorphic Encryption

    North Carolina State University, Raleigh, NC, USA · University of Central Florida, Orlando, FL, USA

  16. Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference

    Apr 2, 2026Dimitrios Danopoulos, Enrico Lupi, Michael Kagan +1Efficient Transformer InferenceEfficient Inference

    European Organization for Nuclear Research (CERN), Geneva, Switzerland · SLAC National Accelerator Laboratory, Menlo Park, CA, USA

  17. GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference

    Mar 30, 2026Soutrik Mukherjee, Sangwhan ChaEfficient Transformer InferenceEfficient Neural Network Inference

    Department of Computer and Information Sciences, Harrisburg University of Science and Technology, Harrisburg, PA 17101, USA

  18. Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization

    Mar 23, 2026Hung-Hsuan ChenEfficient Transformer InferenceRecurrent Transformers

    Computer Science and Information Engineering National Central University Taoyuan, Taiwan

  19. A transformer architecture alteration to incentivise externalised reasoning

    Mar 22, 2026Elizabeth Pavlova, Mariia Koroliuk, Karthik Viswanathan +3Efficient Language Model InferenceLarge Reasoning Models

    MARS · Geodesic Research, equal contribution · University of Amsterdam

  20. OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework

    Mar 20, 2026Weixuan Zeng, Pengcheng Wei, Huaiqing Wang +8Virtual Try-OnDiffusion Transformer

    The Chinese University of Hong Kong, ShenZhen · Beihang University · KuaiShou

  21. DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training

    Mar 19, 2026Maoyang Xiang, Bo WangNeural Network Activation FunctionsEfficient Transformer Inference

    Information Systems Technology and Design Singapore University of Technology and Design Singapore

  22. Efficient Reasoning at Fixed Test-Time Cost via Length-Aware Attention Priors and Gain-Aware Training

    Mar 10, 2026Rian AtriTransformer AttentionEfficient Transformer Inference

    Serval Systems

  23. Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers

    Mar 9, 2026Shubham Aggarwal, Lokendra KumarWalsh-Hadamard TransformMulti-Head Attention

  24. Neural Dynamics Self-Attention for Spiking Transformers

    Mar 9, 2026Dehao Zhang, Fukai Guo, Shuai Wang +6Spiking TransformersSpiking Neural Networks

    University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)

  25. TP-Spikformer: Token Pruned Spiking Transformer

    Feb 28, 2026Wenjie Wei, Xiaolong Zhou, Malu Zhang +8Spiking TransformersToken Pruning

    University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · Northumbria University +1

  26. Incremental Transformer Neural Processes

    Feb 21, 2026Philip Mortimer, Cristiana Diaconu, Tommy Rochussen +2Streaming InferenceTransformer

    University of Cambridge · Helmholtz AI · Technical University of Munich +2

  27. DRetHTR: Linear-Time Decoder-Only Retentive Network for Handwritten Text Recognition

    Feb 19, 2026Changhun Kim, Martin Mayr, Thomas Gorges +4Memory-Efficient InferenceEfficient Transformer Inference

    Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany

  28. Uncertainty-Guided Inference-Time Depth Adaptation for Transformer-Based Visual Tracking

    Feb 18, 2026Patrick Poggi, Divake Kumar, Theja Tulabandhula +1Visual Object TrackingAdaptive Inference

    ∗University of Illinois at Chicago

  29. Revisiting the Shape Convention of Transformer Language Models

    Feb 6, 2026Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin +2Efficient Transformer InferenceTransformer FFNs

    MediaTek Research · National Taiwan University

  30. Transform Trained Transformer for Accelerating Native 4K Video Generation

    Dec 15, 2025Jiangning Zhang, Junwei Zhu, Teng Hu +9Efficient Transformer InferenceHigh-Resolution Video Generation

    Youtu Lab, Tencent · Zhejiang University