Efficient Transformer Inference

Overview

The field aims to make Transformer serving faster and cheaper, especially for autoregressive decoding and long contexts where attention, KV caches, memory movement, and hardware utilization are bottlenecks. Approaches include linear, sparse, windowed, or phase-aware attention; cache compression and sharing; quantization, pruning, and early exits; and optimized kernels, compilers, accelerators, and distributed or encrypted execution. Many methods adapt computation by token, layer, or workload, while others co-design model structure with hardware such as GPUs, CPUs, NPUs, and FPGAs. The central tension is turning theoretical savings into measured end-to-end speed and energy gains without unacceptable quality loss, since results depend on the model, workload, and hardware.

Latest papers

CardsList

127 papers8 papers in the last four weeks, down 53% on the four weeks before. 0.1% of all new papers.

  1. InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers

    Nov 17, 2026Zhaofeng Sun, Lanxiang Hu, Yu-Yang Qian +6Parallel LLM InferenceParallel Decoding

  2. MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference

    Nov 10, 2026Linru Zhang, Xiangning Wang, Jun Jie Sim +5Privacy-Preserving Language ModelsPrivacy-Preserving ML

  3. ElTab: Elastically Looped Tabular Transformers

    Oct 30, 2026Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus +2Efficient Transformer InferenceTabular Prediction

  4. How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

    Oct 29, 2026Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan +5Recurrent TransformersEfficient Transformer Inference

  5. Quantized Edge Inference of a Self-Supervised SAR-Optical Encoder on Commodity x86 Hardware

    Oct 27, 2026Christian Sanchez-Leon, Dongming ChenEfficient Transformer InferenceEdge Inference

  6. GPA-VGGT: Geometry-Preserving Acceleration of VGGT

    Oct 27, 2026Muyeol Sung, Jangwon Kim, Jongwoo LimEfficient Transformer Inference3D ViTs

  7. LoopFold: Parallelizing Recurrent Depth in Looped Transformers

    Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Parallel Training of Recurrent NetworksRecurrent Transformers

  8. Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

    Oct 20, 2026Sergey Salishev, Zeev Volkovich, Oleg GranichinEfficient Transformer InferenceTransformer Attention

  9. LEAF: LUT-aware Efficient Attention on FPGAs

    Oct 19, 2026Arianna Cox, Chang Sun, Lauri Laatu +7Efficient Transformer InferenceFPGA-Based Neural Network Acceleration

  10. TRACE: Transformer Residual Activation Codebook Encoding for Communication-efficient Distributed Edge Inference

    Oct 15, 2026Muhammad Azlan Qazi, Alexandros Iosifidis, Qi ZhangEfficient Transformer InferenceTransformer Inference

  11. Your Transformer Is Secretly Elastic

    Oct 14, 2026Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts +1Elastic Neural NetworksTransformer Pruning

  12. Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

    Oct 9, 2026Yuhong Chou, Zehao Liu, Yuqi Pan +4Transformer AttentionAttention Mechanisms

  13. Tabular Foundation Models Are Effectively Shallow

    Oct 9, 2026Irene Cannistraci, Julia E. VogtTransformer PruningTabular Foundation Models

  14. ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

    Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache QuantizationEfficient Transformer Inference

  15. QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing

    Oct 7, 2026Yueying Li, Zhongle Xie, Ke Chen +1Efficient Transformer InferenceConditional Computation

  16. Efficient World Action Model Inference with Adaptive Intermediate States

    Sep 28, 2026Zhinnan Liu, Haozhi Han, Ruge Zhang +8Efficient Transformer InferenceWorld Model-Based Planning

  17. Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration

    Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiEfficient Transformer InferenceSoftmax Attention

  18. Precision As You Need: Stochastic Computing Is a Dense Adaptive Quantizer

    Sep 26, 2026Haoran Jin, Kangqi Zhang, Jirong Yang +4Adaptive QuantizationEfficient Transformer Inference

  19. Write Back the ΔΔ: Revisiting the Same Tokens with Fresh Representations

    Sep 26, 2026Wencheng Ye, Anning Hu, Xiangdong Zhang +5Efficient Transformer InferenceRecurrent Transformers

  20. FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

    Sep 24, 2026Wanqi Yang, Shiwei LiuEfficient Transformer InferenceRecurrent Transformers

  21. Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers

    Sep 16, 2026Zheng Liu, Xin Gao, Jinchao Zhu +1Parameter-Efficient Fine-TuningEfficient Transformer Inference

  22. LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

    Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Efficient Transformer InferenceSpeculative Decoding