Efficient Transformer Inference

Follow this topic Ask about this topic

The field aims to make Transformer serving faster and cheaper, especially for autoregressive decoding and long contexts where attention, KV caches, memory movement, and hardware utilization are bottlenecks. Approaches include linear, sparse, windowed, or phase-aware attention; cache compression and sharing; quantization, pruning, and early exits; and optimized kernels, compilers, accelerators, and distributed or encrypted execution. Many methods adapt computation by token, layer, or workload, while others co-design model structure with hardware such as GPUs, CPUs, NPUs, and FPGAs. The central tension is turning theoretical savings into measured end-to-end speed and energy gains without unacceptable quality loss, since results depend on the model, workload, and hardware.

Latest papers

122 papers15 papers in the last four weeks, up 7% on the four weeks before. 0.1% of all new papers.

How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference
Oct 29, 2026Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan +5Recurrent Transformers Efficient Transformer Inference
FAIR at Meta · MIT
Quantized Edge Inference of a Self-Supervised SAR-Optical Encoder on Commodity x86 Hardware
Oct 27, 2026Christian Sanchez-Leon, Dongming ChenEfficient Transformer Inference Edge Inference
Software College, Northeastern University, Shenyang 110819, China
GPA-VGGT: Geometry-Preserving Acceleration of VGGT
Oct 27, 2026Muyeol Sung, Jangwon Kim, Jongwoo LimEfficient Transformer Inference 3D ViTs
Seoul National University
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Recurrent Transformers Efficient Transformer Inference
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering
Oct 20, 2026Sergey Salishev, Zeev Volkovich, Oleg GranichinEfficient Transformer Inference Transformer Attention
St. Petersburg State University, Saint Petersburg 199034, Russia · Department of Software Engineering, Braude College of Engineering, Karmiel 2161002, Israel
LEAF: LUT-aware Efficient Attention on FPGAs
Oct 19, 2026Arianna Cox, Chang Sun, Lauri Laatu +7Efficient Transformer Inference FPGA-Based Neural Network Acceleration
Imperial College London, United Kingdom · California Institute of Technology, USA · University of Bristol, United Kingdom
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Oct 16, 2026Wei WangEfficient Transformer Inference GPU Kernel Optimization
TRACE: Transformer Residual Activation Codebook Encoding for Communication-efficient Distributed Edge Inference
Oct 15, 2026Muhammad Azlan Qazi, Alexandros Iosifidis, Qi ZhangEfficient Transformer Inference Transformer Inference
DIGIT and the Department of Electrical and Computer Engineering, Aarhus University, Denmark · the Data Science Research center, Tampere University, Finland
Your Transformer Is Secretly Elastic
Oct 14, 2026Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts +1Elastic Neural Networks Transformer Pruning
University of Amsterdam · TNO, Intelligent Imaging
A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference
Oct 14, 2026Souvik Das, Ariful Azad, Nitesh SaxenaTransformer Inference Privacy-Preserving Language Models
Texas A&M University, College Station, TX, USA
Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules
Oct 13, 2026Sarah Pan, Ali BehrouzLong-Context Language Model Inference Memory-Augmented Language Models
Massachusetts Institute of Technology · Cornell University
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Oct 9, 2026Yuhong Chou, Zehao Liu, Yuqi Pan +4Transformer Attention Attention Mechanisms
The Hong Kong Polytechnic University · University of Chinese Academy of Sciences · Independent Researcher +1
Tabular Foundation Models Are Effectively Shallow
Oct 9, 2026Irene Cannistraci, Julia E. VogtTransformer Pruning Tabular Foundation Models
Department of Computer Science ETH Zurich Switzerland
How Much Computation Does Every Forecast Need? Structured Block Skipping in Two Time-Series Foundation Models
Oct 9, 2026Max BourgeatLayer Skipping in Language Model Decoding Efficient Transformer Inference
Polytechnique Montréal Montréal, QC, Canada
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache Quantization Efficient Transformer Inference
KAIST · Yonsei University · Seoul National University
QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing
Oct 7, 2026Yueying Li, Zhongle Xie, Ke Chen +1Efficient Transformer Inference Conditional Computation
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Cleave: Scaling Tensor Program Optimization via Decoupled Algebraic Search and Operator Scheduling
Oct 6, 2026David Pissarra, Jinkun Lin, Haitian Jiang +2Compiler Optimization GPU Kernel Optimization
New York University New York, USA
Decoding Looped Transformers Better for (Almost) Free
Oct 1, 2026Weihao Liu, Huangjie Zheng, Tianrong Chen +5Efficient Transformer Inference Language Model Decoding
Apple
Efficient World Action Model Inference with Adaptive Intermediate States
Sep 28, 2026Zhinnan Liu, Haozhi Han, Ruge Zhang +8Efficient Transformer Inference World Model-Based Planning
Xiamen University · Institute for AI Industry Research, Tsinghua University · School of Computer Science, Peking University +2
Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration
Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiEfficient Transformer Inference Softmax Attention
University of Illinois Urbana-Champaign
Precision As You Need: Stochastic Computing Is a Dense Adaptive Quantizer
Sep 26, 2026Haoran Jin, Kangqi Zhang, Jirong Yang +4Adaptive Quantization Efficient Transformer Inference
Department of Electrical Engineering and Computer Science, University of Michigan
Write Back the $Δ$ : Revisiting the Same Tokens with Fresh Representations
Sep 26, 2026Wencheng Ye, Anning Hu, Xiangdong Zhang +5Efficient Transformer Inference Recurrent Transformers
Tongji University · Shanghai Jiao Tong University · Shanghai Innovation Institute
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
Sep 24, 2026Wanqi Yang, Shiwei LiuEfficient Transformer Inference Recurrent Transformers
ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems Tübingen AI Center
ShatterQuant: Breaking Uniform Precision with Block-Wise Mixed-Precision on a Systolic Transformer Hardware Accelerator
Sep 20, 2026Mikolaj Walczak, Edward Humes, Chao Fang +2Mixed-Precision Quantization AI Accelerator Inference
Johns Hopkins University Baltimore, Maryland, USA
Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers
Sep 16, 2026Zheng Liu, Xin Gao, Jinchao Zhu +1Parameter-Efficient Fine-Tuning Efficient Transformer Inference
School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China, and also with the Beijing Engineering Research Center of Industrial Spectrum Imaging, Beijing 100083, China · College of Software, Nankai University, Tianjin 300350, China · Department of Automation, BNRist, Tsinghua University, Beijing 100084, China
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Efficient Transformer Inference Speculative Decoding
1Seoul National University · 2KAIST
LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference
Sep 12, 2026Prateek Kumar SikdarEfficient Transformer Inference LLM Inference Acceleration
Accenture
Attention Quantization for Tabular Foundation Models
Sep 11, 2026Jonas M. Kübler, Benjamin Jäger, Klemens Flöge +2Efficient Transformer Inference Tabular Foundation Models
Prior Labs
EFQ-Softmax: Exp-Free Quantization for Softmax
Sep 9, 2026Haohui Han, Yuming Wan, Hongni Wang +4Efficient Transformer Inference Transformer Attention
Xi'an Jiaotong University · Huawei Technologies Co., Ltd · Shandong University of Finance and Economics
Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Sep 8, 2026Vishesh Tripathi, Abhay Kumar, Ramsha KhanEfficient Transformer Inference Grouped-Query Attention
FrontiersMind

Oct 29, 2026J/K move · Enter open · S save

How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale+4

FAIR at Meta · MIT

Recurrent Transformers Efficient Transformer Inference Parallel Decoding

Open paper AskPDF arXiv

Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these trade-offs jointly in dense decoder-only transformers trained from scratch, varying which parts of the model are looped, how many loops are applied, and how information is passed between loops. Our experiments span models with up to 1B parameters, trained on up to 155B tokens with up to

1.07\times10^{21}

FLOPs and 32 loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained. Adding loops is not uniformly beneficial: gains eventually saturate, reverse, or become unstable. This motivates asking whether forcing every loop to use the same weights limits the benefit of deeper looping. We address this by adding small loop-specific low-rank adapters that improve quality, with larger gains as more loops are added. Looped transformers also suffer a decoding slowdown proportional to the number of loops. Parallel Loop Transformers address this by passing hidden states between loops across adjacent token positions, allowing loop stages to run concurrently. However, in our experiments the original formulation does not consistently retain the quality gains of ordinary looping. We find that preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovers strong performance across model scales and training budgets. Together, these results show that effective looping requires matching the design to the resource constraint and inference setting.