Efficient Transformer Inference

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 79

All topics

Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers
Jul 6, 2026Ligong Han, Kai Xu, Hao Wang +3Efficient Transformer Inference Transformer Inference
MBZUAI IFM · Red Hat AI Innovation · MIT-IBM Watson AI Lab +3
SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers
Jul 3, 2026Jianing Deng, Yuanzhe Li, Jialu Wang +4Efficient Transformer Inference 3D ViTs
University of Pittsburgh, Pittsburgh, PA, USA · University of Arizona, Tucson, AZ, USA · Tongji University, Shanghai, China +2
NPUsper: Eliminating Redundant Computation for Real-Time Whisper on Mobile NPUs
Jul 1, 2026Sihyeon Lee, Hojeong Lee, Sungwon Woo +3Efficient Transformer Inference Streaming ASR
Korea University · University of Wisconsin–Madison
The Context-Ready Transformer
Jun 25, 2026Mahesh GodavartiEfficient Transformer Inference Transformer Inference
A Carrot, Inc
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
Jun 25, 2026Huzama Ahmad, Cao Viet Hai Nam, Se-Young YunEfficient Transformer Inference Transformer
KAIST
Decentralised AI Training and Inference with BlockTrain
Jun 23, 2026Peter Toth, Dan OprisaEfficient Transformer Inference Decentralized Learning
Spheroid Labs∗
Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment
Jun 23, 2026Disha PatelEfficient Transformer Inference Adaptive Inference
Department of Computer Science California State University, Fullerton
Unlimited OCR Works
Jun 22, 2026Youyang Yin, Huanhuan Liu, YY +14Efficient Transformer Inference Long-Context Language Model Inference
Baidu Inc.
HEPTv2: End-to-End Efficient Point Transformer for Charged Particle Reconstruction
Jun 18, 2026Siqi Miao, Shitij Govil, Jack P. Rodgers +5Efficient Transformer Inference Transformer
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA. · Department of Physics and Astronomy, Purdue University, West Lafayette, IN, USA. · Department of Physics, University of California San Diego, La Jolla, CA, USA. +1
LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling
Jun 16, 2026Jian Yang, Shawn Guo, Wei Zhang +16Efficient Transformer Inference Test-Time Scaling
1Beihang University · 2IQuest Research · 4Renming University of China +1
Reconfigurable Computing Challenge: Transformer for Jet Tagging on Versal AI Engines
Jun 16, 2026Gram Koski, Sean Lipps, Zhenghua Ma +2Efficient Transformer Inference Efficient Neural Network Inference
Department of Computer Science and Engineering University of California San Diego La Jolla, CA, USA
Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer
Jun 11, 2026Zhanglu Yan, Jiayi Mao, Kaiwen Tang +6Efficient Transformer Inference Spiking Transformers
Department of Computer Science, National University of Singapore · School of Engineering, Westlake University · College of Computer Science and Technology, Zhejiang University +2
TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
Jun 8, 2026Yejin Lee, Junwon Moon, Hyoeun Kim +3Efficient Transformer Inference Audio Token Compression
Sungkyunkwan University · University of Seoul
When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet
Jun 4, 2026Luoming Zhang, Yuwei Ren, Kui Zhang +7Efficient Transformer Inference LLM Inference Acceleration
Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc.
AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference
Jun 4, 2026Xuanzhe Li, Ziyan Weng, Zhiyu Zhu +1Efficient Transformer Inference GPU Acceleration
City University of Hong Kong (Dongguan) · City University of Hong Kong
Depth-Attention: Cross-Layer Value Mixing for Language Models
Jun 3, 2026Boyi Zeng, Yiqin Hao, Zitong Wang +7Efficient Transformer Inference Decoder-Only Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute +2
Operator Fusion for LLM Inference on the Tensix Architecture
Jun 3, 2026Qingbo Wu, Ke Li, Wenzhu Wang +3Efficient Transformer Inference AI Accelerator Inference
KylinSoft Co., Ltd, Tianjin, China · openKylin Community, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology, Changsha, China
DxPTA: An Architecture Design Space Exploration with Optical Dataflow-guided Strategy for HW/SW Co-Design of Photonic Transformer Accelerators
Jun 2, 2026Rachmad Vidya Wicaksana Putra, Solomon Micheal Serunjogi, Mahmoud Rasras +1Efficient Transformer Inference Hardware-Software Co-Design
eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi, United Arab Emirates · Photonic Research Lab (PRL), Division of Engineering, New York University (NYU) Abu Dhabi, United Arab Emirates
Do Transformers Need Three Projections? Systematic Study of QKV Variants
Jun 1, 2026Ali Kayyam, Anusha Madan Gopal, M Anthony LewisEfficient Transformer Inference Transformer Attention
BrainChip Inc., Laguna Hills, CA, USA.
Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference
May 27, 2026Alan FerrariEfficient Transformer Inference Self-Attention
Knowledge Lab AG Zürich, Switzerland
JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search
May 26, 2026Dongyun Zou, Zhuoyang Zhang, Junyu Chen +8Efficient Transformer Inference Vision Foundation Models
MIT · NVIDIA · University of Pennsylvania +1
Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment
May 25, 2026Muhammad Azlan Qazi, Alexandros Iosifidis, Qi ZhangEfficient Transformer Inference Transformer Inference
Aarhus University · Aarhus, Denmark · Tampere University +1
Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers
May 20, 2026Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros +4Efficient Transformer Inference AI Accelerator Inference
Computing Systems Lab Huawei Technologies2026 Switzerland
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Efficient Transformer Inference Recurrent Transformers
1Rice University · 2Apple · 3UC Santa Cruz +1
SparseSAM: Structured Sparsification of Activations in Segment Anything Models
May 17, 2026Hoai-Chau Tran, Chi H. Nguyen, Duy M. H. Nguyen +3Efficient Transformer Inference Image Segmentation
University of Illinois at Urbana-Champaign · VinUni-Illinois Smart Health Center, VinUniversity · College of Engineering & Computer Science, VinUniversity +3
Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices
May 15, 2026Alexander Gräfe, Ding Huo, Vincent de Bakker +3Efficient Transformer Inference Transformer Inference
1RWTH Aachen University · 2TU Darmstadt
N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
May 13, 2026Aleksander Lorenc, Frédéric Berdoz, Joël Mathys +1Efficient Transformer Inference LLM Inference Acceleration
ETH Zurich
Exact Linear Attention
May 13, 2026Weinuo OuEfficient Transformer Inference Self-Attention
Wuyi University
Block-Based Double Decoders
May 11, 2026Asher Labovich, Benjamin Bradley, Vanessa Alexander +1Efficient Transformer Inference LLM Inference Acceleration
Department of Computer Science Brown University Providence, RI 02912
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Efficient Transformer Inference LLM Compression
Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1

Jul 6, 2026, cs.LGJ/K move · Enter open · S save

Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

Ligong Han, Kai Xu, Hao Wang, Ruijiang Gao+2

MBZUAI IFM · Red Hat AI Innovation · MIT-IBM Watson AI Lab+3

Efficient Transformer Inference Transformer Inference Homomorphic Encryption

Open paper AskPDF arXiv

Fully homomorphic encryption (FHE) enables computation on encrypted data, but practical encrypted Transformer inference is bottlenecked by the sequential composition of many nonlinear blocks. We study whether Structured Newton Layer Parallelism (SNLP) can make this inter-layer composition more FHE-friendly: each Transformer block still requires polynomial approximations for operations such as softmax and RMSNorm, but SNLP reduces the layerwise sequential nonlinear depth from L stages to a small number of solver iterations plus linear structured corrections. Using a simulation framework based on Chebyshev polynomial approximations, we measure error accumulation under sequential versus SNLP inference across 8 models and 4 architecture families. On a 0.5B IDN-trained model, SNLP reduces symbolic bootstraps from 53 to 20 (2.65x) with only +1.2% perplexity degradation, while lowering error amplification (1.36x vs. 1.42x). Across all tested models, SNLP has lower amplification than sequential inference. Ablations show that softmax approximation dominates the error budget and CKKS arithmetic noise is negligible in our setting, suggesting that SNLP is complementary to block-level FHE-friendly operator design rather than a replacement for it.

Momentum

Related topics

Latest papers 79