Inference Acceleration

Follow this topic Ask about this topic

Inference acceleration studies how to make trained models produce outputs faster and at lower cost, from language and video generation to real-time, scientific, and recommendation workloads. Common approaches include pruning or reusing intermediate features, speculative decoding and caching with error correction, efficient kernels and numerical formats, and serving techniques such as batching, scheduling, and off-path computation. Methods are evaluated across benchmarks, production services, and edge devices using metrics such as latency, throughput, memory, and retained quality or task success. The central tension is balancing computation and memory savings against approximation error, while making optimizations work across different models and hardware.

Latest papers

18 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Training-free Acceleration for Blind Compressed Image Restoration via Activation-Calibrated Singular Value Decomposition
Oct 21, 2026Suhang Yao, Xin Li, Xijun Wang +3Inference Acceleration Image Restoration
University of Science and Technology of China, Hefei, China · SparcAI Inc. · National University of Singapore, Singapore
ReplaySSM: Cache SSM Inputs, Not State
Oct 9, 2026Ze-Wei Liou, Harris Nover, Amey Naik +4Inference Acceleration LLM Inference Acceleration
Princeton University · NVIDIA
TRAC: Trajectory-aware Reuse and Adaptive Correction for Efficient Autoregressive Video Generation
Oct 2, 2026Jiaxing Song, Weiqi Yan, You Huang +4Autoregressive Video Generation Inference Acceleration
School of Computer Science and Technology, Hainan University · MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University · Zhejiang Normal University, Jinhua, China
One Basis to Animate Them All: Gaussian Blendshape Distillation for Real-Time Avatars
Oct 1, 2026Ramazan Fazylov, Stamatis Lefkimmiatis, Ivan LaptevInference Acceleration Human Avatar Animation
Mohamed bin Zayed University of Artificial Intelligence · MWS AI
CAST: Reconstruction-Coupled Acceleration of Interactive World Models
Sep 28, 2026Leyang Chen, Junyi Wu, Fanqing Kong +2Real-Time Interactive Video Generation Inference Acceleration
Shanghai Jiao Tong University
Cache-Aware Conv3D Lowering Across Embedded World-Model Decoders
Sep 25, 2026Jiaming Zhang, Wu Yang, Shuai Tao +1Inference Acceleration On-Device Inference
University of Wisconsin–Madison, Madison, WI, USA · Beta Infinity, Beijing, China
OMatG-flash: An All-Atom Flow Map with Reinforce Adjoint Matching for Scalable Materials Discovery
Sep 22, 2026Thomas Egg, Harry Winston Sullivan, Ellad B. Tadmor +1Flow Matching Crystal Structure Prediction
New York University · University of Minnesota
How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF
Jul 22, 2026Venkata Naga Sai Vishnu Rohit Pulipaka, Anish Katta, Deva Rohit Reddy PeddireddyEfficient Neural Network Inference Reward Modeling
AMICA-Python: Adaptive Mixture Independent Component Analysis with Anderson Acceleration
Jul 20, 2026Scott Huberty, Christian O'ReillyUnsupervised Learning Independent Component Analysis
University of Southern California, Los Angeles, CA, USA · University of South Carolina, Columbia, SC, USA
CSD: Content-aware Speculative Decoding for Efficient Image Generation
Jun 26, 2026Mingcheng Wang, Junbo Qiao, Yunchen Li +8Speculative Decoding Inference Acceleration
East China Normal University · Huawei Foundation Model · Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE
MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration
Jun 16, 2026Kosmas Alexandridis, Giorgos DimitrakopoulosAI Accelerator Inference Inference Acceleration
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece
FMplex: Model Virtualization for Serving Extensible Foundation Models
Jun 8, 2026Hetvi Shastri, Pragya Sharma, Walid A. Hanafy +3Efficient Neural Network Inference Inference Acceleration
University of Massachusetts Amherst · University of California Los Angeles
C $^3$ ache: Accelerating World Action Models with Cross Inference Chunk Cache
Jun 8, 2026Weisen Zhao, Lam Nguyen, Zhicong Lu +1World Model Learning Action-Conditioned World Models
George Mason University · University of Central Florida
Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend
May 7, 2026Tianlun Hu, Tiancheng Hu, Shengsheng Litang +8Communication-Efficient Mixture-of-Experts Mixture-of-Experts Inference
Huawei Technologies
OptiLookUp: An Optical ROM-Based Lookup Table Engine for Photonic Accelerators
May 5, 2026Ankur Singh, Akhilesh JaiswalNeural Network Activation Functions Inference Acceleration
Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA
SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data
May 1, 2026Shashank Kapadia, Deep Narayan Mishra, Sujal Reddy Alugubelli +3Efficient Neural Network Inference GPU Acceleration
Walmart Inc. · USA
SOLARIS: Speculative Offloading of Latent-bAsed Representation for Inference Scaling
Apr 13, 2026Zikun Liu, Liang Luo, Qianru Li +31Inference Acceleration Recommender Systems
Meta AI, Menlo Park, California, USA
StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models
Mar 2, 2026Keli Liu, Zhendong Wang, Wengang Zhou +1Inference Acceleration Visual Autoregressive Models
University of Science and Technology of China

Oct 21, 2026J/K move · Enter open · S save

Training-free Acceleration for Blind Compressed Image Restoration via Activation-Calibrated Singular Value Decomposition

Suhang Yao, Xin Li, Xijun Wang, Yufei Wang+2

University of Science and Technology of China, Hefei, China · SparcAI Inc. · National University of Singapore, Singapore

Inference Acceleration Image Restoration Singular Value Decomposition Low-Rank Compression

Open paper AskPDF arXiv

Transformer-based models achieve strong performance on blind compressed image restoration (CIR) under unknown compression settings, but their densely stacked projection operators make deployment costly. We study training-free acceleration for such models through an empirical, operator-level analysis. Specifically, we profile where computation is concentrated, characterize channel-wise activation distributions under compressed inputs, and examine the singular-value decay of representative projection weights. These measurements identify the recurring projections in the attention, feed-forward, convolutional, and prompt-conditioning pathways as effective low-rank replacement targets that preserve the original inference graph. They further establish truncated SVD (TSVD) as a principled training-free baseline for blind CIR and reveal why a uniform low-rank objective is mismatched to the channel-imbalanced feature distributions of compressed images. Accordingly, we formulate Activation-Calibrated Singular Value Decomposition (ASVD) for this restoration setting, deriving low-rank factors entirely offline from pretrained weights and lightweight activation statistics gathered from only a few compressed calibration images. By weighting the decomposition with channel-wise activation magnitudes, our ASVD assigns larger penalties to errors on more active channels, requiring neither retraining nor online calibration and leaving the restoration pipeline intact. When instantiated on PromptCIR at rank 64, ASVD consistently improves PSNR-B over TSVD across four standard benchmarks under identical parameter and MAC budgets. At this operating point, the factorized model cuts parameters by 59% (34.75M to 14.34M) and computation by 16.1% (258.79 to 217.16 GMACs for a 256 x 256 input), enabling acceleration without retraining.