Efficient Inference

Follow this topic Ask about this topic

Efficient inference studies how to get useful model outputs with less computation, memory, time, or energy, which matters for large-scale serving and devices with limited resources. Approaches include quantization and distillation, caching and reuse, sparse or low-rank computation, early exits, amortized inference, and adaptive allocation of test-time compute. Systems work complements these methods with batching, scheduling, compilation, and hardware-aware execution, while probabilistic methods seek faster posterior or marginal estimates without losing calibration or exactness. The central tension is reducing cost without undermining accuracy, uncertainty estimates, or robustness, and results can depend strongly on the workload and hardware used for evaluation.

Latest papers

107 papers14 papers in the last four weeks, up 17% on the four weeks before. 0.1% of all new papers.

Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference
Apr 2, 2026Dimitrios Danopoulos, Enrico Lupi, Michael Kagan +1Efficient Transformer Inference Efficient Inference
European Organization for Nuclear Research (CERN), Geneva, Switzerland · SLAC National Accelerator Laboratory, Menlo Park, CA, USA
DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computation
Mar 25, 2026Jingzhi Fang, Xiong Gao, Renwei Zhang +6Efficient Inference Dynamic Neural Networks
Huawei, China · Hunan University
Inference Time Policy Optimization for Offline RL with Differentiable World Models
Mar 23, 2026Rohan Deb, Stephen J. Wright, Arindam BanerjeeInference-Time Optimization Offline RL
Siebel School of Computing and Data Science, Department of Computer Sciences, University of Illinois Urbana-Champaign · University of Wisconsin Madison
Active Inference for Physical AI Agents -- An Engineering Perspective
Mar 21, 2026Bert de VriesActive Inference Embodied AI
Department of Electrical Engineering, Eindhoven University of Technology, the Netherlands
2K Retrofit: Entropy-Guided Efficient Sparse Refinement for High-Resolution 3D Geometry Prediction
Mar 20, 2026Tianbao Zhang, Zhenyu Liang, Zhenbo Song +7Efficient Inference 3D Reconstruction
BUAA · Dim12 AI Inc · Current Robotics +3
VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs
Mar 18, 2026Chaokang Jiang, Desen Zhou, Jiuming Liu +1World Models for Autonomous Driving Interactive World Models
Bosch, Shanghai, Shanghai, China · University of Cambridge, Cambridge, United Kingdom
Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference
Mar 10, 2026Cosmo Santoni, Anmol ThaparEfficient Inference LLM Inference Acceleration
Imperial College London
Efficient Personalized Reranking with Semi-Autoregressive Generation and Online Knowledge Distillation
Mar 7, 2026Kai Cheng, Hao Wang, Wei Guo +4Online Knowledge Distillation Recommender Systems
University of Science and Technology of China, Hefei, China · Huawei, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China +1
FlashEvaluator: Expanding Search Space with Parallel Sequence-Level Evaluation
Mar 3, 2026Chao Feng, Yuanhao Pu, Chenghao Zhang +5Efficient Inference Recommender Systems
Kuaishou Technology, Beijing, China
Jump Like A Squirrel: Optimized Execution Step Order for Anytime Random Forest Inference
Mar 2, 2026Daniel Biebert, Christian Hakert, Kay Heider +3Random Forest Efficient Inference
TU Dortmund University Dortmund, Germany
Lifted Relational Probabilistic Inference via Implicit Learning
Feb 16, 2026Luise Ge, Brendan Juba, Kris Nilsson +1Probabilistic Logic Efficient Inference
Computer Science & Engineering, Washington University in St. Louis, St. Louis, Missouri, USA
Warm-Starting Iterative Gaussian Processes for Faster Sequential Inference
Nov 20, 2025Alan Yufei Dong, Jihao Andreas Lin, José Miguel Hernández-LobatoEfficient Inference Gaussian Processes
Department of Engineering, University of Cambridge
LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling
Oct 21, 2025Yang Liu, Yunjiang Jiang, Ayush Agarwal +3Recommender Systems Efficient Inference
Meta
Predictive Feature Caching for Training-free Acceleration of Molecular Geometry Generation
Oct 6, 2025Johanna Sommer, John Rachwan, Nils Fleischmann +2Flow Matching Efficient Inference
Pruna AI · Technical University of Munich, Germany
GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
Oct 1, 2025Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai +2KV-Cache Compression Efficient Inference
Salesforce AI Research · University of California, Los Angeles
Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI
Jan 16, 2025Wenlong Ji, Lihua Lei, Tijana ZrnicEfficient Inference Prediction-Powered Inference
Stanford University
A polynomial time algebraic solution to exact marginal inference in Markov Random Field models
Sep 25, 2017Ikhlef BecharEfficient Inference Markov Random Fields
Aix Marseille Univ, Universit´e de Toulon, CNRS, ENSAM, LSIS La Garde 83957, France

Apr 2, 2026J/K move · Enter open · S save

Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference

Dimitrios Danopoulos, Enrico Lupi, Michael Kagan, Maurizio Pierini

European Organization for Nuclear Research (CERN), Geneva, Switzerland · SLAC National Accelerator Laboratory, Menlo Park, CA, USA

Efficient Transformer Inference Efficient Inference Softmax Attention AI Accelerator Inference

Open paper AskPDF arXiv

Softmax can become a computational bottleneck in the Transformer model's Multi-Head Attention (MHA) block, particularly in small models under low-precision inference, where exponentiation and normalization incur significant overhead. As such, we suggest using Head-Calibrated Clipped-Linear Softmax (HCCS), a bounded, monotone surrogate to the exponential softmax function, which uses a clipped linear mapping of the max centered attention logits. This approximation produces a stable probability distribution, maintains the ordering of the original logits and has non-negative values. HCCS differs from previous softmax surrogates as it includes a set of lightweight calibration parameters that are optimized offline based on a representative dataset and calibrated for each individual attention head to preserve the statistical properties of the individual heads. We describe a hardware-motivated implementation of HCCS for high-throughput scenarios targeting the AMD Versal AI Engines. The current reference implementations from AMD for this platform rely upon either bfloat16 arithmetic or LUTs to perform the exponential operation, which might limit the throughput of the platform and fail to utilize the high-throughput integer vector processing units of the AI Engine. In contrast, HCCS provides a natural mapping to the AI Engines' int8 multiply accumulate (MAC) units. To the best of our knowledge, this is the first int8 optimized softmax surrogate for AMD AI engines that significantly exceeds the speed performance of other reference implementations while maintaining competitive task accuracy on small or heavily quantized MHA workloads after quantization-aware retraining.