Efficient Inference

Ask about this topic

Overview

Efficient inference studies how to get useful model outputs with less computation, memory, time, or energy, which matters for large-scale serving and devices with limited resources. Approaches include quantization and distillation, caching and reuse, sparse or low-rank computation, early exits, amortized inference, and adaptive allocation of test-time compute. Systems work complements these methods with batching, scheduling, compilation, and hardware-aware execution, while probabilistic methods seek faster posterior or marginal estimates without losing calibration or exactness. The central tension is reducing cost without undermining accuracy, uncertainty estimates, or robustness, and results can depend strongly on the workload and hardware used for evaluation.

Latest papers

CardsList

107 papers14 papers in the last four weeks, up 17% on the four weeks before. 0.1% of all new papers.

  1. Foundations of Reliable Inference: Reliability-Efficiency Co-Design

    May 11, 2026Jiayi HuangEfficient InferenceUncertainty Quantification

    The Department of EngineeringarXiv:2605.10351v1 King’s College London

  2. PixelFlowCast: Latent-Free Precipitation Nowcasting via Pixel Mean Flows

    May 11, 2026Yufeng Zhu, Chunlei Shi, Yongchao Feng +1Precipitation ForecastingEfficient Inference

    Department of Automation, Southeast University, Nanjing 210096, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China

  3. TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling

    May 10, 2026Hongyaoxing Gu, Xinzhe Chen, Lijuan Hu +1Efficient InferenceMixture-of-Experts Inference

    Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences

  4. Latency Analysis and Optimization of Alpamayo 1 via Efficient Trajectory Generation

    May 9, 2026Yunseong Jeon, Namcheol Lee, Yoonsu Lee +4Efficient InferenceEnd-to-End Autonomous Driving

    Graduate School of Automobile and Mobility, Kookmin University, Seoul, Republic of Korea · Department of Electrical and Computer Engineering, Seoul National University, Seoul, Republic of Korea

  5. Exploring and Exploiting Stability in Latent Flow Matching

    May 8, 2026Rania Briq, Michael Kamp, Ohad Fried +2Flow MatchingEfficient Inference

    Forschungszentrum Jülich · Technical University Dortmund · Lamarr Institute +4

  6. StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models

    May 8, 2026Panqi Chen, Yifan Sun, Shikai Fang +2Efficient InferenceStreaming Inference

    College of Information Science and Electronic Engineering, Zhejiang University · School of EECS,Oregon State University

  7. Estimating the expected output of wide random MLPs more efficiently than sampling

    May 6, 2026Wilson Wu, Victor Lecomte, Michael Winer +3Multilayer PerceptronsEfficient Inference

    Alignment Research Center

  8. GraphPI: Efficient Protein Inference with Graph Neural Networks

    May 6, 2026Zheng Ma, Jiazhen Chen, Lei Xin +1Efficient Inference

    Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada, N2L 3G1 · Department of Statistical and Actuarial Science, University of Waterloo, Waterloo, ON, Canada, N2L 3G1 · Bioinformatics Solutions Inc., Waterloo, ON, Canada, N2L 3K8

  9. A Fast Model Counting Algorithm for Two-Variable Logic with Counting and Modulo Counting Quantifiers

    May 5, 2026Shixin Sun, Astrid Klipfel, Ondřej Kuželka +2Efficient Inference

    School of Artificial Intelligence, Jilin University, Changchun 130012, China · CRIL, Université d’Artois, Lens 62307, France · Czech Technical University, Prague 160 00, Czechia +1

  10. Intelligent Elastic Feature Fading: Enabling Model Retrain-Free Feature Efficiency Rollouts at Scale

    May 1, 2026Jieming Di, Xiaoyu Chen, Ying She +21Efficient InferenceCost-Aware Inference

    Meta · USA

  11. Spark Policy Toolkit: Semantic Contracts and Scalable Execution for Policy Learning in Spark

    Apr 27, 2026Zeyu BaiEfficient InferencePolicy Learning

  12. Null-Space Flow Matching for MIMO Channel Estimation in Latency-Constrained Systems

    Apr 23, 2026Junjie Zhao, Guangming Liang, Dongzhu Liu +1Flow MatchingEfficient Inference

    College of Electronic Information and Optical Engineering, Nankai University · School of Computing Science, University of Glasgow · School of Natural and Computing Science, University of Aberdeen

  13. Adaptive Test-Time Compute Allocation with Evolving In-Context Demonstrations

    Apr 22, 2026Bowen Zuo, Dongruo Zhou, Yinglun ZhuEfficient InferenceTest-Time Optimization

    University of California, Riverside · 2Indiana University Bloomington

  14. Fast Bayesian equipment condition monitoring via simulation based inference: applications to heat exchanger health

    Apr 22, 2026Peter Collett, Alexander Johannes Stasik, Simone Casolo +1Simulation-Based InferenceEfficient Inference

    Cognite AS, Oslo, Norway · Department of Data Science, Norwegian University of Life Sciences, Ås, Norway · Department for Mathematics and Cybernetics, SINTEF Digital, Oslo, Norway

  15. Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Trees

    Apr 22, 2026Xueyan Li, Johannes Zenn, Ekaterina Fadeeva +3Language Model DecodingInference-Time Search

    Max Planck Institute for Intelligent Systems · ETH Zurich · AI Center Tübingen +3

  16. Scalable AI Inference: Performance Analysis and Optimization of AI Model Serving

    Apr 22, 2026Hung Cuong Pham, Fatih GedikliEfficient Inference

    Institute of Computer Science University of Applied Sciences Ruhr West Mülheim an der Ruhr, Germany

  17. SAGE: Training-Free Semantic Evidence Composition for Edge-Cloud Inference under Hard Uplink Budgets

    Apr 21, 2026Inhyeok Choi, Hyuncheol ParkEfficient InferenceEdge-Cloud Computing

    School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

  18. EX-FIQA: Leveraging Intermediate Early eXit Representations from Vision Transformers for Face Image Quality Assessment

    Apr 21, 2026Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira +4Efficient Transformer InferenceEfficient Inference

    Fraunhofer Institute for Computer Graphics Research IGD, Darmstadt, Germany · Department of Computer Science, TU Darmstadt, Germany

  19. FlowForge: A Staged Local Rollout Engine for Flow-Field Prediction

    Apr 21, 2026Xiaowen Zhang, Ziming Zhou, Fengnian Zhao +1Efficient InferencePDE Surrogate Modeling

    Shanghai Jiao Tong University · University of Michigan

  20. Streaming Structured Inference with Flash-SemiCRF

    Apr 20, 2026Benjamin K. Johnson, Thomas Goralski, Ayush Semwal +2Efficient InferenceConditional Random Fields

    Department of Epigenetics, Van Andel Institute, 333 Bostwick Ave. NE, Grand Rapids, 49503, MI, USA. · Department of Cell Biology, Van Andel Institute, 333 Bostwick Ave. NE, Grand Rapids, 49503, MI, USA.

  21. FlashFPS: Efficient Farthest Point Sampling for Large-Scale Point Clouds via Pruning and Caching

    Apr 20, 2026Yuzhe Fu, Hancheng Ye, Cong Guo +7Efficient InferenceGPU Acceleration

    Helen · Duke University Durham, North Carolina, USA

  22. KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving

    Apr 17, 2026Yichao Yuan, Mosharaf Chowdhury, Nishil TalatiEfficient InferenceLLM Inference Serving

    University of Illinois, Urbana Champaign Urbana, IL, USA · University of Michigan Ann Arbor, MI, USA

  23. A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits

    Apr 16, 2026Nekane Fernandez, Ivan Valdes, Steven Van Vaerenbergh +2Efficient InferenceEdge Inference

    Nekane Fernandez, Ivan Valdes, Idoia de la Iglesia and Julen Arratibel are with the Dept. of Connected and Distributed Intelligence, Ikerlan Technology Research Centre, Arrasate, Spain · Nekane Fernandez and Steven Van Vaerenbergh are with the Dept. of Mathematics, Statistics and Computation, Universidad de Cantabria, Santander, Spain

  24. Rethinking AI Hardware: A Three-Layer Cognitive Architecture for Autonomous Agents

    Apr 15, 2026Li ChenEdge-Cloud Collaborative InferenceCognitive Architectures for AI Agents

  25. LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deployment

    Apr 12, 2026Gregory MagarshakEfficient InferenceLLM Inference Acceleration

  26. One-Step Score-Based Density Ratio Estimation

    Apr 12, 2026Wei Chen, Qibin Zhao, John Paisley +2Density Ratio EstimationEfficient Inference

    School of Mathematics, South China University of Technology, Guangzhou 510006, China · Tensor Learning Team, RIKEN Center for Advanced Intelligence Project, Tokyo 103-0027, Japan · Department of Electrical Engineering, Columbia University, New York, NY 10027, USA +1

  27. Joint Task Offloading, Inference Optimization and UAV Trajectory Planning for Generative AI Empowered Intelligent Transportation Digital Twin

    Apr 9, 2026Xiaohuan Li, Junchuan Fan, Bingqi Zhang +3Intelligent Transportation SystemsDigital Twins

    Guangxi University Key Laboratory of Intelligent Networking and Scenario System (School of Information and Communication, Guilin University of Electronic Technology), Guilin 541004, China · Research Institute of Highway Ministry of Transport, Beijing 100000, China · School of Automation, Guangdong University of Technology, Guangzhou 510006, China +1

  28. A Tensor-Train Framework for Bayesian Inference in High-Dimensional Systems: Applications to MIMO Detection and Channel Decoding

    Apr 7, 2026Luca Schmid, Dominik Sulz, Shrinivas Chimmalgi +1Bayesian InferenceEfficient Inference

    Communications Engineering Lab (CEL), Karlsruhe Institute of Technology (KIT), Hertzstr. 16, 76187 Karlsruhe, Germany · Technical University of Munich (TUM), Boltzmannstr. 3, 85748 Garching b. München, Germany · Qoherent, Toronto, ON, Canada

  29. FAVE: Flow-based Average Velocity Establishment for Sequential Recommendation

    Apr 6, 2026Ke Shi, Yao Zhang, Feng Guo +4Sequential RecommendationGenerative Recommendation

    University of Electronic Science and Technology of China · State Key Laboratory of Internet Architecture

  30. Domain-Contextualized Inference: A Computable Graph Architecture for Explicit-Domain Reasoning

    Apr 6, 2026Chao Li, Yuru Wang, Chunyi ZhaoEfficient InferenceGraph Reasoning

    Northeast Normal University · University of Otago