Overview

Efficient inference studies how to get useful model outputs with less computation, memory, time, or energy, which matters for large-scale serving and devices with limited resources. Approaches include quantization and distillation, caching and reuse, sparse or low-rank computation, early exits, amortized inference, and adaptive allocation of test-time compute. Systems work complements these methods with batching, scheduling, compilation, and hardware-aware execution, while probabilistic methods seek faster posterior or marginal estimates without losing calibration or exactness. The central tension is reducing cost without undermining accuracy, uncertainty estimates, or robustness, and results can depend strongly on the workload and hardware used for evaluation.

Latest papers

CardsList

118 papers14 papers in the last four weeks, down 22% on the four weeks before. 0.1% of all new papers.

  1. Speed-Variation Distillation for Generative Modeling

    Nov 13, 2026Sukit Seripanitkarn, Phonphrm Thawatdamrongkit, Supasorn SuwajanakornImage GenerationEfficient Inference

  2. How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?

    Nov 8, 2026Cong Yu, Ziting Wang, Songlin Jiang +5Training-Inference MismatchOn-Policy Distillation

  3. BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

    Oct 31, 2026Hangyu Zheng, Zhihao Shu, Kunxiong Zhu +4On-Device Language Model InferenceLLM Inference Acceleration

  4. Gaussian Distillation of Flow Policies

    Oct 29, 2026Youssef Saied, Ziyad Sheebaelhamd, Claire Vernade +1Robot Policy LearningFlow-Based Policy Learning

  5. Frugal Handwritten Text Recognition for Sub-Saharan Civil-Status Registers: A Smartphone-to-Records Pipeline with CPU-Only Inference

    Oct 27, 2026Ousmane Hamadou, Jean Louis Kedieng Ebongue FendjiOptical Character RecognitionDocument Image Understanding

  6. Training-Free Inference Acceleration for Video Large Language Models: Methods, Trade-offs, and Evaluation

    Oct 26, 2026Yuqi Li, Chuanyue Yan, Neil Rodrigues +8Efficient Inference

  7. WorldPager: Zero-Stall Tiered Memory for Long-Horizon World Models on a Single Consumer GPU

    Oct 26, 2026Bingnan Liu, Ruini XueKV-Cache OffloadingInteractive World Models

  8. Inference Latency-Aware Tsetlin Machine Training

    Oct 26, 2026Yefan Zeng, Shengyu Duan, Rishad Shafik +1Efficient InferenceHyperparameter Optimization

  9. GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

    Oct 26, 2026Ye Li, Haoran Ding, Huanan Liu +10Efficient InferenceGUI Agents

  10. HIFI: A Highly Efficient Framework for Statistical Inference with Foundation Models

    Oct 25, 2026Po-Wei Chen, Shan Yu, Chunlin LiEfficient InferenceUncertainty Quantification

  11. Understanding Target Probability Reuse in Speculative Decoding

    Oct 21, 2026Jungseob Lee, Giyoon Jang, Chailin Nam +6Speculative DecodingParallel Speculative Decoding

  12. Learning Goal-Conditioned Subgoals through Frozen Hierarchical World Models

    Oct 19, 2026Xiang Kaibo, Liu QingfengHierarchical World ModelsWorld Model-Based Planning

  13. Replay Pruning and Few-Bit Integer Inference for Hardware-Efficient Speed-Sign Recognition

    Oct 16, 2026Se Un Park, Kyungkeon Chung, Huigeol Kim +1Channel PruningEfficient Inference

  14. FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving

    Oct 13, 2026Yunfei Feng, Jingkai He, Shiqi Liu +3Efficient InferenceMixture-of-Experts Inference

  15. LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild

    Oct 8, 2026Linkai Liu, Yuntian Zhang, Zhenshan Bing +5Robot NavigationLatent World Models

  16. RealtimeWAM: How Fast Can I Run My World Action Model?

    Oct 7, 2026Huanan Liu, Ye Li, Kangye Ji +8Efficient Inference for World Action ModelsWorld Action Models

  17. Finite-Sample Distribution Theory and Efficient Large-Scale Inference for Online Quantile Regression

    Oct 5, 2026Ziyang Wei, Jiaqi Li, Lan Wang +1Quantile RegressionEfficient Inference

  18. SFlexRCA: Lightweight, Scalable, and Flexible Root Cause Analysis for IIoT Edge Systems

    Oct 4, 2026Amr M. Zaki, Farhoud Jafari, Honggeun Ji +2Industrial Internet of ThingsRoot Cause Analysis

  19. Contextual Flow Matching: Adaptive Step Selection in Flow Models for Efficient Visual Generation

    Oct 2, 2026Divya Jyoti Bajpai, Arun Verma, Manjesh Kumar HanawalEfficient InferenceFlow Matching

  20. High-Dimensional Simulation-Based Inference in Latent Spaces

    Sep 29, 2026Lars Kühmichel, Stefan T. Radev, Bhanu Prasanna Koppolu +3Simulation-Based InferenceNeural Posterior Estimation

  21. Accelerator Choice Is Not Enough: AlphaFold2 Inference on Cloud TPUs

    Sep 28, 2026Lorenzo Pazienza, Ihab El BaniAI Accelerator InferenceEfficient Inference

  22. Can Tabular Foundation Models Amortize Statistical Inference?

    Sep 27, 2026Kai Ye, Shijin Gong, Hongyi Zhou +2Amortized InferenceConfidence Region Estimation

  23. Superposed Inference for Hyperdimensional Computing

    Sep 26, 2026Quanling Zhao, Nilesh Prasad Pandey, Ye Tian +1Hyperdimensional ComputingEfficient Inference

  24. Rolling-WAM: World Action Models with Rolling Imagination

    Sep 24, 2026Yinghua Zhou, Junjie Ye, Yiqi Zhao +8Efficient InferenceWorld Models for Robotics

  25. Direct Message Approximation (DMA): A Consistency-Based Framework for Tractable Approximate Inference on Factor Graphs

    Sep 24, 2026Ralf Herbrich, Rainer Schlosser, Jan Lemcke +11Efficient InferenceUncertainty Quantification

  26. Visual Jev: Accurate and Efficient Decisions from Shared Visual Context

    Sep 22, 2026Guanxu Yu, Yuhang YaoVisual Question AnsweringEfficient Inference