Inference-Time Optimization

Overview

Inference-time optimization uses computation after training to improve a model’s answer or adapt its behavior to a particular input, without necessarily changing its weights. Common approaches include best-of-N and tree-based search for language-model reasoning, latent or trajectory refinement for generative models and policies, and adaptive verification or routing that spends more effort on uncertain cases. These methods are tested on settings such as math and coding benchmarks, image generation, robotics, and inverse problems, while systems work also tunes caches, scheduling, and hardware configurations. A central tension is whether quality gains justify extra latency, memory, and compute: the best strategy often depends on the task, model, and hardware, and more inference effort does not always improve end-to-end results.

Latest papers

CardsList

103 papers8 papers in the last four weeks, down 47% on the four weeks before. 0.1% of all new papers.

  1. Learning Stratified Policies for Inference-Time Objectives

    Nov 17, 2026Mridul Mahajan, Xuezhou Zhang, Aldo PacchianoLLM InferenceInference-Time Optimization

  2. Fast Inference-Time Policy Refinement via Action-Affine World Models

    Nov 11, 2026Run Wang, Yilin Mo, Liang WuInference-Time OptimizationAction-Conditioned World Models

  3. Where Is VLA Inference Time Spent? Bottlenecks and Optimization Interactions Across GPUs

    Nov 4, 2026Muhammad Danish Waseem, Ahmed Ali-Eldin HassanEfficient VLA Model InferenceInference-Time Optimization

  4. From Reconstruction to Imputation: Test-Time Optimization for Missing Time-Series Channels

    Nov 4, 2026Björn Ludwig, Alexander Windmann, René Heesch +1Time Series ImputationLearning with Missing Data

  5. One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference

    Oct 26, 2026Aditya Chatterjee, Anand Joseph X M, Sanjay Singh ChauhanEfficient Neural Network InferenceInference-Time Optimization

  6. Answering While You Type: Optimizing Time-to-Answer with Streaming Generation

    Oct 19, 2026Nikita Dragunov, Matvey Mikhalchuk, Anton RazzhigaevStreaming InferenceLanguage Model Decoding

  7. CoRS: Fast Neural Flexible Job-Shop Scheduling via Composite Interval State Reuse

    Oct 19, 2026Sutong Jiang, Peng Zhao, Xuan Wu +6Flexible Job Shop SchedulingNeural Combinatorial Optimization

  8. Robust Explanatory Analysis for Structured Optimization Networks (REASON) in Learning

    Oct 14, 2026Vahid Jebraeeli, Hamid Krim, Keisuke FujiiInference-Time OptimizationNeural Network Robustness

  9. Test-Time Compute for Humanoid Behavior Foundation Models

    Oct 10, 2026Weiji Xie, Junsong Wu, Zibo Zhou +5Loco-ManipulationInference-Time Optimization

  10. FlexiFlow: Bandit-based Model Switching in ML Workflows

    Oct 5, 2026Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala +6Multi-Armed BanditsInference-Time Optimization

  11. AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization

    Oct 5, 2026Yingda Shen, Yao Qian, Yuxuan Hu +7Inference-Time OptimizationAudio Diffusion Models

  12. Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation

    Oct 2, 2026Sripathi Sridhar, Gordon Wichern, Yoshiki Masuyama +2Sound Event DetectionInference-Time Optimization

  13. Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models

    Sep 30, 2026Shokichi Takakura, Akifumi Wachi, Rei Higuchi +2Diffusion Model AlignmentInference-Time Optimization

  14. Diversity Combining for Multi-Path LLM Reasoning

    Sep 30, 2026Guangsheng Yu, Litianyi Zhang, Qin Wang +5Inference-Time OptimizationSelf-Consistency Decoding

  15. Parallel Tempering for Diffusion-Based Combinatorial Optimization

    Sep 29, 2026Arman Mielke, Uwe Bauknecht, Thilo Strauss +1Combinatorial OptimizationNeural Combinatorial Optimization

  16. Trident: Unifying Guarded Dispatch and Host Execution for PyTorch Triton Workloads

    Sep 29, 2026Jinjie Liu, Xiaoyan Liu, Shuhan Zhang +5Inference-Time OptimizationGPU Acceleration

  17. ImbalancE: Inference-Time Latent Search Against Degree Imbalance in Link Prediction

    Sep 29, 2026Alberto Bernardi, Luca Costabello, Christophe GueretInference-Time OptimizationKG Embedding

  18. TabPFN-3.5: Technical Report

    Sep 15, 2026Benjamin Jäger, Nick Erickson, Léo Grinsztajn +44Tabular Foundation ModelsInference-Time Optimization

  19. PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving

    Sep 9, 2026Lin Huang, Yujuan Tan, Weisheng Li +4Retrieval-Augmented GenerationInference-Time Optimization

  20. Inference-Time Nash Alignment

    Sep 8, 2026Hadi Hosseini, Debmalya Mandal, Duohan ZhangInference-Time OptimizationPreference Alignment

  21. Bandits in Prod: Hyperparameter Optimization at Inference Time

    Sep 1, 2026Louis Abraham, Tuan-Anh Nguyen, Nicolas DevatineMulti-Armed BanditsInference-Time Optimization

  22. On the Instance Hardness as a Decision Criterion in TinyML Systems

    Aug 30, 2026Tobiasz Puslecki, Krzysztof WalkowiakEnergy-Efficient MLInference-Time Optimization