Inference-Time Optimization

Follow this topic Ask about this topic

Inference-time optimization uses computation after training to improve a model’s answer or adapt its behavior to a particular input, without necessarily changing its weights. Common approaches include best-of-N and tree-based search for language-model reasoning, latent or trajectory refinement for generative models and policies, and adaptive verification or routing that spends more effort on uncertain cases. These methods are tested on settings such as math and coding benchmarks, image generation, robotics, and inverse problems, while systems work also tunes caches, scheduling, and hardware configurations. A central tension is whether quality gains justify extra latency, memory, and compute: the best strategy often depends on the task, model, and hardware, and more inference effort does not always improve end-to-end results.

Latest papers

97 papers17 papers in the last four weeks, up 240% on the four weeks before. 0.1% of all new papers.

One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference
Oct 26, 2026Aditya Chatterjee, Anand Joseph X M, Sanjay Singh ChauhanEfficient Neural Network Inference Inference-Time Optimization
NVIDIA, Bangalore, India · NVIDIA, Hyderabad, India · NVIDIA, Pune, India
Answering While You Type: Optimizing Time-to-Answer with Streaming Generation
Oct 19, 2026Nikita Dragunov, Matvey Mikhalchuk, Anton RazzhigaevStreaming Inference Language Model Decoding
FusionBrain Lab
CoRS: Fast Neural Flexible Job-Shop Scheduling via Composite Interval State Reuse
Oct 19, 2026Jiang Sutong, Zhao Peng, Wu Xuan +6Flexible Job Shop Scheduling Neural Combinatorial Optimization
College of Computer Science and Technology, Jilin University, China · College of Software, Jilin University, China · College of Computing and Data Science, Nanyang Technological University, Singapore +1
Peak-Memory Schedules You Can Trust: Exact Realization, Certified Optimality, and the Limits of Recomputation for Inference
Oct 16, 2026Ivan Fomin, Ivan Koryabkin, Boris SheludkoInference-Time Optimization Memory-Efficient Inference
Yandex, Moscow, Russia.
Robust Explanatory Analysis for Structured Optimization Networks (REASON) in Learning
Oct 14, 2026Vahid Jebraeeli, Hamid Krim, Keisuke FujiiInference-Time Optimization Neural Network Robustness
ECE Department, North Carolina State University, Raleigh, NC, USA · Graduate School of Informatics, Nagoya University, Nagoya, Japan
Test-Time Compute for Humanoid Behavior Foundation Models
Oct 10, 2026Weiji Xie, Junsong Wu, Zibo Zhou +5Loco-Manipulation Inference-Time Optimization
Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiaotong University · Zhejiang University +1
FlexiFlow: Bandit-based Model Switching in ML Workflows
Oct 5, 2026Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala +6Multi-Armed Bandits Inference-Time Optimization
IIT Delhi India · University of Chicago USA · University of Missouri USA +1
AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization
Oct 5, 2026Yingda Shen, Yao Qian, Yuxuan Hu +7Inference-Time Optimization Audio Diffusion Models
The Chinese University of Hong Kong, Shenzhen · Microsoft Research
InferOpt: Constrained Multi-Objective Search for LLM Inference Configurations
Oct 3, 2026Qi Chen, Yingying Cheng, Zhaoyi Sun +3Inference-Time Optimization LLM Inference Acceleration
Huawei Technologies Co., Ltd.
TreeWalker: Partial Evaluation for Grouped Tree-Ensemble Inference
Oct 2, 2026Durmus Karatay, Richard NewmanEfficient Inference Inference-Time Optimization
Shopify
Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation
Oct 2, 2026Sripathi Sridhar, Gordon Wichern, Yoshiki Masuyama +2Sound Event Detection Inference-Time Optimization
New Jersey Institute of Technology, Newark NJ, USA · Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA
Exact Memory-Time Optimization for Prefix-Cached Language Model Serving
Oct 2, 2026Shivam GuptaLLM Serving Memory-Efficient Inference
Independent Researcher, Delhi, India
RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Models
Sep 30, 2026Chengzhu Bao, Xianglong Yan, Tianao Zhang +3LLM Quantization Overthinking in Language Models
Shanghai Jiao Tong University
Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models
Sep 30, 2026Shokichi Takakura, Akifumi Wachi, Rei Higuchi +2Diffusion Model Alignment Inference-Time Optimization
LY Corporation · The University of Tokyo · RIKEN AIP
Diversity Combining for Multi-Path LLM Reasoning
Sep 30, 2026Guangsheng Yu, Litianyi Zhang, Qin Wang +5Inference-Time Optimization Self-Consistency Decoding
University of Technology Sydney · The University of Sydney · CSIRO +2
Parallel Tempering for Diffusion-Based Combinatorial Optimization
Sep 29, 2026Arman Mielke, Uwe Bauknecht, Thilo Strauss +1Combinatorial Optimization Neural Combinatorial Optimization
ETAS Research · University of Stuttgart · Max Planck Research School for Intelligent Systems (IMPRS-IS) +2
Trident: Unifying Guarded Dispatch and Host Execution for PyTorch Triton Workloads
Sep 29, 2026Jinjie Liu, Xiaoyan Liu, Shuhan Zhang +5Inference-Time Optimization GPU Acceleration
The Chinese University of Hong Kong · Beijing Academy of Artificial Intelligence
ImbalancE: Inference-Time Latent Search Against Degree Imbalance in Link Prediction
Sep 29, 2026Alberto Bernardi, Luca Costabello, Christophe GueretInference-Time Optimization KG Embedding
Accenture Labs, Dublin, Ireland
TabPFN-3.5: Technical Report
Sep 15, 2026Benjamin Jäger, Nick Erickson, Léo Grinsztajn +44Tabular Foundation Models Inference-Time Optimization
PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving
Sep 9, 2026Lin Huang, Yujuan Tan, Weisheng Li +4Retrieval-Augmented Generation Inference-Time Optimization
Chongqing University. No. 55, University Town South Road, Gaoxin District, Chongqing, 401331, China · Inspur Yunzhou Industrial Internet Co., Ltd. No.1036 Langchao Road, Lixia District, Jinan, Shandong, 250101, China. · Guoqi Zhimo (Chongqing) Technology Co., Ltd. 5th Floor, Building B15, Xiantao Data Valley, Yubei District, Chongqing, 401122, China. +4
Inference-Time Nash Alignment
Sep 8, 2026Hadi Hosseini, Debmalya Mandal, Duohan ZhangInference-Time Optimization Preference Alignment
Penn State University, USA · University of Warwick, UK
MomentQuant: an even more minimalist interval method with linear time complexity for time series classification
Sep 4, 2026Johann FaouziTime Series Classification Inference-Time Optimization
Univ Rennes, Ensai, CNRS, CREST - UMR 9194, F-35000 Rennes, France.
Bandits in Prod: Hyperparameter Optimization at Inference Time
Sep 1, 2026Louis Abraham, Tuan-Anh Nguyen, Nicolas DevatineMulti-Armed Bandits Inference-Time Optimization
Tiime Paris, France
On the Instance Hardness as a Decision Criterion in TinyML Systems
Aug 30, 2026Tobiasz Puslecki, Krzysztof WalkowiakEnergy-Efficient ML Inference-Time Optimization
Department of Systems and Computer Networks, Wroclaw University of Science and Technology
Moving the Mean Toward the Known Good, Not Beyond It: What Inference-Time Interventions and Weight Consolidation Buy in Open-Ended Generation
Aug 28, 2026Roberto I. Ono FilhoInference-Time Optimization Self-Training
Independent researcher
Selective Regenerative Decoding: Trajectory-Level Intervention for Inference-Time Reasoning
Aug 25, 2026Sophia Xiao Pu, Yumo Xu, Sailik Sengupta +5Efficient Language Model Reasoning Inference-Time Optimization
University of California, Santa Barbara · Netflix · Amazon Science +1
Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode
Aug 24, 2026Tom PoperszkyEfficient Transformer Inference LLM Inference Acceleration
Independent Research Report
Recirculation
Aug 18, 2026Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer +2LLM Inference Inference-Time Optimization
Google DeepMind · University of Texas, Austin
Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack
Aug 18, 2026Weiwen Xia, Yuxin Cui, E CaoEfficient Transformer Inference Inference-Time Optimization
Intel Corporation
Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos
Aug 17, 2026Mohamed Afham, Christoph Reich, Oliver Hahn +2Video Reasoning Inference-Time Optimization
TU Darmstadt · ELIZA · TU Munich +1

Oct 26, 2026J/K move · Enter open · S save

One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference

Aditya Chatterjee, Anand Joseph X M, Sanjay Singh Chauhan

NVIDIA, Bangalore, India · NVIDIA, Hyderabad, India · NVIDIA, Pune, India

Efficient Neural Network Inference Inference-Time Optimization GPU Kernel Optimization Streaming Inference

Open paper AskPDF arXiv

We address a fundamental question: How far should kernel fusion extend and which boundaries are worth preserving? Unlike the general convention of using one monolith megakernel at lower batch sizes or supporting only a selective model component as megakernel and rest of the model as conventional graph, we show that the best configuration is a composition of multiple megakernels with no overlapping conventional graph component, or a fully conventional graph. We present a framework that selects megakernel regions in a streaming inference model from conventional-graph profiles and standalone kernel benchmarks, without running any candidate design end to end. It simulates each candidate's tile and event graphs to predict time to first output and selects the measured best design in all 13 combinations of model and batch size we tested. Across NH-TTS, Qwen3-TTS and Llama-1B, the best composition ranges from two to four megakernels to fully conventional execution and none is a single megakernel. We introduce four model categories (megakernel-stable/ receding/ advancing/ averse) that capture how a model's best megakernel composition changes with batch size. Using NH-TTS model on H100 NVL GPU, we demonstrate that preserving three specialized megakernels can improve both latency and throughput despite introducing additional kernel boundaries. At batch size 32, this design delivers 21.4% lower mean batch-p95 time to first audio (TTFA) and 28% higher throughput than the evaluated monolith megakernel. Against conventional TensorRT graph execution, it reduces engine-level mean batch TTFA p95 by 4.05x and increases throughput by 19.7%.