Inference-Time Optimization

Ask about this topic

Overview

Inference-time optimization uses computation after training to improve a model’s answer or adapt its behavior to a particular input, without necessarily changing its weights. Common approaches include best-of-N and tree-based search for language-model reasoning, latent or trajectory refinement for generative models and policies, and adaptive verification or routing that spends more effort on uncertain cases. These methods are tested on settings such as math and coding benchmarks, image generation, robotics, and inverse problems, while systems work also tunes caches, scheduling, and hardware configurations. A central tension is whether quality gains justify extra latency, memory, and compute: the best strategy often depends on the task, model, and hardware, and more inference effort does not always improve end-to-end results.

Latest papers

CardsList

97 papers17 papers in the last four weeks, up 240% on the four weeks before. 0.1% of all new papers.

Oct 26, 2026

One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference

We address a fundamental question: How far should kernel fusion extend and which boundaries are worth preserving? Unlike the general convention of using one monolith megakernel at lower batch sizes or supporting only a selective model component as megakernel and rest of the model as conventional graph, we show that the best configuration is a composition of multiple megakernels with no overlapping conventional graph component, or a fully conventional graph. We present a framework that selects megakernel regions in a streaming inference model from conventional-graph profiles and standalone kernel benchmarks, without running any candidate design end to end. It simulates each candidate's tile and event graphs to predict time to first output and selects the measured best design in all 13 combinations of model and batch size we tested. Across NH-TTS, Qwen3-TTS and Llama-1B, the best composition ranges from two to four megakernels to fully conventional execution and none is a single megakernel. We introduce four model categories (megakernel-stable/ receding/ advancing/ averse) that capture how a model's best megakernel composition changes with batch size. Using NH-TTS model on H100 NVL GPU, we demonstrate that preserving three specialized megakernels can improve both latency and throughput despite introducing additional kernel boundaries. At batch size 32, this design delivers 21.4% lower mean batch-p95 time to first audio (TTFA) and 28% higher throughput than the evaluated monolith megakernel. Against conventional TensorRT graph execution, it reduces engine-level mean batch TTFA p95 by 4.05x and increases throughput by 19.7%.

Aditya Chatterjee, Anand Joseph X M, Sanjay Singh Chauhan

NVIDIA, Bangalore, India · NVIDIA, Hyderabad, India · NVIDIA, Pune, India

Efficient Neural Network InferenceInference-Time OptimizationGPU Kernel OptimizationStreaming Inference
AskPDF
Oct 19, 2026

Answering While You Type: Optimizing Time-to-Answer with Streaming Generation

Work on efficient large-language-model inference has concentrated almost entirely on reducing the cost of decoding once a request is complete. The quantity a user experiences, however, is the interval between submitting a request and receiving a usable response, and that interval can be shortened without making decoding itself faster. While a request is being composed it is already partially specified, yet conventional deployments begin no computation until submission. We propose and study streaming generation, a decoding regime in which the model begins producing a response from an incomplete request and revises it as the remaining input arrives, so that only a small residue of the answer remains to be computed once the request is submitted. We evaluate the regime on autoregressive and diffusion language models, characterise the conditions under which it reduces user-visible latency, and analyse the mechanism responsible.

Nikita Dragunov, Matvey Mikhalchuk, Anton Razzhigaev

FusionBrain Lab

Streaming InferenceLanguage Model DecodingLLM Inference AccelerationInference-Time Optimization
AskPDF
Oct 19, 2026

CoRS: Fast Neural Flexible Job-Shop Scheduling via Composite Interval State Reuse

Neural solvers for the Flexible Job-Shop Scheduling Problem (FJSP) typically refresh graph representations and update feasibility after every operation--machine decision. However, only feasibility must remain exact at every step, whereas graph encoding is substantially more expensive and can potentially be refreshed less frequently. We present CoRS, which decouples them: an interval-level graph representation is reused across multiple sequential decisions, while a stepwise feasibility state is updated after every action. This two-timescale design amortizes graph encoding without parallelizing action generation or relaxing scheduling constraints. CoRS further develops a feasibility-preserving composite-interval decoding scheme with dynamic constraint tracking and boundary state synchronization, which reduces representation refresh frequency while maintaining continuous feasible schedule construction. Furthermore, to alleviate representation staleness caused by interval-level state reuse, CoRS introduces enhanced static features as a compensation mechanism. By providing more comprehensive structural information of the scheduling problem, these features improve the model's perception of overall problem characteristics. Extensive experiments on synthetic and multiple public FJSP benchmarks demonstrate that CoRS significantly reduces inference time while maintaining competitive solution quality, achieving favorable speed-quality trade-offs under tight wall-clock budgets and low-latency scenarios.

Jiang Sutong, Zhao Peng, Wu Xuan +6

College of Computer Science and Technology, Jilin University, China · College of Software, Jilin University, China · College of Computing and Data Science, Nanyang Technological University, Singapore +1

Flexible Job Shop SchedulingNeural Combinatorial OptimizationInference-Time Optimization
AskPDF
Oct 16, 2026

Peak-Memory Schedules You Can Trust: Exact Realization, Certified Optimality, and the Limits of Recomputation for Inference

Real-time inference on small devices needs low activation memory without extra latency, which makes operator order the most valuable memory lever: it is free in time. We show that the usual way of deploying a better order, handing it back to the compiler and regenerating code, is unreliable. In PyTorch Inductor the regenerated program misses its predicted peak in 22 of 54 CPU configurations. It is worse than the stock program in 9 of them and in 19 of 50 on a GPU, and on the GPU it also changes numerical results. We instead profile the generated program, model its memory exactly from its own statements and their read/write effects, and permute those statements. We prove that this realizes the modeled peak whenever allocation does not depend on history. It does so to the byte, with bitwise-identical results, in all our CPU runs. On the GPU we characterize why the caching allocator breaks the condition. To know when to stop, we give forward and reverse "trap" lower bounds, each computable with O(n2)O(n^2) minimum cuts. They certify the found order optimal for 94% of held-out wrapper graphs and all 16 microcontroller-class models we study, in under a second. So most inference graphs are already scheduled optimally. Branching networks are the exception, where reordering is free and pays, e.g. 10.0% on DeepLabV3. In compiled training we save up to 7.6%. Beyond the certified optimum we study selective recomputation with per-operator caps. It is strongly NP-hard for every fixed cap even at treewidth two, encodes knapsack exactly on series-parallel graphs, and is solvable exactly on bounded treewidth. It is useless on public tiny models whose peak is a single layer, but on architectures that hold a cheap value for a late reuse it pays: on the public ReDimNet speaker networks, whose order is already optimal, it lowers the peak by 5.2-6.1% at under 0.4% extra MACs, with bitwise-identical outputs.

Ivan Fomin, Ivan Koryabkin, Boris Sheludko

Yandex, Moscow, Russia.

Inference-Time OptimizationMemory-Efficient InferenceCompiler Optimization
AskPDF
Oct 14, 2026

Robust Explanatory Analysis for Structured Optimization Networks (REASON) in Learning

We seek a hierarchical resolution of residual error under a composite learning objective as a principled mechanism for identifying and exploiting additional information, particularly under distribution shift and perturbation. Inspired by the System 1/System 2 paradigm, we formulate this principle as a two-loop mechanism serving a common inference goal. A fast, System 1-like loop resolves the residual addressable by the available evidence; the remaining epistemic residual triggers a medium speed, System 2-like loop that actively acquires additional evidence and measures its informational value through residual reduction. Over nested solution spaces, the medium-loop optimum is provably no worse than the fast-loop optimum. Experiments show that the resulting two-loop model consistently improves upon the fast loop at approximately half the computation of the medium loop, including under unseen corruptions, and, when provided a cleaner observation, outperforms staged classifiers and learning-to-defer methods at equal computational cost.

Vahid Jebraeeli, Hamid Krim, Keisuke Fujii

ECE Department, North Carolina State University, Raleigh, NC, USA · Graduate School of Informatics, Nagoya University, Nagoya, Japan

Inference-Time OptimizationNeural Network RobustnessLearning to OptimizeActive Feature Acquisition
AskPDF
Oct 10, 2026

Test-Time Compute for Humanoid Behavior Foundation Models

Humanoid Behavior Foundation Models (BFMs) can execute impressive motions, but downstream applications such as loco-manipulation demand high whole-body positional accuracy. Existing BFMs suffer from a systematic command-execution gap, where executed motions deviate from desired ones. We show that additional test-time compute can improve execution without retraining the BFM. First, we introduce Test-Time Command Optimization (TTCO), which adapts the input command to the specific BFM to produce an executed trajectory closer to the desired motion. TTCO finds such commands by sampling-based optimization through closed-loop simulations of the BFM and robot dynamics. Across three BFMs on the AMASS dataset, TTCO reduces mean global body-position error by 79-84% in simulation. Second, we analyze test-time compute scaling and derive an approximate compute-optimal strategy that can sustain performance gains as the compute budget increases. Third, TTCO can generate BFM-specific action labels from recorded robot trajectories without requiring the original teleoperation body-motion commands. Simulation and real-robot experiments show that these labels are effective for training high-level loco-manipulation policies. This method also supports reusing existing demonstrations across BFMs.

Weiji Xie, Junsong Wu, Zibo Zhou +5

Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiaotong University · Zhejiang University +1

Loco-ManipulationInference-Time OptimizationRobot Foundation ModelsTest-Time ScalingHumanoid Motion Tracking
AskPDF
Oct 5, 2026

FlexiFlow: Bandit-based Model Switching in ML Workflows

Model optimizations help improve inference performance and accuracy of ML workflows. However, relying on a single model to perform inference across all data batches often fails to maximize accuracy and thus overall performance. In many cases, alternate models could perform better on specific subsets of data where a primary model underperforms. Our experiments with real ML workflows indeed show that switching models improves workflow accuracy by up to 23%. Yet, current systems lack the ability to adaptively switch between models based on performance, forcing users to manually test models in sequence. We present FlexiFlow, a dataflow system that dynamically switches between alternate models when the current model exhibits low accuracy. FlexiFlow learns to rank models using a novel multi-armed bandit approach that accounts for model runtimes, probability of passing user-defined assertions, and the computational structure of the ML workflow. We show that the standard Thompson sampling approach is insufficient for switching models in ML workflows. In contrast, our proposed approaches are effective and scales to complex real-world ML workflows. Experiments show that switching models at runtime while reusing intermediate results provides higher accuracy, but also 48% efficiency gain compared to sequential workflow runs.

Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala +6

IIT Delhi India · University of Chicago USA · University of Missouri USA +1

Multi-Armed BanditsInference-Time OptimizationAutomated Machine Learning
AskPDF
Oct 5, 2026

AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization

Generative speech enhancement models can produce cleaner and more natural-sounding speech than conventional discriminative approaches, but may hallucinate by changing speech content or speaker identity, even with transcript conditioning. We present AuraSE, a flow-matching framework that addresses hallucination through complementary modality and inference designs. First, a double-stream-to-single-stream multimodal Diffusion Transformer (MMDiT) allows transcript and acoustic representations to interact while preserving a dedicated pathway for the degraded input. Second, we find that the best decoder configuration, governed by guidance scale, sampling temperature, and step count, varies substantially across utterances. This observation motivates Inference Policy Optimization (IPO), an online, on-policy preference optimization method. IPO generates multiple candidates from the current model under different inference configurations, ranks them with a multi-objective reward, and learns from their relative preferences. AuraSE-IPO ranks first on 11 of 12 metrics across the synthetic test sets and obtains the highest DNSMOS and blind-listening scores among the evaluated systems on the real DNS blind test set. At deployment, it uses a fixed 1010-step ODE decoder without classifier-free guidance (CFG) or per-utterance configuration search.

Yingda Shen, Yao Qian, Yuxuan Hu +7

The Chinese University of Hong Kong, Shenzhen · Microsoft Research

Inference-Time OptimizationAudio Diffusion ModelsSpeech EnhancementPreference Optimization
AskPDF
Oct 3, 2026

InferOpt: Constrained Multi-Objective Search for LLM Inference Configurations

Serving an LLM means setting dozens of inference-time knobs, from per-layer KV retention to per-layer expert counts. Practice sets them with mechanism-specific heuristics that return a single operating point and do not scale to layer-wise search spaces. We recast inference configuration as constrained multi-objective black-box optimization and build InferOpt, a reusable search framework that requires only variable bounds, a deterministic resource cost, and an evaluation hook. InferOpt searches on a frozen sampled proxy set, rejects over-budget candidates before any model call, tightens the budget adaptively, and re-validates Pareto representatives on full-scale dataset. One pipeline covers a 28-dimensional continuous KV space (Qwen2.5-7B) and a 26-dimensional discrete MoE space (DeepSeek-V2-Lite). On KV, post-prefill pruning cuts the 16K cache by 64.4% and TPOT by 22.9--48.5%, and the searched layer-wise budget by InferOpt beats a matched uniform budget by 7.3% and 14.0% of the Full KV reference points. On MoE, a searched top-k schedule by InferOpt removes 43.0% of routed token--expert pairs while staying within 0.59 points of the default, closer than the matched-budget baselines. Against Random Search, NSGA-II, and MOTPE, InferOpt leads on both spaces, taking the best proxy hypervolume and the lowest retention on KV and staying closest to the uncompressed reference at the lowest experts on MoE.

Qi Chen, Yingying Cheng, Zhaoyi Sun +3

Huawei Technologies Co., Ltd.

Inference-Time OptimizationLLM Inference AccelerationBlack-Box OptimizationKV-Cache Compression
AskPDF
Oct 2, 2026

TreeWalker: Partial Evaluation for Grouped Tree-Ensemble Inference

Many prediction tasks evaluate a trained tree ensemble on groups of rows that share feature values: discrete-time survival models expand each patient into GG time steps, click-through-rate models score every item in a search session, and scenario analyses vary a few inputs while holding the rest fixed. Standard inference treats each row independently and ignores this input-side correlation. We present TreeWalker, which instantiates partial evaluation for grouped tree-ensemble inference: constant features are static, varying features dynamic. TreeWalker walks each tree once per group, partitions a row bitmask at varying splits, and skips empty subtrees via an unsplit shortcut. It reads standard LightGBM and XGBoost models; training is unchanged. We prove a structural work decomposition: per-tree work splits into the constant-projected subtree size ∣Tc∣|T_c|, GG leaf writes, and a predicate-mask provisioning cost QQ. For the trace evaluator, per-row work approaches (dv+1)/(d+1)(d_v+1)/(d+1) as G→∞G \to \infty, giving asymptotic speedup 1/(1−ρˉG)1/(1-\barρ_G). On Intel, TreeWalker is ∼3×{\sim}3\times faster than a row-independent traversal at the reference model size (T=500T{=}500, L=8L{=}8; G=16G{=}16 for SUPPORT/FLCHAIN and empirical groups for Expedia) and 7.77.7--9.1×9.1\times faster at G=128G{=}128 on the survival datasets; on Arm, 2.52.5--2.9×2.9\times and 5.55.5--6.0×6.0\times. On a scenario-analysis benchmark it is faster in all 1616 configurations on both machines. For f64 models, outputs match treelite GTIL up to tree-ordering roundoff; for f32 models, TreeWalker's f64 leaf accumulation is closer than native f32 to an exactly rounded reference on every audited row.

Durmus Karatay, Richard Newman

Shopify

Efficient InferenceInference-Time OptimizationGradient-Boosted Decision Trees
AskPDF
Oct 2, 2026

Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation

Spatial semantic segmentation of sound scenes (S5) involves the detection and extraction of target sound events in audio files. However, in typical detection-extraction pipelines, errors in the detection module propagate to the extraction module, degrading overall performance. In this work, we develop a multichannel detection-extraction model and evaluate inference-time algorithms to improve the class detection and target sound extraction in a multi-stage setup. We evaluate two fitness functions: binary cross entropy and mixture consistency; two relabeling strategies: naive and conditional confusion matrix relabeling; and three source estimate evaluation models: a multi-label classifier, a fine-tuned single-source classifier, and an off-the-shelf audio judge. We thoroughly evaluate the impact of these design choices on the DCASE 2025 Task 4 dataset, and validate entropy-based gating of inference-time updates on the held-out evaluation set, laying the groundwork for inference-time updates of S5 systems in real-world scenarios.

Sripathi Sridhar, Gordon Wichern, Yoshiki Masuyama +2

New Jersey Institute of Technology, Newark NJ, USA · Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA

Sound Event DetectionInference-Time Optimization
AskPDF
Oct 2, 2026

Exact Memory-Time Optimization for Prefix-Cached Language Model Serving

Retaining language-model prefix states trades recomputation against storage time. Optimizing each cached block independently can overcount savings: a resident block is usable only when the required preceding prefix is also available. We introduce Prefix-Certificate Retention (PCR), an exact finite-trace formulation for static, grouped, reset-on-access timeouts. Usable-prefix rewards become nodes whose prerequisites are timeout thresholds and preceding hit certificates. The resulting maximum-weight closure reduces to one minimum cut, with graph size linear in the number of block lookups and timeout choices. A breakpoint theorem extends the construction to all nonnegative timeouts without discretization error. We also derive a linear-time-in-grid-size dynamic program for ordered timeouts and bounds that certify the cost of this restriction. Exhaustive small-instance checks and chronological replay of 39,632 public Mooncake requests validate the formulation. On the fixed grid, ordered timeouts attain the unrestricted training optimum in 118 of 120 trace-grouping-price cases. Heterogeneous retention improves several held-out memory-time tradeoffs, but finer training optimization does not uniformly improve transfer. The contribution is a tractable optimization model and an auditable benchmark for retention policies; the experiments measure usable prefix blocks and storage time, not GPU latency.

Shivam Gupta

Independent Researcher, Delhi, India

LLM ServingMemory-Efficient InferenceInference-Time OptimizationLLM Inference Acceleration
AskPDF
Sep 30, 2026

RATIO: Reasoning Analysis and Token-level Inference Optimization for Quantized Reasoning Models

Post-training quantization (PTQ) has become a widely adopted technique for reducing the memory footprint and inference cost of large language models (LLMs). However, recent studies reveal that when applied to reasoning models, PTQ not only degrades reasoning performance but also exacerbates overthinking, leading to longer reasoning trajectories. These issues may offset the efficiency gains expected from lower-precision inference. Existing approaches mainly rely on complex optimization procedures. More recent lightweight inference strategies instead use predefined overthinking markers, limiting their adaptability across quantized models. To address these issues, we propose Reasoning Analysis and Token-level Inference Optimization (RATIO), a framework that identifies model-specific overthinking tokens and assigns each a tailored penalty. RATIO first introduces Quantization-aware Reasoning Behavior Analysis (QRBA) to identify overthinking tokens by analyzing discrepancies between full-precision and quantized models. It then adopts Token-Specific Penalty Determination (TSPD), which leverages full-precision guidance to derive token-specific penalties without additional training. Extensive experiments show that RATIO achieves a better accuracy-efficiency trade-off than existing token-level interventions. Specifically, RATIO achieves up to 9.8 points accuracy improvement and reduces chain-of-thought (CoT) length by up to 51.3% compared with quantized baselines. The code will be available at https://github.com/steven-bao1/RATIO.

Chengzhu Bao, Xianglong Yan, Tianao Zhang +3

Shanghai Jiao Tong University

LLM QuantizationOverthinking in Language ModelsInference-Time OptimizationCoT ReasoningPost-Training Quantization
AskPDF
Sep 30, 2026

Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models

Inference-time alignment of flow and diffusion-based models is critical for achieving flexible generative modeling. Theoretically, Doob's hh-transform provides an elegant solution to this problem, and most existing methods are based on this principle. However, in practice, estimating the optimal guidance derived from Doob's hh-transform at inference time is challenging. To deal with this issue, we regard inference-time alignment as a sequential optimization problem in the space of probability measures and propose a novel framework called Steepest Guidance, based on the principle of maximizing local improvement in the objective. We provide a theoretical analysis of the proposed method and demonstrate its effectiveness through extensive experiments.

Shokichi Takakura, Akifumi Wachi, Rei Higuchi +2

LY Corporation · The University of Tokyo · RIKEN AIP

Diffusion Model AlignmentInference-Time OptimizationFlow-Based Generative ModelingInference-Time Guidance
AskPDF
Sep 30, 2026

Diversity Combining for Multi-Path LLM Reasoning

Multi-path reasoning methods such as self-consistency (SC) sample KK reasoning paths and choose the most frequent answer. However, their gains quickly plateau as KK increases, and locating this saturation point currently requires fitting a scaling model to accuracies measured at several values of KK or stopping online per query. We formalize multi-path LLM reasoning as a diversity combining problem from wireless communications: each path is a noisy channel observation, and the pairwise correlation of path correctness caps the design-effect effective sample size of the vote at a finite ceiling. Generalized least squares (GLS) analysis shows that, for paths with equal gains, the optimal linear combiner of latent embeddings is uniform exactly when every row of the path covariance has the same sum, as under exchangeability, supporting majority vote as the natural default in standard SC while leaving room for weighting or pruning under heterogeneous prompt-template branches. Across 5 models and 12 benchmarks, prompt-template diversity reduces path correlation in 5555 of 6060 model-benchmark cells, with a mean relative reduction of 16%16\% and reductions in every task domain. We derive an Adaptive-K rule that uses a four-path pilot to select K∗K^*, retaining 9696--103%103\% of MV@K=32K{=}32 accuracy across Math, QA, and NLU.

Guangsheng Yu, Litianyi Zhang, Qin Wang +5

University of Technology Sydney · The University of Sydney · CSIRO +2

Inference-Time OptimizationSelf-Consistency DecodingPrompt EngineeringLarge Reasoning Models
AskPDF
Sep 29, 2026

Parallel Tempering for Diffusion-Based Combinatorial Optimization

Discrete diffusion models have emerged as a powerful paradigm for solving combinatorial optimization (CO) problems on graphs by learning to sample high-quality solutions. A common inference-time approach is to generate multiple candidate solutions independently and return the best-performing sample, improving solution quality at the expense of an increase in computational cost. In this work, we introduce PT-Denoise, an inference-time procedure that allows these concurrent denoising trajectories to interact through parallel tempering, without requiring retraining or fine-tuning of the underlying denoiser. Our method assigns a temperature to each diffusion process and allows processes to swap temperatures based on their relative performance. This dynamically reallocates promising, low-energy trajectories to colder, more concentrated sampling regimes while allowing higher-energy states to escape local minima through randomized exploration. Experiments on canonical graph-structured CO problems show that our approach consistently improves the quality of the best solution found, while only adding minimal computational overhead.

Arman Mielke, Uwe Bauknecht, Thilo Strauss +1

ETAS Research · University of Stuttgart · Max Planck Research School for Intelligent Systems (IMPRS-IS) +2

Combinatorial OptimizationNeural Combinatorial OptimizationInference-Time OptimizationDiffusion SamplingParallel Tempering
AskPDF
Sep 29, 2026

Trident: Unifying Guarded Dispatch and Host Execution for PyTorch Triton Workloads

User-written Triton kernels enable high-performance GPU computation within PyTorch, but their end-to-end latency can remain dominated by host-side orchestration, especially when device execution is short. Although torch.compile can generate native host wrappers for captured graphs, each invocation still passes through runtime-managed specialization lookup, guard evaluation, and preparation before reaching the wrapper. We present Trident, a compiler backend that removes this recurring overhead from the specialization cache-hit path. Trident introduces the Specialization Cache Module (SCM), which compiles guarded specialization selection, argument and execution-environment preparation, and host execution for multiple specializations into a single executable module. An invocation enters the SCM once, remains in compiled code when a specialization matches, and returns to Python only when a new specialization must be compiled. Built on Torch-MLIR, Trident lowers guards and host-side orchestration to native code while retaining calls to optimized runtime implementations of supported ATen operators. Our evalu- ation on two LLMs shows that Trident achieves up to a 1.47x speedup in model-level end-to-end latency over eager execution and up to 1.68x over torch.compile.

Jinjie Liu, Xiaoyan Liu, Shuhan Zhang +5

The Chinese University of Hong Kong · Beijing Academy of Artificial Intelligence

Inference-Time OptimizationGPU AccelerationLLM Inference Acceleration
AskPDF
Sep 29, 2026

ImbalancE: Inference-Time Latent Search Against Degree Imbalance in Link Prediction

Knowledge Graph Embedding models have been extensively used to learn representations of entities and relations in Knowledge Graphs for predicting missing links. However, the quality of the learned representations varies a lot across different areas of the graph. If previous research has loosely linked the problem to relation types or degree bias, we show that it is more widespread and it correlates with the degree imbalance of the entities in test triples. In particular, the prediction of a target entity that has a degree much smaller than the degree of the anchor entity is extremely problematic. This is critical in recommender systems and other use cases, where these triples represent important corner cases. To address this issue, we propose an inference-time latent search optimization method capable of significantly improving model predictions on the most imbalanced triples. Built on top of a pre-trained model, it explores the embedding space at evaluation time, blending known and out-of-band information to mitigate the degree imbalance bias. We show the value of our approach on imbalanced triples from common benchmark datasets, where we outperform conventional methods, opening the door to the successful adoption of Knowledge Graph Embedding models on these critical corner cases.

Alberto Bernardi, Luca Costabello, Christophe Gueret

Accenture Labs, Dublin, Ireland

Inference-Time OptimizationKG EmbeddingKG Completion
AskPDF
Sep 15, 2026

TabPFN-3.5: Technical Report

We introduce TabPFN-3.5, our new flagship Tabular Foundation Model. It significantly outperforms its predecessor, TabPFN-3, and all existing baselines across a broad range of tabular problems. TabPFN-3.5 sets a new state of the art on standard tabular prediction in TabArena, and extends it to the data practitioners encounter in practice: non-i.i.d. data with temporal or grouped splits, tables with strings, text and images, high-cardinality categorical features, and wide tables with many features. These gains carry over to our task-specific harnesses: state of the art on relational data and stronger time-series forecasting. For faster inference, our variant TabPFN-3.5-Fast runs up to 3x faster than TabPFN-3 while keeping most of the accuracy gains. In addition, we upgrade TabPFN-3.5-Plus, expanding our multimodal capabilities with advanced text and date handling alongside proprietary inference optimizations. Finally, we release a new version of our Thinking mode, TabPFN-3.5-Thinking, which scales inference-time computation to push the state of the art further. It benefits from our stronger base model and from inference-time improvements that make it up to 12x faster than TabPFN-3-Thinking.

Benjamin Jäger, Nick Erickson, Léo Grinsztajn +44

Tabular Foundation ModelsInference-Time OptimizationTabular ICLTabular Prediction
AskPDF
Sep 9, 2026

PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving

We present the PACE, a framework for retrieval-augmented dialogue serving that formalizes Perceived Time-to-First-Response (PTFR) as a QoE objective and minimizes it under quality/cost constraints. Unlike prior work on cascaded routing, semantic caching, or adaptive retrieval, PACE jointly controls which answer source composes the response and what fills the waiting window. Deployed on a humanoid-robot sales service, it combines three mechanisms: a load-adaptive cascading router, a joint path-filler controller, and volatility-aware cache admission. On 75k CarQA requests, the cascade halves pure-LLM PTFR at P95 (0.29 vs 0.53s at c16). The adaptive controller reaches 0.41s P95, outperforming RAG by 2.4 times at high load with equal quality. The filler controller cuts calls by 94% with zero conflict. Volatility-aware admission reduces stale answers from 86% to 0%. A gating rule ensures the controller never worse than the baseline, with exposure bounded by one hold period. This is the first quantification of filler-answer conflict risk in deployed services.

Lin Huang, Yujuan Tan, Weisheng Li +4

Chongqing University. No. 55, University Town South Road, Gaoxin District, Chongqing, 401331, China · Inspur Yunzhou Industrial Internet Co., Ltd. No.1036 Langchao Road, Lixia District, Jinan, Shandong, 250101, China. · Guoqi Zhimo (Chongqing) Technology Co., Ltd. 5th Floor, Building B15, Xiantao Data Valley, Yubei District, Chongqing, 401122, China. +4

Retrieval-Augmented GenerationInference-Time OptimizationLLM RoutingConversational AgentsSemantic Caching
AskPDF
Sep 8, 2026

Inference-Time Nash Alignment

Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to the model parameters which are not provided by many state-of-the art models. Inference-time alignment offers a cost-effective alternative without updating model parameters. However, existing inference-time methods rely on a scalar reward model derived under a Bradley-Terry assumption, which cannot represent general preferences. Following recent work on fine-tuning with generalized preferences, in this work, we initiate the study of inference-time alignment under general preferences. We formulate the problem as obtaining a Nash equilibrium of a two-player zero-sum game between policies. We propose two algorithms: Best-of-Nash (BoN) and Nash Mirror Descent (NMD). We prove that both algorithms achieve a duality gap that matches the problem lower bound. Empirically, we implement the two methods on three datasets, which shows that our methods substantially outperform the base policy, converging to the performance of the fine-tuned models. Moreover, our results show that NMD remains robust across the regularization parameter.

Hadi Hosseini, Debmalya Mandal, Duohan Zhang

Penn State University, USA · University of Warwick, UK

Inference-Time OptimizationPreference AlignmentInference-Time Language Model AlignmentNash Equilibrium Computation
AskPDF
Sep 4, 2026

MomentQuant: an even more minimalist interval method with linear time complexity for time series classification

Time series data is very common in many real-world applications and in numerous domains, with increasing interest for automated information extraction using machine learning. One of these subfields is time series classification, which consists in assigning a label to each new, unseen time series. Many algorithms have been developed over the past decades, with the trade-off between predictive performance and computational cost being consistently discussed. Quant, an interval-based algorithm extracting quantiles from recursive, fixed, dyadic intervals, was shown to achieve high accuracy, while being very fast. We propose two changes to make this algorithm even faster. The first one is a better optimized implementation of the exact same algorithm. The second one is to derive approximate quantiles, using the Cornish-Fisher expansion, instead of exact quantiles. This change removes the necessity to sort the time series, leading to a smaller computational complexity. We call this novel algorithm MomentQuant. We provide evidence that our implementation of Quant is faster than the original one, and that MomentQuant is even faster than our implementation of Quant, at the cost of a tiny decrease in predictive performance. These improvements are especially relevant for real-life applications, where inference is performed much more often than training.

Johann Faouzi

Univ Rennes, Ensai, CNRS, CREST - UMR 9194, F-35000 Rennes, France.

Time Series ClassificationInference-Time Optimization
AskPDF
Sep 1, 2026

Bandits in Prod: Hyperparameter Optimization at Inference Time

Many production systems can assess a configuration only by using it on live requests and observing noisy feedback. Modern agentic systems are a prominent example, with inference-time choices such as model selection, retrieval depth, prompting strategy, and decoding temperature, yet often with no representative validation data. We formalize this setting as Online Hyperparameter Optimization (OHPO) and cast it as an infinitely many-armed bandit over mixed and conditional search spaces. We introduce IMABO, a general framework that combines any bandit policy for choosing among already sampled configurations with any oracle for proposing new ones. We instantiate it with IMOSS, a restart-free anytime policy whose active set grows as tβt^β, and prove an expected cumulative quantile-regret bound of O(pρ−1/β+T(1+β)/2)O(p_ρ^{-1/β} + T^{(1+β)/2}), where β∈(0,1)β\in(0,1) controls active-set growth and pρp_ρ lower-bounds the probability that a proposed configuration falls in the top-ρρ fraction of the search space. We combine IMOSS with three practical oracles: a Tree-structured Parzen Estimator, an incumbent-mutation oracle driven by a per-coordinate bandit, and a pretrained tabular foundation model, all three improving over the uniform random oracle baseline. IMABO outperforms all baselines in terms of regret across diverse OHPO settings, from tuning classical machine-learning models to configuring LLM-based agents. Our implementation is available at https://github.com/Tiime-Software/IMABO.

Louis Abraham, Tuan-Anh Nguyen, Nicolas Devatine

Tiime Paris, France

Multi-Armed BanditsInference-Time OptimizationHyperparameter OptimizationBandit Learning
AskPDF
Aug 30, 2026

On the Instance Hardness as a Decision Criterion in TinyML Systems

TinyML includes the implementation of machine learning on devices with limited memory and computing resources. With the development of technology, AI systems continue to scale in terms of size and computational requirements. This forces researchers to adapt methods to be environmentally sustainable by designing techniques for reducing computational costs and energy consumption in inferring AI models, even in small devices. In this work, we present preliminary findings on a novel application of the tree depth prune instance hardness method to the TinyML system. The results indicate that threshold control can change energy consumption with limited classification quality changes. This method allows us to adjust classification accuracy, thereby influencing computational complexity and energy consumption for inference. We present a work in progress with initial results as a proof of concept.

Tobiasz Puslecki, Krzysztof Walkowiak

Department of Systems and Computer Networks, Wroclaw University of Science and Technology

Energy-Efficient MLInference-Time OptimizationTiny ML
AskPDF
Aug 28, 2026

Moving the Mean Toward the Known Good, Not Beyond It: What Inference-Time Interventions and Weight Consolidation Buy in Open-Ended Generation

What does a generation loop gain from learning on its own verified successes? In cycles of generate, verify, select and LoRA-consolidate on online bin packing, training on value-filtered candidates shifts what the model writes on held-out variants toward value (-1.7 points of excess, p=0.008; -3.1 against a random-consolidation control, p=0.004) while the best observed candidate converges to the classic heuristic's level and no further. A confirmation battery replicates the whole procedure three times, with fresh seeds and a never-consulted held-out set read exactly once: the mean was nearly identical in all three lineages (-2.0, -1.8, -1.9), and after aggregating within held-out variant all seven evaluable variants favored consolidation (p=0.008). The best observed candidate moved to the classic heuristic's level, exactly (0.021028, the lowest variant's classic level, in all three lineages and both arms); attract never went beyond its variant's classic level, while the random control did in 7 of 24 lineage-variant cells. A matched SFT-only control shows the supervised anchor, not repulsion from bad candidates, does the concentrating (96% of candidates land exactly at the classic heuristic's level). On far families the tails cut both ways: consolidation lowers the per-candidate rate of better-than-classic candidates (10% to 3.9%) while its larger production yields more such candidates absolutely (5 against 1, on few events). As motivation we report the inference-time ledger that led here: a model-written schematic recap buys judged document integration and nothing buys development; a verifier written into the stream is imitated, 16.4 fabricated verdict lines per notebook. In this arrangement (an 8-bit model in MLX, five cycles), mean quality among valid candidates can be bought and replicated; under attraction the observed best goes to the classic and, so far, never beyond it.

Roberto I. Ono Filho

Independent researcher

Inference-Time OptimizationSelf-TrainingInference-Time Intervention
AskPDF
Aug 25, 2026

Selective Regenerative Decoding: Trajectory-Level Intervention for Inference-Time Reasoning

Inference-time decoding methods improve LLM reasoning by exploring multiple candidate trajectories, yet treat each trajectory as atomic: either retaining it whole or discarding it irreversibly. This wastes computation on partially promising candidates whose high-quality prefixes are abandoned alongside degraded suffixes. We introduce Selective Regenerative Decoding (SRD), which routes each candidate to discard, keep, or refine only the degraded portion of the suffix while preserving the useful prefix of borderline candidates, without requiring a larger target model. Under mild assumptions, SRD achieves a provable 1.28-to-1.36-fold gain in sample efficiency over rejection sampling with strictly higher expected trajectory quality, with the gain growing as the candidate pool grows. Across MATH500, GPQA Diamond, HotpotQA, and AlpacaEval with multiple generation-reward model pairs, SRD matches Best-of-N accuracy with substantially fewer generated tokens and outperforms speculative rejection in low-compute regimes. By enabling segment-level intervention rather than whole-trajectory selection, SRD opens a previously underexplored region of the accuracy-compute tradeoff for inference-time reasoning.

Sophia Xiao Pu, Yumo Xu, Sailik Sengupta +5

University of California, Santa Barbara · Netflix · Amazon Science +1

Efficient Language Model ReasoningInference-Time OptimizationInference-Time Search
AskPDF
Aug 24, 2026

Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

Single-token autoregressive decode on CPUs is bound by memory bandwidth, not arithmetic: a modern CPU sustains roughly 1 TFLOP/s of compute but only about 50 GB/s from main memory, and each generated token must stream every active weight once. This report argues that the most effective response is to co-design the model architecture and the inference runtime together. It presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures whose inter-layer dependency graphs are constructed to permit a vertical, stage-major execution schedule. cflow stores weights as L2-sized tiles in compute-consumption order, reads only the top-k experts of each mixture-of-experts layer, fuses projections, and executes a delay-aware schedule from per-model dependency parameters. Across five architectures trained on TinyStories, one (arch2_4_combined) achieves a 2.00x reduction in critical-path weight bandwidth (9.00 to 4.50 MB/token) within 0.24 perplexity of the best candidate, and the tile layout incurs 7.29x fewer L1-data read misses than a row-major baseline. On a 30.9-billion-parameter pipeline-native MoE, cflow decodes at 5.94 tokens/s (tok/s) on a 32-vCPU Ice Lake server, ahead of llama.cpp (4.75) and the vLLM CPU backend (1.65) on comparably sized dense models. Realizing the expert-delay window as asynchronous I/O overlap on a disk-resident expert tier yields a further net win of up to 1.68x, matching the overlap model within 1%. Measurement refutes one of the eight design claims and leaves a second inconclusive; both are reported in full, with the conditions under which they would hold.

Tom Poperszky

Independent Research Report

Efficient Transformer InferenceLLM Inference AccelerationAutoregressive DecodingInference-Time OptimizationTransformerMixture-of-Experts Inference
AskPDF
Aug 18, 2026

Recirculation

We describe an inference-time architectural enhancement for off-the-shelf foundation models that systematically reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation---which is better reserved for complex inferences rather than basic state tracking---as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves surprising gains on the Gemma3 family, including a consistent reduction in perplexity on a suite of datasets, a 21% increase in relative accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.

Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer +2

Google DeepMind · University of Texas, Austin

LLM InferenceInference-Time OptimizationEfficient Language Model Inference
AskPDF
Aug 18, 2026

Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack

Small NLP models, especially BERT-family encoders, remain important in industrial workloads such as classification, ranking, and retrieval even in the era of large language models. On server CPUs, INT8 quantization offers an attractive latency-throughput-cost trade-off, but users increasingly expect such acceleration to be available directly in the native PyTorch stack. We integrate SmoothQuant into TorchAO and optimize the resulting inference path for Intel Xeon CPUs through graph-level fusion in TorchInductor and efficient INT8 GEMM kernel selection across oneDNN-, AVX512_VNNI-, and AMX-based implementations. Across BERT, DistilBERT, and XLM-RoBERTa benchmarks, the approach delivers up to 5.8x end-to-end throughput speedup with negligible---and in some cases no measurable---accuracy loss relative to the FP32 baseline. We also validated our work by detailed performance analysis with roofline models. The implementation has been upstreamed to PyTorch and TorchAO, enabling out-of-the-box deployment with native PyTorch tooling

Weiwen Xia, Yuxin Cui, E Cao

Intel Corporation

Efficient Transformer InferenceInference-Time OptimizationPost-Training Quantization
AskPDF
Aug 17, 2026

Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos

Procedure planning seeks to estimate a sequence of actions to transition from an observed initial state to a given goal state. Current procedure planning approaches directly predict action sequences from latent representations using feed-forward neural networks or diffusion-based inference. These paradigms treat every action as plausible, lacking the ability to enforce task-specific logical constraints that render certain actions irrelevant or not plausible. We propose CEFITO, a procedure planning approach that learns a predictor to express an action-conditioned representation space. Based on this representation space, we formulate procedure planning as a task-constrained optimization problem. Unlike prior methods, CEFITO explicitly reasons over the action space by omitting irrelevant actions during inference-time planning. This reformulation enables effective procedure planning and achieves state-of-the-art accuracy on two established procedure planning benchmarks.

Mohamed Afham, Christoph Reich, Oliver Hahn +2

TU Darmstadt · ELIZA · TU Munich +1

Video ReasoningInference-Time OptimizationEnergy-Based Models
AskPDF