Inference-Time Optimization

Follow this topic Ask about this topic

Inference-time optimization uses computation after training to improve a model’s answer or adapt its behavior to a particular input, without necessarily changing its weights. Common approaches include best-of-N and tree-based search for language-model reasoning, latent or trajectory refinement for generative models and policies, and adaptive verification or routing that spends more effort on uncertain cases. These methods are tested on settings such as math and coding benchmarks, image generation, robotics, and inverse problems, while systems work also tunes caches, scheduling, and hardware configurations. A central tension is whether quality gains justify extra latency, memory, and compute: the best strategy often depends on the task, model, and hardware, and more inference effort does not always improve end-to-end results.

Latest papers

97 papers17 papers in the last four weeks, up 240% on the four weeks before. 0.1% of all new papers.

Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation
Mar 10, 2026Jingtao Wang, Yucong Wang, Jun Ding +2LLM Inference Inference-Time Adaptation of Language Models
Meakins-Christe Laboratories, Research Institute of McGill University Health Centre, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · Department of Medicine, Division of Experimental Medicine, McGill University, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · The College of Computer Science and Technology, Zhejiang Gongshang University, Hangzhou, China +4
Jump Like A Squirrel: Optimized Execution Step Order for Anytime Random Forest Inference
Mar 2, 2026Daniel Biebert, Christian Hakert, Kay Heider +3Random Forest Efficient Inference
TU Dortmund University Dortmund, Germany
Inference-time optimization for experiment-grounded protein ensemble generation
Feb 27, 2026Advaith Maddipatla, Anar Rzayev, Marco Pegoraro +5Diffusion Model Guidance Inference-Time Optimization
Institute of Science and Technology, Austria · University of Zurich, Switzerland · Tel Hai – University of Kiryat Shmona in the Galilee, Israel +4
Invariant Transformation and Resampling based Epistemic-Uncertainty Reduction
Feb 26, 2026Sha HuInference-Time Optimization Test-Time Augmentation
Lund Research Center, Huawei Technologies Sweden AB, Sweden.
Framework of Thoughts: A Foundation Framework for Dynamic and Optimized Reasoning based on Chains, Trees, and Graphs
Feb 18, 2026Felix Fricke, Simon Malberg, Georg GrohInference-Time Optimization Tree of Thoughts
School of Computation, Information and Technology Technical University of Munich
DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation
Jan 29, 2026Haozhe Xie, Beichen Wen, Jiarui Zheng +4Inference-Time Optimization Efficient VLM Inference
Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputation
Jan 15, 2025Geon-Woo Kim, Donghyun Kim, Jeongyoon Moon +5Efficient Neural Network Inference Graph Neural Networks
The University of Texas at Austin · Georgia Institute of Technology

Mar 10, 2026J/K move · Enter open · S save

Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation

Jingtao Wang, Yucong Wang, Jun Ding, Rui Cai+1

Meakins-Christe Laboratories, Research Institute of McGill University Health Centre, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · Department of Medicine, Division of Experimental Medicine, McGill University, 1001 Decarie Blvd, Montreal, H4A 3J1, Quebec, Canada · The College of Computer Science and Technology, Zhejiang Gongshang University, Hangzhou, China+4

LLM Inference Inference-Time Adaptation of Language Models Inference-Time Optimization

Open paper AskPDF arXiv

Large language models (LLMs) achieve remarkable performance, yet further gains often require costly training. This has motivated growing interest in post-training techniques-especially training-free approaches that improve models at inference time without updating weights. Most training-free methods treat the model as a black box and improve outputs via input/output-level interventions, such as prompt design and test-time scaling through repeated sampling, reranking/verification, or search. In contrast, they rarely offer a plug-and-play mechanism to intervene in a model's internal computation. We propose ARACH(Attention Reallocation via an Adaptive Context Hub), a training-free inference-time plug-in that augments LLMs with an adaptive context hub to aggregate context and reallocate attention. Extensive experiments across multiple language modeling tasks show consistent improvements with modest inference overhead and no parameter updates. Attention analyses further suggest that ARACH mitigates the attention sink phenomenon. These results indicate that engineering a model's internal computation offers a distinct inference-time strategy, fundamentally different from both prompt-based test-time methods and training-based post-training approaches.