Inference-Time Optimization

Inference-time optimization uses computation after training to improve a model’s answer or adapt its behavior to a particular input, without necessarily changing its weights. Common approaches include best-of-N and tree-based search for language-model reasoning, latent or trajectory refinement for generative models and policies, and adaptive verification or routing that spends more effort on uncertain cases. These methods are tested on settings such as math and coding benchmarks, image generation, robotics, and inverse problems, while systems work also tunes caches, scheduling, and hardware configurations. A central tension is whether quality gains justify extra latency, memory, and compute: the best strategy often depends on the task, model, and hardware, and more inference effort does not always improve end-to-end results.