Inference-Time Compute Allocation

Overview

This field studies how to distribute limited prediction-time resources—such as reasoning tokens, candidate samples, model calls, search steps, or GPU time—across questions, reasoning paths, and agent actions. Common approaches use difficulty or uncertainty estimates to route tasks, adapt sampling, guide verifier-based search, stop early, or escalate to stronger models; systems work also schedules requests against latency and budget constraints. Evaluations span mathematical reasoning and coding benchmarks as well as retrieval, embodied planning, and multi-tenant inference. A central challenge is that allocation gains depend on whether available signals predict the value of extra computation, and apparent improvements can vanish when costs, selection rules, runtime conditions, and held-out evaluation are controlled.

Latest papers

91 papers13 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

May 30, 2025

TIP-Search: Time-Predictable Inference Scheduling for Market Prediction under Uncertain Load

Real-time market prediction services need correct predictions before a decision deadline; a correct prediction delivered late is not usable. TIP-Search studies time-predictable inference scheduling over fixed market predictors under uncertain load. It filters conformal latency-quantile feasible models, dispatches over finite workers, and uses shielded constrained online experts to trade accuracy, queue pressure, and deadline risk. On the optimized deployable pool, TIP-Search reaches 0.994 raw accuracy and 0.991 timely accuracy. On official TLOB FI-2010 h=10, TIP-Search++ raises timely accuracy from 0.156 to 0.239 and deadline satisfaction from 0.391 to 0.962. In matched h10 profiled systems replay, OCO-ACPO reaches 0.303 timely accuracy and 0.951 deadline satisfaction, with paired gains over RAMSIS/SneakPeek/utility-style comparators of +0.00285+0.00285 timely accuracy (p=0.0118p=0.0118) and +0.0146+0.0146 deadline satisfaction (p=1.5×10−5p=1.5{\times}10^{-5}). SA-OCO-ACPO improves timely/deadline service by 0.188--0.417 over CPO under nonstationary stress. The claim is a systems scheduling result, not a broad LOB classifier leaderboard.