Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

Adam Fisch; Shubhendu Trivedi; Fantine Huot; William W. Cohen; Michael Kaisers; Mirella Lapata; Kate Larson; Jacob Eisenstein

cs.AIAug 20, 2026

Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

Authors: Adam Fisch, Shubhendu Trivedi, Fantine Huot, William W. Cohen, Michael Kaisers, Mirella Lapata, Kate Larson, Jacob Eisenstein

Organizations: Google DeepMind

Abstract

Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost. Routing requires estimating each specialist's expected return, but this value estimation has a cost. Cheap estimators (e.g., embedding-based predictors) are fast but noisy, while accurate estimators (e.g., fine-tuned models with access to retrieval results or partial reasoning traces) are expensive. We formalize this tradeoff as an instance of Pandora's Box, the classical problem of optimal search with costly inspection. Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that determine, for each specialist and input, whether refining the value estimate is worth its cost. We call the centralized policy Pandora's Router. We extend this to a decentralized setting, Pandora's Bidder, where specialists independently decide whether to invest in self-assessment before accepting an offered price to claim a query. Experiments across three domains---a standard multi-LLM benchmark, retrieval-augmented specialists, and LLMs with variable inference-time reasoning---show that Pandora's Router matches the routing quality of exhaustive estimation, while querying the expensive estimator far less often. In the decentralized setting, value-of-information reasoning improves allocative efficiency when competing estimates are accurate; when competing estimates are noisy, however, it can increase the strategic specialist's utility at the expense of others.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the Pandora framework. The system balances cheap screening and costly reasoning for routing. (1) A concrete prompt $x$ is received. (2) Multiple specialists $m\in\{1,\ldots,M\}$ generate cheap, noisy value estimates $f_{m}(x)$ . (3) The system computes reservation prices that quantify the expected value-of-information for querying a more costly and accurate value estimate $g_{m}(x)$ , e.g., by running the specialist for a few reasoning tokens and checking if it is on the right track. (4) This unified principle supports allocation by centralized routing and decentralized auctions.

Setting	Estimator	MSE	${c_{g}}/{c_{f}}$	Description
MATH	$f$ : KNN	$.154$	$5.8$	$k$ -nearest neighbors on prompt embeddings
	$g$ : SFT-CoT- $20$	$.096$	$5.8$	Fine-tuned transformer on prompt + first $20$ tokens of the model’s chain-of-thought (CoT) reasoning trace
RAG	$f$ : KNN	$.175$	$>7000$	$k$ -nearest neighbors on prompt embeddings
	$g$ : SFT-retrievals	$.109$	$>7000$	Fine-tuned transformer on prompt + retrieval results
EmbedLLM	$f$ : KNN	$.266$	$1.6$	$k$ -nearest neighbors on prompt embeddings
	$g$ : SFT-prompt	$.198$	$1.6$	Fine-tuned transformer on prompt

Table 1: Value estimators across experimental domains. The cheap estimator

f

is always computed; the costly estimator

g

is queried selectively. The mean-squared error (MSE) is measured on a held-out calibration set. The cost ratios

{c_{g}}/{c_{f}}

are discussed in Section 2.2 and derived in Section D.3 .

Method	MATH			RAG			EmbedLLM
Method	Regret	Cost	Total	Regret	Cost	Total	Regret	Cost	Total
$f$ -only	0.117	0.000	0.117	0.150	0.000	0.150	0.393	0.000	0.393
$g$ -only	0.090	0.038	0.128	0.084	0.057	0.141	0.370	1.986	2.356
Top- $2$	0.090	0.038	0.128	0.107	0.038	0.146	0.402	0.036	0.438
Coin Flip	0.107	0.019	0.127	0.122	0.029	0.151	0.398	0.992	1.390
Random- $N_{\mathrm{pr}}$	0.164	0.011	0.175	0.135	0.027	0.162	0.363	0.075	0.438

Table 2: Routing evaluation results averaged across query costs

c_{g}

. Lower is better on all metrics; the lowest average regret + cost measured per setting is bolded . See Section D.4 in the Appendix for a breakdown of results per cost level

c_{g}

, together with paired statistical significance tests.

Panel of Figure 2 — Figure 2: Routing performance on the MATH, RAG, and EmbedLLM domains for varying costs of querying $g$ , with $f=\text{KNN}_{3}$ . The ideal value estimation policy would minimize regret + inspection cost at every cost level. Pandora’s Router achieves near-minimal total cost across the full range of $c_{g}$ (where $c_{m}=c_{g}$ for all $m$ ), querying $g$ frequently when it is cheap and rarely when it is expensive.

Panel of Figure 3 — Figure 3: Results for the posted-price auction across the three experimental domains for varying refinement cost $c_{g}$ . The VoI-based bidding strategy interpolates between querying $g$ when cheap and declining when expensive, while static baselines either over-invest or under-invest in refinement.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Number of queries to $g$ during model routing.

Panel of Figure 5 — Figure 5: Specialist surplus regret and total allocative efficiency regret of the posted-price auction, when bidding against opponents who have access only to $f$ , rather than $g$ as in Figure 3 .

Panel of Figure 6 — Figure 6: Routing performance versus monetary inspection cost, as estimated in Section D.3 .

Cost	f-only	g-only	Top- $2$	Coin Flip	Random- $N_{\mathrm{pr}}$	Margin- $N_{\mathrm{pr}}$	Pandora’s Router
1.5e-03	0.117	0.093	0.093	0.109	0.194	0.093	0.095
2.1e-03	0.117	0.094	0.094	0.109	0.193	0.097	0.096
3.2e-03	0.117	0.096	0.096	0.110	0.198	0.098	0.098
4.6e-03	0.117	0.099	0.099	0.112	0.197	0.099	0.099
6.8e-03	0.117	0.104	0.104	0.114	0.193	0.101	0.101
1.0e-02	0.117	0.110	0.110	0.117	0.199	0.104 *	0.107

Appendix

Table 3: Numerical results on MATH routing. Bold indicates lowest regret + inspection cost, asterisk indicates significance at

p<.05

(paired bootstrap).

Cost	f-only	g-only	Top- $2$	Coin Flip	Random- $N_{\mathrm{pr}}$	Margin- $N_{\mathrm{pr}}$	Pandora’s Router
1.5e-03	0.150	0.088	0.110	0.124	0.129	0.100	0.090
2.1e-03	0.150	0.090	0.112	0.125	0.134	0.103	0.090
3.2e-03	0.150	0.093	0.114	0.127	0.146	0.110	0.092
4.6e-03	0.150	0.098	0.117	0.129	0.155	0.113	0.098
6.8e-03	0.150	0.104	0.121	0.132	0.155	0.119	0.104
1.0e-02	0.150	0.114	0.127	0.137	0.170	0.124	0.110

Appendix

Table 4: Numerical results on RAG routing. Bold indicates lowest regret + inspection cost, asterisk indicates significance at

p<.05

(paired bootstrap).

Cost	f-only	g-only	Top- $2$	Coin Flip	Random- $N_{\mathrm{pr}}$	Margin- $N_{\mathrm{pr}}$	Pandora’s Router
1.0e-05	0.393	0.371	0.402	0.398	0.412	0.377	0.372
1.0e-04	0.393	0.381	0.402	0.403	0.421	0.377	0.372
3.0e-04	0.393	0.403	0.403	0.414	0.436	0.378	0.374
1.0e-03	0.393	0.480	0.404	0.453	0.454	0.387	0.382
3.0e-03	0.393	0.700	0.408	0.563	0.469	0.398	0.390
1.0e-02	0.393	1.470	0.422	0.947	0.484	0.403	0.399

Appendix

Table 5: Numerical results on EmbedLLM routing. Bold indicates lowest regret + inspection cost, asterisk indicates significance at

p<.05

(paired bootstrap).

Panel of Figure 7 — Figure 7: Evaluation of the 2 algorithm, which uses $\mu_{m}$ instead of $u^{\text{backup}}_{m}$ for evaluating committing policies.

Setting	Method	Gaussian		KNN
		Regret + Cost	Queries	Regret + Cost	Queries
MATH	Pandora’s Router	0.1049	0.58	0.1090	0.78
RAG	Pandora’s Router	0.1181	1.40	0.1218	1.22
EmbedLLM	Pandora’s Router	0.3867	3.71	0.3913	0.40

Appendix

Table 6: Comparison between Gaussian and KNN approximations for Pandora’s Router.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 351 papers

Oct 16, 2026

Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

Multi-turn LLM routing selects a model at every turn before its response is available, yet task success is revealed only when the interaction ends, leaving each intermediate choice without a label of its own. Logged trajectories, however, retain the response each call produced, which exposes what each turn required and how well the chosen model served it, evidence that outcomes alone hide. We introduce RIVET (Response-Informed Value Estimation for Turn-Level Routing), which exploits this post-call evidence during training while keeping routing strictly pre-call at deployment. RIVET jointly trains two value estimators over a shared state-model representation: a deployable Routing Value Estimator that scores candidate models from pre-call information, and a training-only Response-Informed Value Estimator that additionally observes the realized response. Conditioning on the response turns response-induced return variation from residual noise into an observed covariate, and we show that this reduces the leading asymptotic estimation error of the shared representation. On the MINT benchmark, RIVET solves more held-out tasks than the strongest multi-turn baseline while using 74.7% less total call cost.

Haotian Yan, Jiaqi Liu, Bingyi Jing +1

Southwest University of Finance and Economics · Shenzhen Loop Area Institute · Dalian University of Technology +1

LLM Routing Cost-Aware Inference Learning Using Privileged Information

Ask PDF

Jun 17, 2026

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.cn. Different from model-level response evaluation, RouteJudge focuses on router-level decision quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints. The selected model responses are then presented to users through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Each evaluation record stores the query, routing decisions, model responses, preference labels, cost, latency, and task metadata, enabling preference-aware, cost-aware, and task-conditioned analysis of LLM routers. To support the continuous expansion of routing methods in RouteJudge, we further release ORBIT (Optimal Routing and Budgeted Inference Toolbox), a modular and extensible toolbox that standardizes the end-to-end workflow of LLM routing. ORBIT provides unified interfaces for benchmark loading, query representation, router implementation, budget-aware evaluation, and method comparison, allowing researchers to develop and evaluate routing algorithms under consistent protocols. It also serves as the submission and integration layer for RouteJudge: researchers can implement routing methods within ORBIT, validate them on existing routing benchmarks, and submit compatible routers for online preference-based evaluation. The code of ORBIT is available at https://github.com/LAMDA-Model-Reuse/ORBIT.

Guannan Lai, Haoran Hu, Han-Jia Ye

School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI

LLM Evaluation Pairwise Preference Evaluation Cost-Aware Inference LLM Routing

Ask PDF

Sep 26, 2026

Planner-as-Router: Joint Plan-Time Model Routing for Cost-Efficient Multi-Agent Workflows

Running large language model (LLM) agents in production gets expensive fast. A frontier model (the largest, most capable tier) is accurate but can cost 25 times what a small model costs per token, and the gap compounds once a workflow chains several calls together. Planner-as-Router (PaR) attacks this from a different angle. Instead of leaving model-tier selection to some component downstream, it folds the choice into planning itself. As the planner breaks a query into subtasks, it also assigns each one a model size tier (small, mid, or frontier, ordered by capability and price), so the dependencies between subtasks are visible before any specialist runs. Unlike per-call routers such as cascade routing, which look at one node at a time, PaR sees the whole workflow up front and needs no separate router model or training data. We evaluate PaR with EntBench, a benchmark of 54 enterprise agentic tasks across seven classes, graded by actually running the generated Structured Query Language (SQL) and MongoDB queries against live databases. Over 1,157 evaluations spanning eight routers and three seeds, PaR stays on the observed cost-accuracy frontier. It matches a sink-frontier heuristic (frontier model on terminal nodes only) in accuracy at comparable cost and a faithful FrugalGPT cascade at lower cost, and cuts cost 44% against all-frontier routing while giving up 2.9 points of accuracy. Several accuracy gaps fall inside the plus-or-minus six-point confidence interval of a 54-task study, so we frame PaR's advantage as frontier position rather than a clean accuracy win. We also report a preliminary observation, not a validated result: a small pilot hints that cheap routing may carry a hidden compounding penalty on compositional workflows, which we frame as a hypothesis for future measurement. PaR, EntBench, and all evaluation code are open source.

Vivek Kumar Singh, Preeti Priyam, Gautam Bhowmick

Independent Researcher McKinney, TX, USA · Independent Researcher Chicago, IL, USA

LLM Agent Workflow Optimization LLM Routing Multi-Agent LLM Systems LLM Planning

Ask PDF

Abstract

LLM Routing

Explore similar work

Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

Planner-as-Router: Joint Plan-Time Model Routing for Cost-Efficient Multi-Agent Workflows