Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost. Routing requires estimating each specialist's expected return, but this value estimation has a cost. Cheap estimators (e.g., embedding-based predictors) are fast but noisy, while accurate estimators (e.g., fine-tuned models with access to retrieval results or partial reasoning traces) are expensive. We formalize this tradeoff as an instance of Pandora's Box, the classical problem of optimal search with costly inspection. Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that determine, for each specialist and input, whether refining the value estimate is worth its cost. We call the centralized policy Pandora's Router. We extend this to a decentralized setting, Pandora's Bidder, where specialists independently decide whether to invest in self-assessment before accepting an offered price to claim a query. Experiments across three domains---a standard multi-LLM benchmark, retrieval-augmented specialists, and LLMs with variable inference-time reasoning---show that Pandora's Router matches the routing quality of exhaustive estimation, while querying the expensive estimator far less often. In the decentralized setting, value-of-information reasoning improves allocative efficiency when competing estimates are accurate; when competing estimates are noisy, however, it can increase the strategic specialist's utility at the expense of others.
Figures & tables
Figure 1: Overview of the Pandora framework. The system balances cheap screening and costly reasoning for routing. (1) A concrete prompt x is received. (2) Multiple specialists m∈{1,…,M} generate cheap, noisy value estimates fm(x) . (3) The system computes reservation prices that quantify the expected value-of-information for querying a more costly and accurate value estimate gm(x) , e.g., by running the specialist for a few reasoning tokens and checking if it is on the right track. (4) This unified principle supports allocation by centralized routing and decentralized auctions.
Setting
Estimator
MSE
cg/cf
Description
MATH
f : KNN
.154
5.8
k -nearest neighbors on prompt embeddings
g : SFT-CoT- 20
.096
Fine-tuned transformer on prompt + first 20 tokens of the model’s chain-of-thought (CoT) reasoning trace
RAG
f : KNN
.175
>7000
k -nearest neighbors on prompt embeddings
g : SFT-retrievals
.109
Fine-tuned transformer on prompt + retrieval results
EmbedLLM
f : KNN
.266
1.6
k -nearest neighbors on prompt embeddings
g : SFT-prompt
.198
Fine-tuned transformer on prompt
Table 1: Value estimators across experimental domains. The cheap estimator f is always computed; the costly estimator g is queried selectively. The mean-squared error (MSE) is measured on a held-out calibration set. The cost ratios cg/cf are discussed in Section 2.2 and derived in Section D.3 .
Method
MATH
RAG
EmbedLLM
Regret
Cost
Total
Regret
Cost
Total
Regret
Cost
Total
f -only
0.117
0.000
0.117
0.150
0.000
0.150
0.393
0.000
0.393
g -only
0.090
0.038
0.128
0.084
0.057
0.141
0.370
1.986
2.356
Top- 2
0.090
0.038
0.128
0.107
0.038
0.146
0.402
0.036
0.438
Coin Flip
0.107
0.019
0.127
0.122
0.029
0.151
0.398
0.992
1.390
Random- Npr
0.164
0.011
0.175
0.135
0.027
0.162
0.363
0.075
0.438
Table 2: Routing evaluation results averaged across query costs cg . Lower is better on all metrics; the lowest average regret + cost measured per setting is bolded . See Section D.4 in the Appendix for a breakdown of results per cost level cg , together with paired statistical significance tests.
Figure 2: Routing performance on the MATH, RAG, and EmbedLLM domains for varying costs of querying g , with f=KNN3 . The ideal value estimation policy would minimize regret + inspection cost at every cost level. Pandora’s Router achieves near-minimal total cost across the full range of cg (where cm=cg for all m ), querying g frequently when it is cheap and rarely when it is expensive.
Figure 3: Results for the posted-price auction across the three experimental domains for varying refinement cost cg . The VoI-based bidding strategy interpolates between querying g when cheap and declining when expensive, while static baselines either over-invest or under-invest in refinement.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Number of queries to g during model routing.
Figure 5: Specialist surplus regret and total allocative efficiency regret of the posted-price auction, when bidding against opponents who have access only to f , rather than g as in Figure 3 .
Figure 6: Routing performance versus monetary inspection cost, as estimated in Section D.3 .
Cost
f-only
g-only
Top- 2
Coin Flip
Random- Npr
Margin- Npr
Pandora’s Router
1.5e-03
0.117
0.093
0.093
0.109
0.194
0.093
0.095
2.1e-03
0.117
0.094
0.094
0.109
0.193
0.097
0.096
3.2e-03
0.117
0.096
0.096
0.110
0.198
0.098
0.098
4.6e-03
0.117
0.099
0.099
0.112
0.197
0.099
0.099
6.8e-03
0.117
0.104
0.104
0.114
0.193
0.101
0.101
1.0e-02
0.117
0.110
0.110
0.117
0.199
0.104 *
0.107
Appendix
Table 3: Numerical results on MATH routing. Bold indicates lowest regret + inspection cost, asterisk indicates significance at p<.05 (paired bootstrap).
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI