Small language models are increasingly deployed on edge devices and robotic systems to meet cost and latency constraints, yet their limited capabilities can make reliable deployment challenging. Stronger models can provide assistance when needed, but deciding when to route a weak-model response to a stronger model often relies on signals of response correctness. In open-ended and agentic tasks, however, obtaining such signals can require costly ground-truth supervision that is difficult to obtain at scale. We revisit the foundations of weak-to-strong routing and ask whether routing decisions can instead be guided by signals that do not require ground-truth supervision. We first characterize the optimal weak-to-strong routing policy, while allowing for multiple weak-model responses generated through local inference. Building on this characterization, we derive a principled approximation to the optimal routing based on model-to-model comparison. In particular, we introduce a notion of model disagreement that can be measured directly from model-generated responses, without ground-truth supervision. Importantly, the approximation becomes increasingly accurate as the strong model becomes a better proxy for the ground-truth distribution. Across agentic and single-turn tasks, our approach effectively navigates the trade-off between routing cost and response accuracy without ground-truth supervision. Moreover, even when ground-truth correctness signals are available for training, disagreement-based routing can outperform methods trained directly to predict correctness
Figures & tables
Figure 1: Corrected disagreement better approximates optimal routing than alternatives.
Figure 2: Action-level routing performance in agentic environments. For clarity, we plot our proposed signals against only the best-performing label-free baseline. Results are averaged over m=5 runs; comprehensive baselines, standard deviations, and tabular data are provided in Appendix A.4 .
Figure 3: Finite-sample routing performance on single-turn benchmarks for one-shot ( N=1 ) and multi-sample ( N=8 ) regimes. Corrected disagreement consistently outperforms raw disagreement and the best label-free baselines. Multi-sample baselines only close the gap on closed-form tasks (MATH) where clustering is trivial, but break down on open-ended tasks (Spider, AceCode).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Population-level comparison of routing signals with N=1 weak-model response. Each panel shows routing accuracy over the full budget range B∈[0,1] for a different dataset and weak–strong model pair. We compare optimal routing, corrected disagreement, raw weak–strong disagreement, and weak-model correctness.
Figure 5: Population-level comparison of routing signals with N=4 weak-model responses per prompt. Panels and routing signals follow Figure 4 .
Figure 6: Population-level comparison of routing signals with N=8 weak-model responses per prompt. Panels and routing signals follow Figure 4 .
Setting
Routing score
5%
10%
15%
20%
25%
30%
50%
N=1 weak sample per prompt ( b=0 : 0.380)
Corrected disagreement
ours
0.419
0.452
0.482
0.513
0.540
0.568
0.659
Weak–strong disagreement
ours
0.397
0.418
0.440
0.464
0.491
0.519
0.631
Token entropy
baseline
0.404
0.427
0.447
0.468
0.489
0.509
0.584
Proxy reward †
baseline
0.396
0.415
0.436
0.460
0.487
0.514
0.629
Oracle routing gain
oracle
0.429
0.479
0.529
0.578
0.626
0.670
0.776
Appendix
Table 1: Math -500 routing results for the weak–strong pair Phi-3.5-mini-instruct → GPT-4o-mini . The student answer is kept unless routed to the strong model. Each panel reports accuracy at different routing budgets. The b=0 value is the accuracy of the kept weak answer before routing.
Setting
Routing score
5%
10%
15%
20%
25%
30%
50%
N=1 weak sample per prompt ( b=0 : 0.499)
Corrected disagreement
ours
0.537
0.563
0.586
0.608
0.628
0.648
0.716
Weak–strong disagreement
ours
0.533
0.559
0.582
0.605
0.626
0.646
0.716
Token entropy
baseline
0.515
0.530
0.547
0.562
0.577
0.591
0.647
Proxy reward †
baseline
0.532
0.557
0.579
0.601
0.623
0.643
0.714
Oracle routing gain
oracle
0.549
0.598
0.648
0.698
0.744
0.783
0.818
Appendix
Table 2: Spider routing results for the weak–strong pair Qwen3-1.7B → OmniSQL-7B . The weak query is kept unless routed to the strong model. Each panel reports accuracy at different routing budgets. The b=0 value is the accuracy of the kept weak query before routing.
Routing score
Type
5%
10%
15%
20%
25%
30%
50%
N=1 weak sample per prompt ( b=0 : 0.447)
Corrected disagreement
ours
0.468
0.486
0.503
0.519
0.535
0.552
0.608
Weak–strong disagreement
ours
0.464
0.484
0.502
0.518
0.535
0.551
0.606
Token entropy
baseline
0.460
0.475
0.489
0.504
0.518
0.532
0.591
Proxy reward †
baseline
0.458
0.471
0.486
0.501
0.517
0.534
0.594
Oracle routing gain
oracle
0.497
0.547
0.596
0.646
0.690
0.708
0.709
Appendix
Table 3: AceCode routing results for the weak–strong pair Qwen2.5-Coder-1.5B-Instruct → Qwen3-Coder-10B-A3B-Instruct . The weak program is kept unless routed to the strong model. Each panel reports accuracy at different routing budgets. The b=0 value is the accuracy of the kept weak program before routing.
Figure 7: Effect of weak-response selection on Math -500 for Phi-3.5-mini-instruct → GPT-4o-mini , using N=8 weak responses per prompt and 500 test prompts. Each panel fixes the routing score and reports cascade accuracy for different selection policies and routing budgets. At b=0 , accuracy depends only on the selection policy. Bold values indicate the highest accuracy in each column. † denotes methods that use ground-truth supervision during training.
Figure 8: Effect of weak-response selection on Spider . Routing uses corrected disagreement throughout. We compare selection policies across routing budgets and different numbers of weak-model samples N .
Figure 9: Effect of weak-response selection on AceCode . Routing uses corrected disagreement throughout. We compare selection policies across routing budgets and different numbers of weak-model samples N .
Routing Strategy
Success Rate (SE)
Average Task Score
Realized Budget ( ≈ )
20%
40%
60%
20%
40%
60%
Corrected disagreement (ours)
0.210 (0.029)
0.460 (0.035)
0.765 (0.030)
0.665
0.748
0.828
Weak–strong disagreement (ours)
0.065 (0.017)
0.260 (0.031)
0.715 (0.032)
0.466
0.716
0.880
Self-consistency
0.045 (0.015)
0.145 (0.025)
0.410 (0.035)
0.448
0.536
0.755
Token entropy
0.040 (0.014)
0.150 (0.025)
0.405 (0.035)
0.448
0.535
0.698
Random baseline
0.160 (0.026)
0.200 (0.028)
0.505 (0.035)
0.543
0.590
0.776
Appendix
Table 4: WebShop routing performance (200 evaluation tasks) comparing realized routing budget, end-to-end success rate, and average task score across multiple label-free baselines.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
LLM routing aims to select a capable model at low cost, but specialized routers can require updates as models and tasks change. We propose COMPASS, an agentic router in which an LLM follows a routing recipe, gathers performance and cost evidence through tools, and optionally inspects candidate answers before selecting a model. Our evaluations on LLMRouterBench and DeepSWE show that agentic routing can improve accuracy and lower model-call cost relative to specialized and trained routing baselines, without routing-specific training. On LLMRouterBench, COMPASS reaches 69.85% accuracy at $139.57, compared with RLCascadeRouter's 69.14% at $180.22. On DeepSWE under fixed five-fold cross-validation, it reaches 78.1% at $215, compared with 74.1% at $500 for the best single model. COMPASS also supports routing across benchmarks and adapts to an expanded model pool by reading new evaluation records, without changing the router's weights or prompt.
An LLM router picks which model should answer each query. The appeal is that models fail on different questions. Whatever single model is best overall still gets some wrong, and another model in the pool gets many of those right. Getting that choice right every time is the ceiling, and a router is an attempt to approach it. However, recent work reports that routers do not get close. Across 21 routing methods on five benchmarks, sharply different designs land within a fraction of a point of each other, and all of them stay far below that ceiling. Learned routers often fail to beat simply always calling the strongest model. We ask what those missed questions have in common. We set fourteen models to answer all 294 questions, with 7 task types across 3 languages: Korean, English and Hindi. We ran the whole matrix twice, changing nothing, but 5.37% of the 4,116 model-question pairs came out scored differently anyway. Run-to-run movement like that is normal, and we argue that a small win does not show that routing did anything, ours or anyone else's. Counting an answer correct only when the model got it right in both runs, 29 questions on this matrix can be improved with routing. Every correct-answer count here is on that rule. Task type accounts for most of them: assigning each task type one model in advance, chosen once and never updated, improves 21 of the 29. Splitting each task type by language improves 2 more and leaves 6 of 294 unoptimized. That handful is what a learned router would have been built for, and it is smaller than the run-to-run movement above, which is a share of pairs rather than of questions. The static table we adopted answers 262 of 294 questions at $3.33 per run, against the best single model's 245 at $7.69. All of this is fitted and scored on the same 294 questions with no holdout.
Set-based language model routing aims to increase the coverage of the event where at least one model answers a query correctly while controlling the number of inference calls made. Existing risk-controlled routing policies bound the all-miss risk (the probability that every model in the selected set fails) by calibrating a single conformal threshold and applying it to a nested family of model sets. Because each model is scored in isolation, these policies ignore the dependence structure of model failures. Models that share training data or architecture tend to fail on the same queries, so two individually strong models can be jointly redundant while a weaker but complementary model covers the queries that the others miss. To account for this dependence, which existing risk-controlled policies ignore, we propose CopRoute, which orders models by how many new queries each one answers correctly beyond those already selected, decides per query how far down that order to go, and then picks the smallest cutoff that still meets the target risk. We show that CopRoute's ordering attains, on the design data, the best guarantee any efficient fixed-ordering algorithm can give for the average number of calls to the first correct model, and that a separate calibration step certifies the all-miss risk of the deployed sets. On RouterBench and RouterEval, CopRoute attains the fewest certified calls on nine of ten datasets at a target risk of 0.05 and on eight of ten at 0.10, making up to 79 percent fewer calls (median 24 percent) than the state-of-the-art risk-controlled policy at the same certified all-miss risk.