Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The primary benchmark comprises 4,181 competition-level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conservative policies under-escalate, whereas higher-solve frozen routers often over-escalate. A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informative for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozen gpt-oss-120b router and 92.4% for a retrospective fixed-order oracle. Across 10 paired model-condition settings, the oracle adds 23.2-58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5-28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost-aware routing remains unresolved.
Figures & tables
Class
Policy
Solve (%) [95% CI]
Avg. tokens (K) [95% CI]
Excess (K) [95% CI]
Fixed
Baseline
56.3 [51.5, 61.0]
18.2 [17.1, 19.2]
1.7 [1.1, 2.4]
Heuristic
Tier-majority
65.0 [60.5, 69.5]
28.9 [26.0, 31.9]
5.7 [3.9, 7.7]
Frozen LLM
gpt-oss-120b
73.8 [69.5, 77.5]
71.3 [56.5, 86.7]
37.1 [24.2, 50.6]
Frozen LLM
gpt-oss + cost prompt
78.3 [74.2, 82.0]
88.6 [71.0, 107.6]
51.6 [35.2, 68.9]
Confidence
Self-confidence gate
78.0 [74.0, 81.8]
45.0 [41.6, 48.7]
14.8 [12.2, 17.4]
Retrospective
Fixed-order oracle
92.4 [89.8, 95.0]
101.1 [80.1, 124.5]
0.0 [0.0, 0.0]
Table 1: Representative routing policies on the primary held-out test split ( n=423 ). Intervals use 2,000 problem-level bootstrap resamples. Excess is positive token overpayment above the realized fixed-order oracle. The protocol executions and router/probe calls use deterministic decoding, so the intervals do not measure fresh-run variability.
Target
Prev.
AUROC
AUPRC
Baseline fails
43.4
0.8847
0.8950
Any collaboration helps
36.0
0.8544
0.7683
PER first success
8.8
0.7259
0.1674
Broadcast-only success
4.2
0.7639
0.1041
Table 2: The same post-answer failure score on increasingly specific targets ( n=4,151 parseable cases). Prevalence is a percentage. “PER first” means Baseline and Single fail but PER succeeds; “Broadcast-only success” means Baseline, Single, and PER fail while Broadcast succeeds in the matched realized runs.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Routed unit
Fixed solver?
Matched outcomes?
Main target
Model routing/cascades
Base model
No
Usually no
Quality–price tradeoff
Adaptive computation
Steps/depth
Usually
Usually no
Stop or continue
Self-consistency routing
Sampled reasoning paths
Yes
Usually no
Number of samples
Tool/agent routing
Tools, roles, topology
Varies
Usually no
Workflow selection
This work
Collaboration protocol
Yes
Yes
Realized solve–token value
Appendix
Table S1: Scope comparison. “Matched” means that every candidate action is observed for every evaluated problem, enabling paired offline analysis.
Setting
n
Baseline
Single
PER
Broadcast
Oracle
gpt-oss-120b
OmniMath 2
4,181
56.83 [55.4,58.4]
78.78 [77.6,80.0]
85.17 [84.1,86.2]
89.21 [88.3,90.2]
92.68 [91.9,93.5]
JEEBench
515
41.55 [37.5,45.8]
55.34 [51.1,59.6]
91.46 [88.9,94.0]
94.95 [93.0,96.7]
96.50 [94.8,98.1]
SciBench
565
62.12 [58.1,66.0]
72.21 [68.5,75.8]
87.43 [84.6,90.1]
89.73 [87.1,92.2]
91.15 [88.7,93.5]
LAB strict
741
19.03 [16.1,21.9]
30.23 [27.0,33.7]
47.64 [44.0,51.3]
74.22 [71.1,77.5]
77.33 [74.2,80.3]
LAB text-no-tool
1,542
30.54 [28.3,32.9]
55.64 [53.2,58.1]
59.27 [56.7,61.7]
81.26 [79.4,83.2]
86.38 [84.6,88.1]
Appendix
Table S2: Solve/coverage percentages with 95% problem-level bootstrap intervals across all 10 matched settings. Point estimates show two decimal places and interval endpoints one; machine-readable results retain full precision. Each point estimate is based on one deterministic realized run per problem–protocol pair.
Solver
Setting
n
Baseline
Single
PER
Broadcast
None
Gemma-4-31B-it
OmniMath 2
4,181
69.39
17.01
6.41
2.87
4.33
gpt-oss-120b
OmniMath 2
4,181
56.83
22.91
8.75
4.19
7.32
Gemma-4-31B-it
JEEBench
515
70.49
11.07
14.76
2.91
0.78
gpt-oss-120b
JEEBench
515
41.55
15.15
35.53
4.27
3.50
Gemma-4-31B-it
SciBench
565
70.27
9.38
11.68
2.12
6.55
gpt-oss-120b
SciBench
565
62.12
11.86
13.81
3.36
8.85
Appendix
Table S3: Fixed-order-oracle label distributions (percent). The action order is Baseline → Single → PER → Broadcast → None.
Solver
Setting
n
Parse
Failure AUROC [CI]
ECE [CI]
Brier [CI]
Conf. correct
Conf. wrong
Gemma-4-31B-it
OmniMath
4,181
.999
.8012 [.7872,.8162]
.1594 [.1478,.1712]
.1726 [.1618,.1829]
.8674
.3735
gpt-oss-120b
OmniMath
4,181
.993
.8847 [.8732,.8955]
.0852 [.0742,.0965]
.1314 [.1233,.1398]
.8122
.2596
Gemma-4-31B-it
LAB strict
741
.989
.8637 [.8359,.8894]
.1460 [.1219,.1732]
.1506 [.1326,.1697]
.6225
.1268
gpt-oss-120b
LAB strict
741
.970
.5814 [.5410,.6165]
.3474 [.3184,.3849]
.3432 [.3173,.3728]
.6188
.4532
Gemma-4-31B-it
LAB text-no-tool
1,542
1.000
.7337 [.7110,.7562]
.1100 [.0891,.1321]
.2117 [.1999,.2237]
.6387
.3703
gpt-oss-120b
LAB text-no-tool
1,542
.979
.6069 [.5808,.6347]
.2132 [.1919,.2393]
.2874 [.2714,.3025]
.4980
.4059
Appendix
Table S4: Post-answer, pre-collaboration confidence metrics. The AUROC target is Baseline failure and the score is 1−P(Baseline correct) . Unparseable outputs are excluded, not imputed.
Solver/setting
Target
n
Prev.
AUROC
AUPRC
Gemma OmniMath
Baseline fails
4,178
.306
.8012
.5871
Any collaboration helps
4,178
.263
.7828
.5057
PER first success
4,178
.064
.7290
.1255
Broadcast-only success
4,178
.029
.7389
.0626
gpt-oss OmniMath
Baseline fails
4,151
.434
.8847
.8950
Any collaboration helps
4,151
.360
.8544
.7683
Appendix
Table S5: The same no-leakage failure score evaluated against coarse and protocol-specific targets. “PER first” means Baseline and Single fail and PER succeeds. “Broadcast-only success” means Baseline, Single, and PER fail while Broadcast succeeds in the matched realized runs. AUPRC should be read relative to target prevalence.
Solver
Setting
n
Base
Router [95% CI]
Oracle
Gap
Macro-F1
Route B/S/P/R/N (%)
Gemma-4-31B-it
OmniMath
628
.694
.766 [.731,.798]
.957
.191
.268
49.8/24.0/12.7/5.6/7.8
gpt-oss-120b
OmniMath
628
.568
.667 [.631,.704]
.927
.260
.284
45.7/19.9/15.6/7.2/11.6
Gemma-4-31B-it
LAB strict
112
.446
.705 [.616,.786]
.911
.205
.495
57.1/3.6/6.2/26.8/6.2
gpt-oss-120b
LAB strict
112
.188
.562 [.473,.661]
.768
.205
.499
14.3/15.2/29.5/25.0/16.1
Gemma-4-31B-it
LAB text
232
.418
.776 [.720,.828]
.961
.185
.485
36.6/43.1/4.3/13.8/2.2
gpt-oss-120b
LAB text
232
.306
.573 [.509,.638]
.862
.289
.456
34.9/29.7/3.0/16.4/15.9
Appendix
Table S6: Held-out text+metadata router evaluation. B/S/P/R/N denote Baseline, Single, PER, Broadcast, and None. Baseline, router, and oracle use identical held-out problem identifiers within each setting.
Solver
Setting
n
Router − Tier-majority [95% CI]
Router − Baseline [95% CI]
Gemma-4-31B-it
OmniMath
628
+7.2 [4.0,10.2]
+7.2 [4.0,10.2]
gpt-oss-120b
OmniMath
628
+3.3 [0.0,6.5]
+9.9 [6.5,13.2]
Gemma-4-31B-it
LAB strict
112
+25.0 [16.1,33.9]
+25.9 [17.9,34.0]
gpt-oss-120b
LAB strict
112
− 8.0 [ − 17.0,1.8]
+37.5 [28.6,46.4]
Gemma-4-31B-it
LAB text
232
+10.8 [5.2,16.4]
+35.8 [29.7,42.2]
gpt-oss-120b
LAB text
232
+0.9 [ − 5.2,6.5]
+26.7 [20.7,32.8]
Appendix
Table S7: Paired solve-rate differences in percentage points with 2,000 problem-level bootstrap resamples. Performance relative to Tier-majority is heterogeneous even though every router improves over Baseline.
Solver
Setting
n
PER
Broad.
Broad. − PER [CI]
PER-only
Broad.-only
Both
Neither
Gemma-4-31B-it
OmniMath
569
47.1
57.8
10.7 [6.0,15.3]
59 (10.4)
120 (21.1)
209 (36.7)
181 (31.8)
gpt-oss-120b
OmniMath
847
43.2
52.8
9.6 [5.8,13.3]
94 (11.1)
175 (20.7)
272 (32.1)
306 (36.1)
Gemma-4-31B-it
LAB strict
294
29.3
73.8
44.6 [38.4,51.0]
7 (2.4)
138 (46.9)
79 (26.9)
70 (23.8)
gpt-oss-120b
LAB strict
493
31.0
62.7
31.6 [27.0,36.3]
16 (3.2)
172 (34.9)
137 (27.8)
168 (34.1)
Gemma-4-31B-it
LAB text
370
45.1
75.9
30.8 [24.6,37.6]
33 (8.9)
147 (39.7)
134 (36.2)
56 (15.1)
gpt-oss-120b
LAB text
651
29.2
63.9
34.7 [30.9,38.9]
25 (3.8)
251 (38.6)
165 (25.3)
210 (32.3)
Appendix
Table S8: PER versus Broadcast conditional on Baseline and Single both failing. Solve rates, differences, and parenthesized outcome rates are percentages; cells before parentheses give counts. CIs are paired problem-level bootstrap intervals.
Label
Solve and cost
Error direction
Router
Acc.
F1
Solve
Tok. K
Excess K
Cost/Solve K
Missed
Under
Over
Always Baseline
56.3
14.4
56.3
18.2
1.7
32.3
36.2
36.2
7.6
Always Single
22.9
7.5
78.5
47.6
17.3
60.7
13.9
13.2
63.8
Always PER
9.0
3.3
84.9
401.9
308.0
473.5
7.6
4.3
86.8
Always Broadcast
4.3
1.6
88.9
622.1
523.4
699.8
3.5
0.0
95.7
Tier-majority
60.0
23.0
65.0
28.9
5.7
44.5
29.7
27.4
12.5
Appendix
Table S9: Extended routing comparison on the held-out test split. Percent-valued columns are percentages; token columns are thousands. Missed is the share of oracle-solvable problems not solved by the selected protocol. Cost/Solve is average tokens per solved problem.
Router
Solve (%)
Avg tok. (K)
Excess (K)
Baseline
56.3 [51.5, 61.0]
18.2 [17.1, 19.2]
1.7 [1.1, 2.4]
Tier-majority
65.0 [60.5, 69.5]
28.9 [26.0, 31.9]
5.7 [3.9, 7.7]
Meta-only
60.8 [56.0, 65.0]
44.3 [32.3, 58.5]
20.5 [10.3, 33.7]
Text+meta
61.2 [56.5, 66.0]
41.6 [30.9, 54.5]
18.2 [9.0, 29.5]
gpt-oss-120b
73.8 [69.5, 77.5]
71.3 [56.5, 86.7]
37.1 [24.2, 50.6]
gpt-oss + cost prompt
78.3 [74.2, 82.0]
88.6 [71.0, 107.6]
51.6 [35.2, 68.9]
Appendix
Table S10: Percentile bootstrap confidence intervals for the main comparison, computed with 2,000 problem-level resamples drawn with replacement from the held-out test set.
Router
Solve
Tok.
Excess
Under
Over
Fall.
(%)
(K)
(K)
(%)
(%)
Embed-only kNN
66.0
91.8
54.5
26.0
22.9
0
Embed-only logreg
68.6
176.0
124.1
21.7
33.8
0
Embed+meta kNN
66.0
52.4
22.5
26.5
15.4
0
Embed+meta logreg
70.9
215.7
153.8
19.6
34.0
0
gpt-oss + cost prompt
78.3
88.6
51.6
13.0
45.2
1
Appendix
Table S11: Robustness ablations: stronger text features and a cost-aware prompt for the frozen router. Sentence embedding routers recover additional solve signal, but not at the cost-efficiency of difficulty_tier_majority . The gpt-oss-120b numeric-cost/few-shot prompt improves solve rate over the original frozen prompt but still over-escalates substantially.
Step
Δ Solve
Δ Tok.
Tok./extra solve
(pp)
(K/problem)
(K)
Baseline → Tier-majority
8.7
10.8
123
Tier-maj. → Self-conf.
13.0
16.1
124
Self-conf. → gpt-oss
-4.3
26.4
dominated
Self-conf. → Llama
2.4
110.7
4,684
Self-conf. → Gemma-4
3.8
150.6
3,981
Appendix
Table S12: Marginal cost of additional solves. The last column divides the increase in average tokens per problem by the increase in solve probability; “dominated” means the destination policy is more expensive and solves fewer problems. The marginal-cost view makes the self-confidence gate’s efficiency visible: later frozen-router gains are much more expensive.
Router
Main tok.
Router tok.
Tok. incl.
Excess incl.
(K)
(K)
(K)
(K)
gpt-oss-20b
29.1
0.4
29.5
11.3
gpt-oss-120b
71.3
0.4
71.8
37.6
gpt-oss + cost prompt
88.6
0.8
89.3
52.4
Llama-3.1-70B
155.7
0.4
156.1
114.4
Gemma-4-E4B
195.6
0.4
195.9
130.0
Appendix
Table S13: Router-call token sensitivity for frozen LLM-as-router policies. The main tables count the selected protocol trace cost; the sensitivity table additionally adds each router decision call from the API usage logs. The added router calls are small relative to protocol execution costs, so the solve-cost ordering is unchanged. The self-confidence-gate row in the main tables already includes its self-assessment query cost, averaging 0.34K tokens per problem.
Router
Solve
Tok.
Excess
F1
(%)
(K)
(K)
Tier-majority
65.0
28.9
5.7
23.0
Source-maj. w/o tier
62.6
33.6
10.8
21.4
Meta-only
60.8
44.3
20.5
23.0
Meta-only w/o tier
60.8
55.7
30.2
23.7
Text+meta
61.2
41.6
18.2
23.1
Appendix
Table S14: Difficulty-metadata ablation on the held-out test split. Rows marked “w/o tier” remove both difficulty and difficulty_tier . Learned routers use the same split and dev-selected hyperparameter rule as the main learned-router baselines.
Router
Solve
Tok.
Excess
Under
Over
(%)
(K)
(K)
(%)
(%)
Tier-maj. cross-src
68.7
21.7
2.9
25.3
9.1
Source-maj. fallback
65.7
17.1
1.2
28.4
5.9
Meta-only w/o tier
78.0
42.5
17.7
15.4
46.7
Text+meta w/o tier
74.9
58.2
30.8
18.4
39.0
Text-only
73.0
61.5
36.2
20.4
26.5
Appendix
Table S15: Source-holdout stress test over the six largest sources ( n=2994 ). Rows are aggregated across separate held-out-source runs. The source-majority row falls back to the global train-set majority because the held-out source is unseen during training.
Probe
Coverage
AUROC
Tier AUROC
ECE
Brier
(%)
(pp)
gpt-oss self
77.8
0.859
0.773
7.5
0.152
Llama cross-model
100.0
0.743
0.756
28.7
0.288
Gemma-3 cross-model
100.0
0.768
0.756
15.8
0.217
Appendix
Table S16: Predictive and calibration summary for the pre-answer confidence probe. The same-model probe is the cleaned gpt-oss-120b self-assessment used by the self-confidence gate. The cross-model checks ask Llama-3.1-70B and Gemma-3-27B to estimate whether gpt-oss-120b will solve the same problems in one direct attempt. Expected calibration error (ECE) uses ten equal-width confidence bins; lower Brier score indicates better probabilistic accuracy.
Gemma-3 check
n
Base
Single
PER
Broad.
None
Protocol solve
833
42.5
60.4
65.5
58.6
–
Oracle share
833
42.5
18.4
10.2
2.8
26.2
Gemma router pred.
833
17.6
25.6
56.8
0.0
0.0
Appendix
Table S17: Reduced Gemma-3 actor-stack check on a tier-sampled subset with more balanced difficulty coverage than the full benchmark. Values are percentages. The first row reports realized solve rates for each protocol with Gemma-3 actors. The second row reports the resulting fixed-order-oracle label share. The third row reports a Gemma-3 frozen-router prompt on the same subset.
Policy
Selected threshold(s)
Solve (%)
Avg tok. (K)
Under (%)
Over (%)
Self-confidence gate
τ=70
78.0 [74.0, 81.8]
45.0 [41.5, 48.6]
13.7 [10.6, 17.0]
42.1 [37.1, 46.6]
Two-threshold cascade
τlow=10,τhigh=70
75.2 [70.9, 79.2]
41.7 [38.1, 45.3]
16.5 [13.0, 20.3]
28.4 [23.6, 32.6]
Appendix
Table S18: Self-confidence gate and two-threshold cascade on the held-out test split. Thresholds are tuned on the dev split using confidence on a 0–100 scale. Costs include the pre-answer confidence-query tokens. Bracketed ranges are 95% bootstrap confidence intervals.
Policy
Subset
n
Solve (%)
Avg tok. (K)
Self-conf. gate
Full test
423
78.0
45.0
Self-conf. gate
Confidence available
329
75.4
46.4
Two-threshold
Full test
423
75.2
41.7
Two-threshold
Confidence available
329
75.4
46.7
Appendix
Table S19: Missing-confidence ablation for the confidence policies. The full-test rows keep each policy’s built-in behavior on problems without a usable cleaned confidence estimate; the confidence-available rows restrict to examples with cleaned confidence values.
Policy
Dev value
Solve
Tok.
Excess
Under
Over
(%)
(K)
(K)
(%)
(%)
Value threshold
30
67.4
37.9
12.3
25.1
14.9
Value utility
0.1
78.0
52.9
22.0
13.7
61.5
Appendix
Table S20: Exploratory protocol-value probe. Threshold selects the first protocol in cost order whose estimated success probability exceeds 30%. Utility selects the protocol maximizing estimated solve probability minus 0.1 times prompt-visible token cost in thousands. Probe coverage means that all four requested probabilities were parsed; missing rows fall back to tier-majority.
Tier
n
Mean conf.
Actual pass
1
15
95.0
100.0
2
38
85.9
86.8
3
2
59.0
100.0
4
87
61.4
65.5
5
82
48.3
51.2
6
33
43.7
30.3
Appendix
Table S21: Cleaned pre-answer self-assessment by difficulty tier. Confidence and pass rates are percentages. Tiers with very small n should be interpreted cautiously.
Confidence bin
n
Baseline
Single
PER
Broadcast
None
0–20
29
13.8
27.6
24.1
10.3
24.1
20–40
104
22.1
40.4
12.5
11.5
13.5
40–60
70
48.6
30.0
17.1
2.9
1.4
60–80
52
75.0
15.4
5.8
0.0
3.8
80–100
74
97.3
2.7
0.0
0.0
0.0
Appendix
Table S22: Oracle label composition within cleaned pre-answer confidence bins. Low confidence does not identify a single next-best protocol; bins with n<30 should be interpreted cautiously.
Quantity
Value
Total test problems
423
Usable cleaned estimates
329
Coverage
77.8%
Original JSON parses
310
Recovered direct estimates
11
Recovered ranges
8
Appendix
Table S23: Coverage decomposition for the cleaned pre-answer self-assessment data.
Split
n
Base
Single
PER
Broad.
None
Train
3342
56.8
23.0
8.7
4.2
7.3
Dev
416
57.0
23.1
8.7
4.1
7.2
Test
423
56.3
22.9
9.0
4.3
7.6
Appendix
Table S24: Split sizes and oracle-label percentages. Splits are stratified by the fixed-order-oracle label.
Protocol
Avg. tokens
Solve rate
Baseline
18.2K
56.3
Single
47.6K
78.5
PER
401.9K
84.9
Broadcast
622.1K
88.9
Appendix
Table S25: Protocol-level cost and solve rate on the held-out test split.
Oracle label
Count
Fraction
Baseline
238
56.3
Single
97
22.9
PER
38
9.0
Broadcast
18
4.3
none
32
7.6
Appendix
Table S26: Fixed-order-oracle protocol distribution on the test split.
Figure S4: Additional learned-router diagnostics. (a) The lightweight text+metadata router begins to overfit after early epochs. (b) The aggregate comparison is not explained by a single benchmark source.
LLM agent routing selects models, tools, skills, or collaboration workflows for each task or step to balance capability, cost, latency, and reliability.
Large language model (LLM)-based multi-agent systems tackle complex reasoning by orchestrating how multiple agents are configured and how they collaborate. A central challenge is to adapt orchestration to the evolving collaboration state. Routing from the query alone cannot adapt to intermediate progress or errors, which hurts accuracy. Routing from the complete execution history supplies this missing context, but forces later decisions to process every prior step, including redundant or low-utility ones. This creates an execution-history overload that inflates cost. Effective orchestration instead requires a compact state that captures useful progress without accumulating redundant context. We propose Gated-Memory Routing, which conditions each decision on the query and a learned execution memory. A learned Memory Write Gate commits only non-redundant reasoning steps, and a learned Retrieval Gate supplies each agent a compact, relevant subset, so every decision conditions on a clean, informative state. At each step, the system selects the next role and backbone from this memory, while an Adaptive Halting Controller stops execution once the memory contains sufficient evidence for answering. Across five reasoning and code-generation benchmarks, our framework is both effective and efficient: it attains the best average accuracy, exceeding the strongest baseline by 2.44 points, while reducing HumanEval inference cost by 31.9% relative to that baseline. Code is available at https://github.com/rajibrhasan/gated-memory-routing
No single Large Language Model (LLM) is uniformly reliable across queries, motivating multi-model inference systems that either route among models or combine their outputs. However, routing stops after selecting an initial model, while dense collaboration invokes peers on every query. We show that collaboration is non-monotonic: peers can recover failures that no model solves alone, but can also corrupt initially correct answers. We introduce COMED (Controlled Model Escalation for Multi-LLM Deliberation), a post-anchor controller for selective cross-model collaboration. COMED uses anchor self-consistency, router margin, and a lightweight peer probe to accept confident answers, verify ambiguous cases, and escalate only when collaboration is likely beneficial. We formalize this trade-off with a rescue-harm decomposition showing that selective collaboration improves when rescued errors outweigh collaboration-induced harms. Across medical, scientific, and general reasoning benchmarks, COMED improves fixed and routed anchors in all 16 open-weight settings, with gains up to +10.7 percentage points on MedQA while invoking fewer models and using fewer decoded tokens than dense collaboration. On HLE with frontier models, COMED improves GPT-5.5 from 23.1% to 28.1%, outperforming dense collaboration and achieving the best results.
Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.
Zhuohan Wang, Haoran Ma, Tianyu Wu +3
Harvard University · Rutgers University · The Hong Kong University of Science and Technology