cs.AIAug 14, 2026

LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks

Authors: Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang, Vikram Vasudevan, Huihuo Zheng, Venkatram Vishwanath, Rajeev Thakur

Organizations: Argonne National Laboratory, Lemont, IL, USA · Oregon State University, Corvallis, OR, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The primary benchmark comprises 4,181 competition-level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conservative policies under-escalate, whereas higher-solve frozen routers often over-escalate. A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informative for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozen gpt-oss-120b router and 92.4% for a retrospective fixed-order oracle. Across 10 paired model-condition settings, the oracle adds 23.2-58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5-28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost-aware routing remains unresolved.

Figures & tables

ClassPolicySolve (%) [95% CI]Avg. tokens (K) [95% CI]Excess (K) [95% CI]
FixedBaseline56.3 [51.5, 61.0]18.2 [17.1, 19.2]1.7 [1.1, 2.4]
HeuristicTier-majority65.0 [60.5, 69.5]28.9 [26.0, 31.9]5.7 [3.9, 7.7]
Frozen LLMgpt-oss-120b73.8 [69.5, 77.5]71.3 [56.5, 86.7]37.1 [24.2, 50.6]
Frozen LLMgpt-oss + cost prompt78.3 [74.2, 82.0]88.6 [71.0, 107.6]51.6 [35.2, 68.9]
ConfidenceSelf-confidence gate78.0 [74.0, 81.8]45.0 [41.6, 48.7]14.8 [12.2, 17.4]
RetrospectiveFixed-order oracle92.4 [89.8, 95.0]101.1 [80.1, 124.5]0.0 [0.0, 0.0]
Table 1: Representative routing policies on the primary held-out test split ( n=423n=423 ). Intervals use 2,000 problem-level bootstrap resamples. Excess is positive token overpayment above the realized fixed-order oracle. The protocol executions and router/probe calls use deterministic decoding, so the intervals do not measure fresh-run variability.
TargetPrev.AUROCAUPRC
Baseline fails43.40.88470.8950
Any collaboration helps36.00.85440.7683
PER first success8.80.72590.1674
Broadcast-only success4.20.76390.1041
Table 2: The same post-answer failure score on increasingly specific targets ( n=4,151n=4{,}151 parseable cases). Prevalence is a percentage. “PER first” means Baseline and Single fail but PER succeeds; “Broadcast-only success” means Baseline, Single, and PER fail while Broadcast succeeds in the matched realized runs.
Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

FamilyRouted unitFixed solver?Matched outcomes?Main target
Model routing/cascadesBase modelNoUsually noQuality–price tradeoff
Adaptive computationSteps/depthUsuallyUsually noStop or continue
Self-consistency routingSampled reasoning pathsYesUsually noNumber of samples
Tool/agent routingTools, roles, topologyVariesUsually noWorkflow selection
This workCollaboration protocolYesYesRealized solve–token value
Appendix
Table S1: Scope comparison. “Matched” means that every candidate action is observed for every evaluated problem, enabling paired offline analysis.
SettingnnBaselineSinglePERBroadcastOracle
gpt-oss-120b
OmniMath 24,18156.83 [55.4,58.4]78.78 [77.6,80.0]85.17 [84.1,86.2]89.21 [88.3,90.2]92.68 [91.9,93.5]
JEEBench51541.55 [37.5,45.8]55.34 [51.1,59.6]91.46 [88.9,94.0]94.95 [93.0,96.7]96.50 [94.8,98.1]
SciBench56562.12 [58.1,66.0]72.21 [68.5,75.8]87.43 [84.6,90.1]89.73 [87.1,92.2]91.15 [88.7,93.5]
LAB strict74119.03 [16.1,21.9]30.23 [27.0,33.7]47.64 [44.0,51.3]74.22 [71.1,77.5]77.33 [74.2,80.3]
LAB text-no-tool1,54230.54 [28.3,32.9]55.64 [53.2,58.1]59.27 [56.7,61.7]81.26 [79.4,83.2]86.38 [84.6,88.1]
Appendix
Table S2: Solve/coverage percentages with 95% problem-level bootstrap intervals across all 10 matched settings. Point estimates show two decimal places and interval endpoints one; machine-readable results retain full precision. Each point estimate is based on one deterministic realized run per problem–protocol pair.
SolverSettingnnBaselineSinglePERBroadcastNone
Gemma-4-31B-itOmniMath 24,18169.3917.016.412.874.33
gpt-oss-120bOmniMath 24,18156.8322.918.754.197.32
Gemma-4-31B-itJEEBench51570.4911.0714.762.910.78
gpt-oss-120bJEEBench51541.5515.1535.534.273.50
Gemma-4-31B-itSciBench56570.279.3811.682.126.55
gpt-oss-120bSciBench56562.1211.8613.813.368.85
Appendix
Table S3: Fixed-order-oracle label distributions (percent). The action order is Baseline →\rightarrow Single →\rightarrow PER →\rightarrow Broadcast →\rightarrow None.
SolverSettingnnParseFailure AUROC [CI]ECE [CI]Brier [CI]Conf. correctConf. wrong
Gemma-4-31B-itOmniMath4,181.999.8012 [.7872,.8162].1594 [.1478,.1712].1726 [.1618,.1829].8674.3735
gpt-oss-120bOmniMath4,181.993.8847 [.8732,.8955].0852 [.0742,.0965].1314 [.1233,.1398].8122.2596
Gemma-4-31B-itLAB strict741.989.8637 [.8359,.8894].1460 [.1219,.1732].1506 [.1326,.1697].6225.1268
gpt-oss-120bLAB strict741.970.5814 [.5410,.6165].3474 [.3184,.3849].3432 [.3173,.3728].6188.4532
Gemma-4-31B-itLAB text-no-tool1,5421.000.7337 [.7110,.7562].1100 [.0891,.1321].2117 [.1999,.2237].6387.3703
gpt-oss-120bLAB text-no-tool1,542.979.6069 [.5808,.6347].2132 [.1919,.2393].2874 [.2714,.3025].4980.4059
Appendix
Table S4: Post-answer, pre-collaboration confidence metrics. The AUROC target is Baseline failure and the score is 1−P(Baseline correct)1-P(\text{Baseline correct}) . Unparseable outputs are excluded, not imputed.
Solver/settingTargetnnPrev.AUROCAUPRC
Gemma OmniMathBaseline fails4,178.306.8012.5871
 Any collaboration helps4,178.263.7828.5057
 PER first success4,178.064.7290.1255
 Broadcast-only success4,178.029.7389.0626
gpt-oss OmniMathBaseline fails4,151.434.8847.8950
 Any collaboration helps4,151.360.8544.7683
Appendix
Table S5: The same no-leakage failure score evaluated against coarse and protocol-specific targets. “PER first” means Baseline and Single fail and PER succeeds. “Broadcast-only success” means Baseline, Single, and PER fail while Broadcast succeeds in the matched realized runs. AUPRC should be read relative to target prevalence.
SolverSettingnnBaseRouter [95% CI]OracleGapMacro-F1Route B/S/P/R/N (%)
Gemma-4-31B-itOmniMath628.694.766 [.731,.798].957.191.26849.8/24.0/12.7/5.6/7.8
gpt-oss-120bOmniMath628.568.667 [.631,.704].927.260.28445.7/19.9/15.6/7.2/11.6
Gemma-4-31B-itLAB strict112.446.705 [.616,.786].911.205.49557.1/3.6/6.2/26.8/6.2
gpt-oss-120bLAB strict112.188.562 [.473,.661].768.205.49914.3/15.2/29.5/25.0/16.1
Gemma-4-31B-itLAB text232.418.776 [.720,.828].961.185.48536.6/43.1/4.3/13.8/2.2
gpt-oss-120bLAB text232.306.573 [.509,.638].862.289.45634.9/29.7/3.0/16.4/15.9
Appendix
Table S6: Held-out text+metadata router evaluation. B/S/P/R/N denote Baseline, Single, PER, Broadcast, and None. Baseline, router, and oracle use identical held-out problem identifiers within each setting.
SolverSettingnnRouter −- Tier-majority [95% CI]Router −- Baseline [95% CI]
Gemma-4-31B-itOmniMath628+7.2 [4.0,10.2]+7.2 [4.0,10.2]
gpt-oss-120bOmniMath628+3.3 [0.0,6.5]+9.9 [6.5,13.2]
Gemma-4-31B-itLAB strict112+25.0 [16.1,33.9]+25.9 [17.9,34.0]
gpt-oss-120bLAB strict112−- 8.0 [ −- 17.0,1.8]+37.5 [28.6,46.4]
Gemma-4-31B-itLAB text232+10.8 [5.2,16.4]+35.8 [29.7,42.2]
gpt-oss-120bLAB text232+0.9 [ −- 5.2,6.5]+26.7 [20.7,32.8]
Appendix
Table S7: Paired solve-rate differences in percentage points with 2,000 problem-level bootstrap resamples. Performance relative to Tier-majority is heterogeneous even though every router improves over Baseline.
SolverSettingnnPERBroad.Broad. −- PER [CI]PER-onlyBroad.-onlyBothNeither
Gemma-4-31B-itOmniMath56947.157.810.7 [6.0,15.3]59 (10.4)120 (21.1)209 (36.7)181 (31.8)
gpt-oss-120bOmniMath84743.252.89.6 [5.8,13.3]94 (11.1)175 (20.7)272 (32.1)306 (36.1)
Gemma-4-31B-itLAB strict29429.373.844.6 [38.4,51.0]7 (2.4)138 (46.9)79 (26.9)70 (23.8)
gpt-oss-120bLAB strict49331.062.731.6 [27.0,36.3]16 (3.2)172 (34.9)137 (27.8)168 (34.1)
Gemma-4-31B-itLAB text37045.175.930.8 [24.6,37.6]33 (8.9)147 (39.7)134 (36.2)56 (15.1)
gpt-oss-120bLAB text65129.263.934.7 [30.9,38.9]25 (3.8)251 (38.6)165 (25.3)210 (32.3)
Appendix
Table S8: PER versus Broadcast conditional on Baseline and Single both failing. Solve rates, differences, and parenthesized outcome rates are percentages; cells before parentheses give counts. CIs are paired problem-level bootstrap intervals.
 LabelSolve and costError direction
RouterAcc.F1SolveTok. KExcess KCost/Solve KMissedUnderOver
Always Baseline56.314.456.318.21.732.336.236.27.6
Always Single22.97.578.547.617.360.713.913.263.8
Always PER9.03.384.9401.9308.0473.57.64.386.8
Always Broadcast4.31.688.9622.1523.4699.83.50.095.7
Tier-majority60.023.065.028.95.744.529.727.412.5
Appendix
Table S9: Extended routing comparison on the held-out test split. Percent-valued columns are percentages; token columns are thousands. Missed is the share of oracle-solvable problems not solved by the selected protocol. Cost/Solve is average tokens per solved problem.
RouterSolve (%)Avg tok. (K)Excess (K)
Baseline56.3 [51.5, 61.0]18.2 [17.1, 19.2]1.7 [1.1, 2.4]
Tier-majority65.0 [60.5, 69.5]28.9 [26.0, 31.9]5.7 [3.9, 7.7]
Meta-only60.8 [56.0, 65.0]44.3 [32.3, 58.5]20.5 [10.3, 33.7]
Text+meta61.2 [56.5, 66.0]41.6 [30.9, 54.5]18.2 [9.0, 29.5]
gpt-oss-120b73.8 [69.5, 77.5]71.3 [56.5, 86.7]37.1 [24.2, 50.6]
gpt-oss + cost prompt78.3 [74.2, 82.0]88.6 [71.0, 107.6]51.6 [35.2, 68.9]
Appendix
Table S10: Percentile bootstrap confidence intervals for the main comparison, computed with 2,000 problem-level resamples drawn with replacement from the held-out test set.
RouterSolveTok.ExcessUnderOverFall.
 (%)(K)(K)(%)(%) 
Embed-only kNN66.091.854.526.022.90
Embed-only logreg68.6176.0124.121.733.80
Embed+meta kNN66.052.422.526.515.40
Embed+meta logreg70.9215.7153.819.634.00
gpt-oss + cost prompt78.388.651.613.045.21
Appendix
Table S11: Robustness ablations: stronger text features and a cost-aware prompt for the frozen router. Sentence embedding routers recover additional solve signal, but not at the cost-efficiency of difficulty_tier_majority . The gpt-oss-120b numeric-cost/few-shot prompt improves solve rate over the original frozen prompt but still over-escalates substantially.
StepΔ\Delta SolveΔ\Delta Tok.Tok./extra solve
 (pp)(K/problem)(K)
Baseline →\rightarrow Tier-majority8.710.8123
Tier-maj. →\rightarrow Self-conf.13.016.1124
Self-conf. →\rightarrow gpt-oss-4.326.4dominated
Self-conf. →\rightarrow Llama2.4110.74,684
Self-conf. →\rightarrow Gemma-43.8150.63,981
Appendix
Table S12: Marginal cost of additional solves. The last column divides the increase in average tokens per problem by the increase in solve probability; “dominated” means the destination policy is more expensive and solves fewer problems. The marginal-cost view makes the self-confidence gate’s efficiency visible: later frozen-router gains are much more expensive.
RouterMain tok.Router tok.Tok. incl.Excess incl.
 (K)(K)(K)(K)
gpt-oss-20b29.10.429.511.3
gpt-oss-120b71.30.471.837.6
gpt-oss + cost prompt88.60.889.352.4
Llama-3.1-70B155.70.4156.1114.4
Gemma-4-E4B195.60.4195.9130.0
Appendix
Table S13: Router-call token sensitivity for frozen LLM-as-router policies. The main tables count the selected protocol trace cost; the sensitivity table additionally adds each router decision call from the API usage logs. The added router calls are small relative to protocol execution costs, so the solve-cost ordering is unchanged. The self-confidence-gate row in the main tables already includes its self-assessment query cost, averaging 0.34K tokens per problem.
RouterSolveTok.ExcessF1
 (%)(K)(K) 
Tier-majority65.028.95.723.0
Source-maj. w/o tier62.633.610.821.4
Meta-only60.844.320.523.0
Meta-only w/o tier60.855.730.223.7
Text+meta61.241.618.223.1
Appendix
Table S14: Difficulty-metadata ablation on the held-out test split. Rows marked “w/o tier” remove both difficulty and difficulty_tier . Learned routers use the same split and dev-selected hyperparameter rule as the main learned-router baselines.
RouterSolveTok.ExcessUnderOver
 (%)(K)(K)(%)(%)
Tier-maj. cross-src68.721.72.925.39.1
Source-maj. fallback65.717.11.228.45.9
Meta-only w/o tier78.042.517.715.446.7
Text+meta w/o tier74.958.230.818.439.0
Text-only73.061.536.220.426.5
Appendix
Table S15: Source-holdout stress test over the six largest sources ( n=2994n=2994 ). Rows are aggregated across separate held-out-source runs. The source-majority row falls back to the global train-set majority because the held-out source is unseen during training.
ProbeCoverageAUROCTier AUROCECEBrier
 (%)  (pp) 
gpt-oss self77.80.8590.7737.50.152
Llama cross-model100.00.7430.75628.70.288
Gemma-3 cross-model100.00.7680.75615.80.217
Appendix
Table S16: Predictive and calibration summary for the pre-answer confidence probe. The same-model probe is the cleaned gpt-oss-120b self-assessment used by the self-confidence gate. The cross-model checks ask Llama-3.1-70B and Gemma-3-27B to estimate whether gpt-oss-120b will solve the same problems in one direct attempt. Expected calibration error (ECE) uses ten equal-width confidence bins; lower Brier score indicates better probabilistic accuracy.
Gemma-3 checknnBaseSinglePERBroad.None
Protocol solve83342.560.465.558.6–
Oracle share83342.518.410.22.826.2
Gemma router pred.83317.625.656.80.00.0
Appendix
Table S17: Reduced Gemma-3 actor-stack check on a tier-sampled subset with more balanced difficulty coverage than the full benchmark. Values are percentages. The first row reports realized solve rates for each protocol with Gemma-3 actors. The second row reports the resulting fixed-order-oracle label share. The third row reports a Gemma-3 frozen-router prompt on the same subset.
PolicySelected threshold(s)Solve (%)Avg tok. (K)Under (%)Over (%)
Self-confidence gateτ=70\tau=7078.0 [74.0, 81.8]45.0 [41.5, 48.6]13.7 [10.6, 17.0]42.1 [37.1, 46.6]
Two-threshold cascadeτlow=10, τhigh=70\tau_{\text{low}}=10,\ \tau_{\text{high}}=7075.2 [70.9, 79.2]41.7 [38.1, 45.3]16.5 [13.0, 20.3]28.4 [23.6, 32.6]
Appendix
Table S18: Self-confidence gate and two-threshold cascade on the held-out test split. Thresholds are tuned on the dev split using confidence on a 0–100 scale. Costs include the pre-answer confidence-query tokens. Bracketed ranges are 95% bootstrap confidence intervals.
PolicySubsetnnSolve (%)Avg tok. (K)
Self-conf. gateFull test42378.045.0
Self-conf. gateConfidence available32975.446.4
Two-thresholdFull test42375.241.7
Two-thresholdConfidence available32975.446.7
Appendix
Table S19: Missing-confidence ablation for the confidence policies. The full-test rows keep each policy’s built-in behavior on problems without a usable cleaned confidence estimate; the confidence-available rows restrict to examples with cleaned confidence values.
PolicyDev valueSolveTok.ExcessUnderOver
  (%)(K)(K)(%)(%)
Value threshold3067.437.912.325.114.9
Value utility0.178.052.922.013.761.5
Appendix
Table S20: Exploratory protocol-value probe. Threshold selects the first protocol in cost order whose estimated success probability exceeds 30%. Utility selects the protocol maximizing estimated solve probability minus 0.10.1 times prompt-visible token cost in thousands. Probe coverage means that all four requested probabilities were parsed; missing rows fall back to tier-majority.
TiernnMean conf.Actual pass
11595.0100.0
23885.986.8
3259.0100.0
48761.465.5
58248.351.2
63343.730.3
Appendix
Table S21: Cleaned pre-answer self-assessment by difficulty tier. Confidence and pass rates are percentages. Tiers with very small nn should be interpreted cautiously.
Confidence binnnBaselineSinglePERBroadcastNone
0–202913.827.624.110.324.1
20–4010422.140.412.511.513.5
40–607048.630.017.12.91.4
60–805275.015.45.80.03.8
80–1007497.32.70.00.00.0
Appendix
Table S22: Oracle label composition within cleaned pre-answer confidence bins. Low confidence does not identify a single next-best protocol; bins with n<30n<30 should be interpreted cautiously.
QuantityValue
Total test problems423
Usable cleaned estimates329
Coverage77.8%
Original JSON parses310
Recovered direct estimates11
Recovered ranges8
Appendix
Table S23: Coverage decomposition for the cleaned pre-answer self-assessment data.
SplitnnBaseSinglePERBroad.None
Train334256.823.08.74.27.3
Dev41657.023.18.74.17.2
Test42356.322.99.04.37.6
Appendix
Table S24: Split sizes and oracle-label percentages. Splits are stratified by the fixed-order-oracle label.
ProtocolAvg. tokensSolve rate
Baseline18.2K56.3
Single47.6K78.5
PER401.9K84.9
Broadcast622.1K88.9
Appendix
Table S25: Protocol-level cost and solve rate on the held-out test split.
Oracle labelCountFraction
Baseline23856.3
Single9722.9
PER389.0
Broadcast184.3
none327.6
Appendix
Table S26: Fixed-order-oracle protocol distribution on the test split.
Figure S4: Additional learned-router diagnostics. (a) The lightweight text+metadata router begins to overfit after early epochs. (b) The aggregate comparison is not explained by a single benchmark source.
More in this topic

LLM Agent Routing

LLM agent routing selects models, tools, skills, or collaboration workflows for each task or step to balance capability, cost, latency, and reliability.

See all 98 papers

Explore similar work

CardsList

  1. Learning What to Retain: Gated-Memory Routing for Efficient Collaboration in Multi-Agent LLM Systems

    Aug 31, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouMulti-Agent LLM SystemsMulti-Agent System Optimization

    University of Central Florida

  2. COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

    Sep 22, 2026Norah Alballa, Wenxuan Zhang, Salma Kharrat +4LLM Inference EfficiencyLLM Routing

    KAUST · LUMS

  3. Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

    Sep 26, 2026Zhuohan Wang, Haoran Ma, Tianyu Wu +3LLM Mathematical ReasoningLLM Evaluation

    Harvard University · Rutgers University · The Hong Kong University of Science and Technology