cs.MAAug 18, 2026

What Confidence Routing Is Actually Doing: Auditing Routing, Calibration, and Commitment in Multi-Agent Deliberation

Authors: Jingyan Jiang, Huihuo Zheng, Rajeev Thakur, Chih-Hsuan Yang

Organizations: Argonne National Laboratory, Lemont, IL 60439, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A common multi-agent design asks agents to report confidence and lets the highest-scoring agent speak next, implicitly using one scalar both to route the conversation and to estimate uncertainty. We audit this confidence-routed broadcast protocol by separating three trace-level questions: whether it selects the right candidate (routing), whether reported confidence behaves like a probability (calibration), and whether the selected agent publicly states the answer that won the turn (commitment). Our primary study covers 4,181 gpt-oss-120b olympiad-math traces; we repeat the audit on a 2-by-2 actor-by-benchmark grid that adds gemma-4-31B-it and a biology multiple-choice benchmark. In the primary cell, confidence discriminates correct from wrong candidates (AUROC 0.72) but is strongly overconfident (79% mean stated confidence versus 52% accuracy). A cross-fitted, tier-stratified isotonic procedure reduces Expected Calibration Error from 0.278 to 0.008 on held-out candidates, but it does not recover missing discrimination: raw AUROC is only 0.537 and 0.440 in the two Gemma cells. Routing is likewise setting-dependent. Fixed routers differ by at most 1.1 percentage points on gpt-oss/math, whereas raw-confidence argmax performs 5.6 and 11.2 points below random-valid selection in the Gemma cells. Commitment is distinct again: in the primary cell, poll and spoken answers diverge in 20.4% of valid pairs, 62.4% of those revisions are fresh generations, and the unconditional correctness shift is -1.7 points; the other three cells instead range from +0.9 to +12.2 points. The transferable lesson is procedural: routing discrimination, probability calibration, and public commitment must be measured separately before raw confidence is used for deployment decisions.

Figures & tables

Figure 1: The audited protocol has two answer-bearing calls. In round tt , agent ii returns a structured poll with candidate answer ci,tc_{i,t} and stated confidence si,ts_{i,t} . The router selects the highest-confidence agent i∗i^{*} , which then emits a free-form public message ui∗,tu_{i^{*},t} with extracted spoken candidate c^i∗,t\hat{c}_{i^{*},t} . The poll candidate is used only to allocate the turn, whereas the spoken candidate enters the public transcript. The audit treats the routing decision, the calibration of si,ts_{i,t} , and the relationship between ci∗,tc_{i^{*},t} and c^i∗,t\hat{c}_{i^{*},t} as three distinct objects.
RouterExpected successGap to oracle
Runtime confidence argmax0.4970.052
Top-confidence tie-random0.4930.055
Random valid candidate0.4900.059
Exact-answer majority vote0.4960.052
Confidence-weighted cluster vote0.5010.048
Learned router ( P(correct)P(\text{correct}) )0.4990.050
Table 1: Router counterfactuals on the gpt-oss/math cell, over 9,653 all-routeable poll blocks. Five fixed-rule routers cluster within 1.1 pp, and a learned router trained to predict candidate correctness lands in the same band. The oracle ceiling is 4.8–5.9 pp above them all. For this broadcast protocol, the routing rule matters less than whether the block contains a correct candidate.
CalibratorECE ↓\downarrowBrier ↓\downarrowAUROCmean conf.
Raw (as reported)0.2780.3020.7210.790
Temperature ( T=4.35T{=}4.35 )0.1420.2420.7220.580
Isotonic0.0080.2080.7200.518
Table 2: Post-hoc recalibration on the gpt-oss /math cell, cross-fitted by problem_uid ( K=2K{=}2 , fold assignment stratified by tier) over 23,391 valid poll candidates. ECE uses 10 equal-width bins. The isotonic procedure reduces ECE from 0.278 to 0.008 and Brier score from 0.302 to 0.208; candidate-level AUROC changes only from 0.721 to 0.720. These are held-out candidate-level estimates, not a rerun of the router on calibrated values.
Primary classN% of diverg.Δ\Delta correct (pp, speak −- poll)
self-revert36719.0−11.4-11.4
incumbent-pull1200 6.2+1.7+1.7
peer-echo23812.4−5.0-5.0
novel1,20262.4−9.1-9.1
Table 3: Sources of divergent spoken answers ( N=1,927N=1{,}927 divergent valid pairs), by precedence-resolved primary class (self-revert ≻\succ incumbent-pull ≻\succ peer-echo ≻\succ novel). Negative Δ\Delta correct means the spoken revision is net-harmful on that subset. Overlapping (non-exclusive) match flags are in Appendix E .
Predictor (z-scored)β\betaOR per SDpp
Pool-solvable rate (router-indep.)+4.80+4.80≈121\approx 1211.3⋅10−121.3\cdot 10^{-12}
Uncond. improvement-event rate+1.30+1.303.683.680.0030.003
Tier difficulty+0.39+0.391.481.485.1⋅10−45.1\cdot 10^{-4}
Selected-invalid rate+0.58+0.581.781.781.3⋅10−51.3\cdot 10^{-5}
Divergence rate (PSDR)+0.15+0.151.161.160.220.22
Uncond. degradation-event rate+0.09+0.091.091.090.910.91
Table 4: Exploratory within-broadcast outcome linkage. Logistic regression with cluster-robust SEs by problem cluster ( n=4,181n=4{,}181 , nclusters=421n_{\text{clusters}}=421 ). Pool-solvable rate has the largest same-trace association. For mean selected confidence, the reported MLE is small and non-significant (OR 0.84, p=0.35p=0.35 ), while sensitivity refits span OR 0.84–1.11 (Appendix I ); its sign is therefore not stable.
 mathbiobiomath
Actorgpt-ossgpt-ossgemmagemma
Valid poll cand.23,3912,9124,85214,129
Candidate acc.0.5180.5930.4910.491
Mean stated conf.0.7900.7290.8790.839
ECE raw0.2780.1670.4380.443
ECE isotonic0.0080.0230.0080.001
Table 5: Descriptive point estimates from the three-axis audit over a 2×22\times 2 actor ×\times benchmark grid. The first column is the primary result. Routing rows are per-block success over all-routeable blocks; net commitment shift is the unconditional poll-to-speak correctness shift in percentage points. Correctness labels in all cells use the same fixed gpt-oss-120b equivalence judge, so the Gemma columns are cross-family-labeled. Per-cell confidence intervals are not reproduced in this preprint.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

AxisMetricEstimate95% CIDenominator
RoutingCRA0.506[0.486, 0.527]9,467 sel-valid blocks
RoutingHRWP0.906[0.896, 0.915]5,295 possible-hit blocks
RoutingMCR0.058[0.052, 0.065]7,970 eligible blocks
RoutingPCCA (tie-aware)0.570[0.549, 0.593]1,184 blocks / 2,079 pairs
RoutingCTR0.327[0.305, 0.349]2,079 corr-vs-wrong pairs
RoutingSelected-invalid rate0.149[0.139, 0.159]11,122 selected blocks
Appendix
Table 6: Headline metrics with explicit denominators and problem-cluster bootstrap 95% CIs. Commitment metrics are reported both conditional on divergence and unconditional over all valid labeled selected pairs.
StageVisible contextOutput contractDownstream role
PollProblem, outer-loop evaluator feedback, current candidate answer, public discussion context, deferred private notes, own speaking historyStructured JSON with score , reason , intent , candidate_answer , has_candidate_answerProduces the candidate and confidence score used for turn allocation
RouterThe per-agent poll outputs; runtime rule is confidence argmax with deterministic first-seen tie resolutionSelected speaker identity for the current turnAllocates the floor; the top-confidence tie-random counterfactual reported in the main paper is the order-effect control
SpeakSelected agent sees the public transcript and system incumbent, but not peers’ private poll JSONFree-form public message, optionally with a boxed answerPublic commitment that enters the shared transcript and may diverge from the polled candidate
EvaluatorFinal submitted answer plus evaluator-only reference materialPASS/FAIL, correctness bit, non-leaking repair hintSupplies the external correctness signal used by the original run and some post-hoc labels
Appendix
Table 7: Runtime protocol surface for the audited broadcast configuration. The central asymmetry is that the poll is structured and private to the router, while the speak step is free-form and public. This is the protocol fact behind the poll/speak commitment gap.
SettingAudited value
Deliberation poolThree homogeneous openai/gpt-oss-120b peers at temperature 0, plus an evaluator from the same model family
Discussion / approval scheduleFour discussion turns per outer round, followed by two approval rounds
Routing ruleConfidence argmax with deterministic first-seen tie resolution
Deferred-note thresholdNon-selected agents write deferred private notes only if their poll score is at least 65
Public-memory regimesummary_plus_recent with recent_history_window = 5
Approval / candidate update policyUnanimous approval; reviewed candidate updated by majority_revision ; system incumbent persists until overridden
Appendix
Table 8: Load-bearing configuration knobs for the audited broadcast run. These settings matter because they determine what information the poll and speak stages actually see, how much context survives truncation, and how the system-level incumbent is updated between public speak events. They make the audit object more precise than “prompting style” alone.
SymbolDefinitionIntro
CRACandidate Routing Accuracy: fraction of selected-valid blocks where the routed candidate is correctmain paper
HRWPHit Rate When Possible: among blocks where any valid candidate is correct, fraction where the selected one is also correctmain paper
MCRMissed Correct Rate: fraction of blocks where the selected valid candidate is wrong but a non-selected valid candidate is correctmain paper
PCCAPairwise Confidence-Correctness Alignment: block-macro average of each mixed block’s tie-aware correct–wrong comparisons; 0.50.5 means no within-block ordering signalmain paper
CTRConfidence Tie Rate: fraction of within-block (correct, wrong) candidate pairs that are tied in confidencemain paper
ECEExpected Calibration Error: weighted mean absolute gap between binned mean confidence and bin accuracymain paper
Appendix
Table 9: Abbreviation glossary for the body and appendix metrics. Routers and router-counterfactual names ( runtime_argmax , top_confidence_tie_random , random_valid , majority_vote , confidence_weighted_vote , oracle ) are defined in-context in the main paper; the pool-solvable rate used in the outcome-linkage regression is also defined there.
TierHRWPPCCACTRPSDRNPCS∣D\mathrm{NPCS}_{\mid\mathcal{D}}ECE
010.9910.5000.0000.0080.0000.103
020.9620.5720.3830.1250.0590.128
03 ∗0.8530.5750.3680.170−0.250-0.2500.181
040.9360.5870.3530.166−0.086-0.0860.202
050.9150.6020.3610.201+0.022+0.0220.292
060.8710.5780.3070.233−0.093-0.0930.356
Appendix
Table 10: Per-tier routing, commitment, and calibration metrics. NPCS is conditional on divergent pairs. Tiers marked with ∗ have small denominators and should be interpreted with caution: tier 03 has 2 problem clusters and 47 valid pairs; tier 10 has 2 clusters and 54 valid pairs. Headline pooled estimates use cluster bootstrap over all 421 problem clusters.
Figure 3: Per-tier rank-versus-calibration split derived from Table 10 . Left: PCCA stays above the 0.5 no-signal floor across most tiers, but weakens toward harder slices. Right: ECE climbs sharply with difficulty, reaching the 0.31–0.41 range on tiers 06–10. Hollow markers indicate the small-denominator tiers 03 and 10. The point of the figure is not that either curve is perfectly monotone; it is that the probability interpretation degrades faster than the rank signal.
ModelTermCoef.95% CIORpp
Correctnessscore (norm)6.551[5.844, 7.258]700.11.18⋅10−731.18\cdot 10^{-73}
Correctnessscore ×\times difficulty−1.993-1.993[ −2.303-2.303 , −1.683-1.683 ]0.1362.44⋅10−362.44\cdot 10^{-36}
Correctnessround index−0.903-0.903[ −1.022-1.022 , −0.785-0.785 ]0.4051.26⋅10−501.26\cdot 10^{-50}
Divergencepoll correct−1.171-1.171[ −1.333-1.333 , −1.009-1.009 ]0.3101.23⋅10−451.23\cdot 10^{-45}
Divergencepoll correct ×\times difficulty0.254[0.181, 0.328]1.2908.85⋅10−128.85\cdot 10^{-12}
Divergencescore (norm)−1.208-1.208[ −1.886-1.886 , −0.530-0.530 ]0.2994.83⋅10−44.83\cdot 10^{-4}
Appendix
Table 11: Cluster-robust pooled logistic regressions with agent fixed effects. Correctness model uses 23,391 valid poll candidates; divergence model uses 9,436 valid selected poll-speak pairs. Both use cluster-robust SEs by problem cluster ( nclusters=421n_{\text{clusters}}=421 ). The outcome-linkage regression reported in the main paper is fit on the 4,181-trace level.
TermCoef.ORpp
role == self-prior-speak (vs. peer)−0.010-0.0100.990.89
role == incumbent (vs. peer)+0.519+0.5191.688.1⋅10−38.1\cdot 10^{-3}
lag (rounds back)+0.004+0.0041.000.95
source confidence (analyst covariate)+1.471+1.4714.357.3⋅10−47.3\cdot 10^{-4}
source was correct (vs. gold)+0.314+0.3141.374.2⋅10−44.2\cdot 10^{-4}
candidate is most-recent (1/0)+0.198+0.1981.224.7⋅10−24.7\cdot 10^{-2}
Appendix
Table 12: Adoption logit. One row per (divergent speak event, visible candidate), with adopted=1 when the spoken answer matches that candidate. Cluster-robust SEs use problem_uid . Source confidence is post-hoc metadata from the candidate’s originating private poll; it was not visible to the speaker, so its coefficient is an association rather than a confidence-visibility effect.
Match flag (non-exclusive)N% of diverg.
matches_self_revert36719.0
matches_incumbent_pull22411.6
matches_peer_echo55028.5
No match (novel)1,20262.4
Appendix
Table 13: Independent match flags over the same N=1,927N=1{,}927 divergent valid pairs as the precedence-resolved primary-class table in the main paper. A single divergent spoken answer can fire more than one flag, so the counts sum to more than 1,9271{,}927 and the percentages to more than 100100 . The flag-versus-primary gap sits almost entirely in peer-echo: a peer’s prior public answer matches 28.5%28.5\% of divergences, but only 12.4%12.4\% survive as primary peer-echo, because the other 312312 also match the speaker’s own earlier answer or the system incumbent, which take precedence. Self-revert and novel are unaffected by precedence, so their flag and primary counts coincide. Read against the primary-class table, this bounds how much divergence could be attributed to peer anchoring under a maximally permissive attribution rule.
ConcernCheckMain resultWhere
Confidence ranking is just noise; PCCA / CTR are artifacts of the metricWithin-block confidence permutation null and noisy-oracle synthetic on the same trace data, no model re-runsPermutation drives PCCA to 0.5000.500 and changes CTR from 0.3270.327 to 0.300±0.0060.300\pm 0.006 ; noisy oracle drives PCCA toward 11 and CTR toward 00 . This validates PCCA’s no-signal reference, not CTR invariance.App. Table 15
Self-judge artifact: gpt-oss-120b judges its own outputs and may bias NPCSTwo-judge relabeling of 4,379 residual pairs with google/gemma-4-31B-it ( κ=0.82\kappa{=}0.82 ) and google/gemma-4-E4B-it ( κ=0.75\kappa{=}0.75 )Conditional NPCS over the 9,263-pair tiers 02–09 population has the same negative sign under all three residual-label variants ( −0.083-0.083 , −0.061-0.061 , −0.060-0.060 ).App. H
Outcome linkage depends on a single regression specificationFit a related linkage logit with selected-correct rate and routing slack entered separately, using the same remaining predictors and cluster-robust SEsThe same-trace content variables remain strongly associated with outcome, but neither specification is interpreted causally.App. I
Single benchmark only (olympiad math throughout)Repeat the audit on LAB-Bench biology ( 15 ) for both actors, using the same extraction pipeline and fixed judgeOverconfidence and scale repair recur, while routing and commitment effects differ: raw-confidence argmax trails random-valid selection, and unconditional commitment shifts are positive in the biology cells.App. J
Single actor family only ( gpt-oss-120b deliberators throughout)Cross-family replication with google/gemma-4-31B-it deliberators on both benchmarks and a fixed gpt-oss-120b judgeGemma polls parse at 98.4%; scale correction lowers ECE, while raw AUROC remains weak (0.537) or inverted (0.440), and raw-confidence argmax trails random-valid selection.App. J
Appendix
Table 14: High-priority robustness and scope checks. The table distinguishes descriptive sensitivity from causal or deployment claims; no row turns a same-trace association into an intervention estimate.
ConstructionCRAHRWPPCCACTR
Runtime (headline reproduction)0.5060.9060.5700.327
A: permutation null ( n=50n{=}50 reps)0.520 ±\pm .0020.762 ±\pm .0030.500 ±\pm .0110.300 ±\pm .006
B: noisy oracle, σ=0.05\sigma{=}0.050.622 ±\pm .0011.0001.0000.000
B: noisy oracle, σ=0.10\sigma{=}0.100.620 ±\pm .0011.0001.0000.000
B: noisy oracle, σ=0.20\sigma{=}0.200.621 ±\pm .0011.0001.0000.000
B: noisy oracle, σ=0.50\sigma{=}0.500.605 ±\pm .0010.961 ±\pm .0020.913 ±\pm .0090.023 ±\pm .004
Appendix
Table 15: Sanity-check constructions over the existing traces. The permutation null confirms that runtime PCCA is meaningfully above the no-signal floor, while the noisy oracle confirms that PCCA saturates under a clean signal. CRA remains capped because selected-valid accuracy cannot exceed the underlying content ceiling.
Judge pairPairs decidedAgreementCohen’s κ\kappaConfusion (judge1: gpt-oss / judge2)
gpt-oss vs gemma-4-31B-it4,0480.9080.8161776/311/62/1899
gpt-oss vs gemma-4-E4B-it3,9450.8730.7481670/433/68/1774
Appendix
Table 16: Two-judge cross-check on full residual corpus. Confusion entries are (judge1=div, judge2=div) / (judge1=div, judge2=equ) / (judge1=equ, judge2=div) / (judge1=equ, judge2=equ). Both Gemma judges show an asymmetric disagreement pattern: when gpt-oss flagged a pair as divergent, the Gemma judge “rescues” it as equivalent more often (311 / 433) than the opposite direction (62 / 68).
Figure 4: Asymmetric disagreement in the cross-judge audit. On residual pairs where gpt-oss and Gemma disagree, the dominant observed direction is gpt-oss says divergent, Gemma says equivalent . The figure describes disagreement direction; it does not determine which label is correct.
Metricgpt-oss-120b tiers 02–09gemma-4 31B-itgemma-4 E4B-it
Divergent pairs1,9212,4182,537
PSDR0.2070.2610.274
PSIR (cond. divergence)0.1740.1320.126
PSDRg (cond. divergence)0.2570.1920.186
NPCS∣D\mathrm{NPCS}_{\mid\mathcal{D}}−0.083-0.083−0.061-0.061−0.060-0.060
Appendix
Table 17: Commitment metrics under three residual-case label variants over the same 9,263 valid pairs from tiers 02–09. PSIR, PSDRg, and NPCS∣D\mathrm{NPCS}_{\mid\mathcal{D}} use the judge-specific divergent denominator shown in the first row. Conditional NPCS has the same negative sign across variants; this is a sensitivity check, not ground-truth adjudication.
Predictor (z-scored)β\betaOR per SDpp
Selected-correct rate+5.94+5.94≈380\approx 3803.1⋅10−113.1\cdot 10^{-11}
Uncond. improvement-event rate+1.40+1.404.084.080.0020.002
Routing slack (oracle −- argmax)+1.08+1.082.962.969.0⋅10−49.0\cdot 10^{-4}
Tier difficulty+0.42+0.421.521.522.9⋅10−42.9\cdot 10^{-4}
Selected-invalid rate+0.60+0.601.811.819.1⋅10−69.1\cdot 10^{-6}
Divergence rate (PSDR)+0.19+0.191.211.210.100.10
Appendix
Table 18: Related decomposition specification with selected-correct rate and routing slack entered separately. Selected-correct rate is mechanically closer to the broadcast outcome than pool-solvable rate. McFadden pseudo- R2R^{2} is 0.802 here and 0.798 in the collapsed specification. Both use same-trace summaries and are interpreted descriptively.
 mathbiobiomath
Actorgpt-ossgpt-ossgemmagemma
Valid poll cand.23,3912,9124,85214,129
Candidate acc.0.5180.5930.4910.491
Mean stated conf.0.7900.7290.8790.839
ECE raw0.2780.1670.4380.443
ECE isotonic0.0080.0230.0080.001
Appendix
Table 19: Descriptive point estimates over the 2×22\times 2 actor ×\times benchmark grid. The first column is the primary result. Routing rows are per-block success over all-routeable blocks; net commitment shift is the unconditional poll-to-speak correctness shift in percentage points. Correctness labels in all cells use the same fixed gpt-oss-120b equivalence judge. Per-cell confidence intervals are not reproduced in this preprint.
ModePass rateAvg. tokensAvg. callsAvg. wall (s)
baseline_llm0.56818.4K9.6943.6
single_agent0.78848.1K18.074.3
broadcast0.892616.5K134.6294.9
Appendix
Table 20: Task-level context from the same broader verifier-gated math evaluation over the same 4,181 problems. Here baseline_llm is a single direct LLM solve, single_agent is one LLM with multi-turn self-revision, and broadcast is the multi-agent protocol audited in the body. Broadcast outperforms the same-task single_agent comparator in final pass rate (0.892 vs. 0.788), but at much higher average cost: 12.8 ×\times tokens, 7.5 ×\times model calls, and 4.0 ×\times wall time. These rows are contextual protocol comparisons, not equal-compute frontiers for the broadcast trace audit. Aggregates are weighted across tier-level summaries from that evaluation.
FieldR/ODescription
Protocol identity
protocol_nameRShort name for the deliberation protocol.
protocol_versionRVersion tag.
paper_of_recordOCitation for protocol definition.
Inference setup
agent_modelRModel id(s); homogeneous or heterogeneous.
Appendix
Table 21: Trace-Audit Card v0.1 schema. The card describes the measurement properties of a deliberation protocol; it does not describe a single model (model cards do that) or a dataset (data cards do that).
More in this topic

Multi-Agent Deliberation

See all 34 papers

Explore similar work

CardsList

  1. Confidence Composition for Multiagent Language Model Systems

    Jun 11, 2026Ali Elahi, Michael J. Curry, Barbara Di EugenioMulti-Agent LLM SystemsProbability Calibration

    University of Illinois Chicago

  2. Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities

    Oct 2, 2026Yikai Zhao, Saurabh Pandey, Pradeep Kumar MisraLLM Agent ReliabilityLLM Auditing

    Amazon

  3. MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

    May 21, 2026Joss ArmstrongAI Agent ReliabilityMulti-Agent Coordination

    Ericsson, Athlone, Ireland