A common multi-agent design asks agents to report confidence and lets the highest-scoring agent speak next, implicitly using one scalar both to route the conversation and to estimate uncertainty. We audit this confidence-routed broadcast protocol by separating three trace-level questions: whether it selects the right candidate (routing), whether reported confidence behaves like a probability (calibration), and whether the selected agent publicly states the answer that won the turn (commitment). Our primary study covers 4,181 gpt-oss-120b olympiad-math traces; we repeat the audit on a 2-by-2 actor-by-benchmark grid that adds gemma-4-31B-it and a biology multiple-choice benchmark. In the primary cell, confidence discriminates correct from wrong candidates (AUROC 0.72) but is strongly overconfident (79% mean stated confidence versus 52% accuracy). A cross-fitted, tier-stratified isotonic procedure reduces Expected Calibration Error from 0.278 to 0.008 on held-out candidates, but it does not recover missing discrimination: raw AUROC is only 0.537 and 0.440 in the two Gemma cells. Routing is likewise setting-dependent. Fixed routers differ by at most 1.1 percentage points on gpt-oss/math, whereas raw-confidence argmax performs 5.6 and 11.2 points below random-valid selection in the Gemma cells. Commitment is distinct again: in the primary cell, poll and spoken answers diverge in 20.4% of valid pairs, 62.4% of those revisions are fresh generations, and the unconditional correctness shift is -1.7 points; the other three cells instead range from +0.9 to +12.2 points. The transferable lesson is procedural: routing discrimination, probability calibration, and public commitment must be measured separately before raw confidence is used for deployment decisions.
Figures & tables
Figure 1: The audited protocol has two answer-bearing calls. In round t , agent i returns a structured poll with candidate answer ci,t and stated confidence si,t . The router selects the highest-confidence agent i∗ , which then emits a free-form public message ui∗,t with extracted spoken candidate c^i∗,t . The poll candidate is used only to allocate the turn, whereas the spoken candidate enters the public transcript. The audit treats the routing decision, the calibration of si,t , and the relationship between ci∗,t and c^i∗,t as three distinct objects.
Router
Expected success
Gap to oracle
Runtime confidence argmax
0.497
0.052
Top-confidence tie-random
0.493
0.055
Random valid candidate
0.490
0.059
Exact-answer majority vote
0.496
0.052
Confidence-weighted cluster vote
0.501
0.048
Learned router ( P(correct) )
0.499
0.050
Table 1: Router counterfactuals on the gpt-oss/math cell, over 9,653 all-routeable poll blocks. Five fixed-rule routers cluster within 1.1 pp, and a learned router trained to predict candidate correctness lands in the same band. The oracle ceiling is 4.8–5.9 pp above them all. For this broadcast protocol, the routing rule matters less than whether the block contains a correct candidate.
Calibrator
ECE ↓
Brier ↓
AUROC
mean conf.
Raw (as reported)
0.278
0.302
0.721
0.790
Temperature ( T=4.35 )
0.142
0.242
0.722
0.580
Isotonic
0.008
0.208
0.720
0.518
Table 2: Post-hoc recalibration on the gpt-oss /math cell, cross-fitted by problem_uid ( K=2 , fold assignment stratified by tier) over 23,391 valid poll candidates. ECE uses 10 equal-width bins. The isotonic procedure reduces ECE from 0.278 to 0.008 and Brier score from 0.302 to 0.208; candidate-level AUROC changes only from 0.721 to 0.720. These are held-out candidate-level estimates, not a rerun of the router on calibrated values.
Primary class
N
% of diverg.
Δ correct (pp, speak − poll)
self-revert
367
19.0
−11.4
incumbent-pull
120
0 6.2
+1.7
peer-echo
238
12.4
−5.0
novel
1,202
62.4
−9.1
Table 3: Sources of divergent spoken answers ( N=1,927 divergent valid pairs), by precedence-resolved primary class (self-revert ≻ incumbent-pull ≻ peer-echo ≻ novel). Negative Δ correct means the spoken revision is net-harmful on that subset. Overlapping (non-exclusive) match flags are in Appendix E .
Predictor (z-scored)
β
OR per SD
p
Pool-solvable rate (router-indep.)
+4.80
≈121
1.3⋅10−12
Uncond. improvement-event rate
+1.30
3.68
0.003
Tier difficulty
+0.39
1.48
5.1⋅10−4
Selected-invalid rate
+0.58
1.78
1.3⋅10−5
Divergence rate (PSDR)
+0.15
1.16
0.22
Uncond. degradation-event rate
+0.09
1.09
0.91
Table 4: Exploratory within-broadcast outcome linkage. Logistic regression with cluster-robust SEs by problem cluster ( n=4,181 , nclusters=421 ). Pool-solvable rate has the largest same-trace association. For mean selected confidence, the reported MLE is small and non-significant (OR 0.84, p=0.35 ), while sensitivity refits span OR 0.84–1.11 (Appendix I ); its sign is therefore not stable.
math
bio
bio
math
Actor
gpt-oss
gpt-oss
gemma
gemma
Valid poll cand.
23,391
2,912
4,852
14,129
Candidate acc.
0.518
0.593
0.491
0.491
Mean stated conf.
0.790
0.729
0.879
0.839
ECE raw
0.278
0.167
0.438
0.443
ECE isotonic
0.008
0.023
0.008
0.001
Table 5: Descriptive point estimates from the three-axis audit over a 2×2 actor × benchmark grid. The first column is the primary result. Routing rows are per-block success over all-routeable blocks; net commitment shift is the unconditional poll-to-speak correctness shift in percentage points. Correctness labels in all cells use the same fixed gpt-oss-120b equivalence judge, so the Gemma columns are cross-family-labeled. Per-cell confidence intervals are not reproduced in this preprint.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Axis
Metric
Estimate
95% CI
Denominator
Routing
CRA
0.506
[0.486, 0.527]
9,467 sel-valid blocks
Routing
HRWP
0.906
[0.896, 0.915]
5,295 possible-hit blocks
Routing
MCR
0.058
[0.052, 0.065]
7,970 eligible blocks
Routing
PCCA (tie-aware)
0.570
[0.549, 0.593]
1,184 blocks / 2,079 pairs
Routing
CTR
0.327
[0.305, 0.349]
2,079 corr-vs-wrong pairs
Routing
Selected-invalid rate
0.149
[0.139, 0.159]
11,122 selected blocks
Appendix
Table 6: Headline metrics with explicit denominators and problem-cluster bootstrap 95% CIs. Commitment metrics are reported both conditional on divergence and unconditional over all valid labeled selected pairs.
Stage
Visible context
Output contract
Downstream role
Poll
Problem, outer-loop evaluator feedback, current candidate answer, public discussion context, deferred private notes, own speaking history
Produces the candidate and confidence score used for turn allocation
Router
The per-agent poll outputs; runtime rule is confidence argmax with deterministic first-seen tie resolution
Selected speaker identity for the current turn
Allocates the floor; the top-confidence tie-random counterfactual reported in the main paper is the order-effect control
Speak
Selected agent sees the public transcript and system incumbent, but not peers’ private poll JSON
Free-form public message, optionally with a boxed answer
Public commitment that enters the shared transcript and may diverge from the polled candidate
Evaluator
Final submitted answer plus evaluator-only reference material
PASS/FAIL, correctness bit, non-leaking repair hint
Supplies the external correctness signal used by the original run and some post-hoc labels
Appendix
Table 7: Runtime protocol surface for the audited broadcast configuration. The central asymmetry is that the poll is structured and private to the router, while the speak step is free-form and public. This is the protocol fact behind the poll/speak commitment gap.
Setting
Audited value
Deliberation pool
Three homogeneous openai/gpt-oss-120b peers at temperature 0, plus an evaluator from the same model family
Discussion / approval schedule
Four discussion turns per outer round, followed by two approval rounds
Routing rule
Confidence argmax with deterministic first-seen tie resolution
Deferred-note threshold
Non-selected agents write deferred private notes only if their poll score is at least 65
Public-memory regime
summary_plus_recent with recent_history_window = 5
Approval / candidate update policy
Unanimous approval; reviewed candidate updated by majority_revision ; system incumbent persists until overridden
Appendix
Table 8: Load-bearing configuration knobs for the audited broadcast run. These settings matter because they determine what information the poll and speak stages actually see, how much context survives truncation, and how the system-level incumbent is updated between public speak events. They make the audit object more precise than “prompting style” alone.
Symbol
Definition
Intro
CRA
Candidate Routing Accuracy: fraction of selected-valid blocks where the routed candidate is correct
main paper
HRWP
Hit Rate When Possible: among blocks where any valid candidate is correct, fraction where the selected one is also correct
main paper
MCR
Missed Correct Rate: fraction of blocks where the selected valid candidate is wrong but a non-selected valid candidate is correct
main paper
PCCA
Pairwise Confidence-Correctness Alignment: block-macro average of each mixed block’s tie-aware correct–wrong comparisons; 0.5 means no within-block ordering signal
main paper
CTR
Confidence Tie Rate: fraction of within-block (correct, wrong) candidate pairs that are tied in confidence
main paper
ECE
Expected Calibration Error: weighted mean absolute gap between binned mean confidence and bin accuracy
main paper
Appendix
Table 9: Abbreviation glossary for the body and appendix metrics. Routers and router-counterfactual names ( runtime_argmax , top_confidence_tie_random , random_valid , majority_vote , confidence_weighted_vote , oracle ) are defined in-context in the main paper; the pool-solvable rate used in the outcome-linkage regression is also defined there.
Tier
HRWP
PCCA
CTR
PSDR
NPCS∣D
ECE
01
0.991
0.500
0.000
0.008
0.000
0.103
02
0.962
0.572
0.383
0.125
0.059
0.128
03 ∗
0.853
0.575
0.368
0.170
−0.250
0.181
04
0.936
0.587
0.353
0.166
−0.086
0.202
05
0.915
0.602
0.361
0.201
+0.022
0.292
06
0.871
0.578
0.307
0.233
−0.093
0.356
Appendix
Table 10: Per-tier routing, commitment, and calibration metrics. NPCS is conditional on divergent pairs. Tiers marked with ∗ have small denominators and should be interpreted with caution: tier 03 has 2 problem clusters and 47 valid pairs; tier 10 has 2 clusters and 54 valid pairs. Headline pooled estimates use cluster bootstrap over all 421 problem clusters.
Figure 3: Per-tier rank-versus-calibration split derived from Table 10 . Left: PCCA stays above the 0.5 no-signal floor across most tiers, but weakens toward harder slices. Right: ECE climbs sharply with difficulty, reaching the 0.31–0.41 range on tiers 06–10. Hollow markers indicate the small-denominator tiers 03 and 10. The point of the figure is not that either curve is perfectly monotone; it is that the probability interpretation degrades faster than the rank signal.
Model
Term
Coef.
95% CI
OR
p
Correctness
score (norm)
6.551
[5.844, 7.258]
700.1
1.18⋅10−73
Correctness
score × difficulty
−1.993
[ −2.303 , −1.683 ]
0.136
2.44⋅10−36
Correctness
round index
−0.903
[ −1.022 , −0.785 ]
0.405
1.26⋅10−50
Divergence
poll correct
−1.171
[ −1.333 , −1.009 ]
0.310
1.23⋅10−45
Divergence
poll correct × difficulty
0.254
[0.181, 0.328]
1.290
8.85⋅10−12
Divergence
score (norm)
−1.208
[ −1.886 , −0.530 ]
0.299
4.83⋅10−4
Appendix
Table 11: Cluster-robust pooled logistic regressions with agent fixed effects. Correctness model uses 23,391 valid poll candidates; divergence model uses 9,436 valid selected poll-speak pairs. Both use cluster-robust SEs by problem cluster ( nclusters=421 ). The outcome-linkage regression reported in the main paper is fit on the 4,181-trace level.
Term
Coef.
OR
p
role = self-prior-speak (vs. peer)
−0.010
0.99
0.89
role = incumbent (vs. peer)
+0.519
1.68
8.1⋅10−3
lag (rounds back)
+0.004
1.00
0.95
source confidence (analyst covariate)
+1.471
4.35
7.3⋅10−4
source was correct (vs. gold)
+0.314
1.37
4.2⋅10−4
candidate is most-recent (1/0)
+0.198
1.22
4.7⋅10−2
Appendix
Table 12: Adoption logit. One row per (divergent speak event, visible candidate), with adopted=1 when the spoken answer matches that candidate. Cluster-robust SEs use problem_uid . Source confidence is post-hoc metadata from the candidate’s originating private poll; it was not visible to the speaker, so its coefficient is an association rather than a confidence-visibility effect.
Match flag (non-exclusive)
N
% of diverg.
matches_self_revert
367
19.0
matches_incumbent_pull
224
11.6
matches_peer_echo
550
28.5
No match (novel)
1,202
62.4
Appendix
Table 13: Independent match flags over the same N=1,927 divergent valid pairs as the precedence-resolved primary-class table in the main paper. A single divergent spoken answer can fire more than one flag, so the counts sum to more than 1,927 and the percentages to more than 100 . The flag-versus-primary gap sits almost entirely in peer-echo: a peer’s prior public answer matches 28.5% of divergences, but only 12.4% survive as primary peer-echo, because the other 312 also match the speaker’s own earlier answer or the system incumbent, which take precedence. Self-revert and novel are unaffected by precedence, so their flag and primary counts coincide. Read against the primary-class table, this bounds how much divergence could be attributed to peer anchoring under a maximally permissive attribution rule.
Concern
Check
Main result
Where
Confidence ranking is just noise; PCCA / CTR are artifacts of the metric
Within-block confidence permutation null and noisy-oracle synthetic on the same trace data, no model re-runs
Permutation drives PCCA to 0.500 and changes CTR from 0.327 to 0.300±0.006 ; noisy oracle drives PCCA toward 1 and CTR toward 0 . This validates PCCA’s no-signal reference, not CTR invariance.
App. Table 15
Self-judge artifact: gpt-oss-120b judges its own outputs and may bias NPCS
Two-judge relabeling of 4,379 residual pairs with google/gemma-4-31B-it ( κ=0.82 ) and google/gemma-4-E4B-it ( κ=0.75 )
Conditional NPCS over the 9,263-pair tiers 02–09 population has the same negative sign under all three residual-label variants ( −0.083 , −0.061 , −0.060 ).
App. H
Outcome linkage depends on a single regression specification
Fit a related linkage logit with selected-correct rate and routing slack entered separately, using the same remaining predictors and cluster-robust SEs
The same-trace content variables remain strongly associated with outcome, but neither specification is interpreted causally.
App. I
Single benchmark only (olympiad math throughout)
Repeat the audit on LAB-Bench biology ( 15 ) for both actors, using the same extraction pipeline and fixed judge
Overconfidence and scale repair recur, while routing and commitment effects differ: raw-confidence argmax trails random-valid selection, and unconditional commitment shifts are positive in the biology cells.
App. J
Single actor family only ( gpt-oss-120b deliberators throughout)
Cross-family replication with google/gemma-4-31B-it deliberators on both benchmarks and a fixed gpt-oss-120b judge
Gemma polls parse at 98.4%; scale correction lowers ECE, while raw AUROC remains weak (0.537) or inverted (0.440), and raw-confidence argmax trails random-valid selection.
App. J
Appendix
Table 14: High-priority robustness and scope checks. The table distinguishes descriptive sensitivity from causal or deployment claims; no row turns a same-trace association into an intervention estimate.
Construction
CRA
HRWP
PCCA
CTR
Runtime (headline reproduction)
0.506
0.906
0.570
0.327
A: permutation null ( n=50 reps)
0.520 ± .002
0.762 ± .003
0.500 ± .011
0.300 ± .006
B: noisy oracle, σ=0.05
0.622 ± .001
1.000
1.000
0.000
B: noisy oracle, σ=0.10
0.620 ± .001
1.000
1.000
0.000
B: noisy oracle, σ=0.20
0.621 ± .001
1.000
1.000
0.000
B: noisy oracle, σ=0.50
0.605 ± .001
0.961 ± .002
0.913 ± .009
0.023 ± .004
Appendix
Table 15: Sanity-check constructions over the existing traces. The permutation null confirms that runtime PCCA is meaningfully above the no-signal floor, while the noisy oracle confirms that PCCA saturates under a clean signal. CRA remains capped because selected-valid accuracy cannot exceed the underlying content ceiling.
Judge pair
Pairs decided
Agreement
Cohen’s κ
Confusion (judge1: gpt-oss / judge2)
gpt-oss vs gemma-4-31B-it
4,048
0.908
0.816
1776/311/62/1899
gpt-oss vs gemma-4-E4B-it
3,945
0.873
0.748
1670/433/68/1774
Appendix
Table 16: Two-judge cross-check on full residual corpus. Confusion entries are (judge1=div, judge2=div) / (judge1=div, judge2=equ) / (judge1=equ, judge2=div) / (judge1=equ, judge2=equ). Both Gemma judges show an asymmetric disagreement pattern: when gpt-oss flagged a pair as divergent, the Gemma judge “rescues” it as equivalent more often (311 / 433) than the opposite direction (62 / 68).
Figure 4: Asymmetric disagreement in the cross-judge audit. On residual pairs where gpt-oss and Gemma disagree, the dominant observed direction is gpt-oss says divergent, Gemma says equivalent . The figure describes disagreement direction; it does not determine which label is correct.
Metric
gpt-oss-120b tiers 02–09
gemma-4 31B-it
gemma-4 E4B-it
Divergent pairs
1,921
2,418
2,537
PSDR
0.207
0.261
0.274
PSIR (cond. divergence)
0.174
0.132
0.126
PSDRg (cond. divergence)
0.257
0.192
0.186
NPCS∣D
−0.083
−0.061
−0.060
Appendix
Table 17: Commitment metrics under three residual-case label variants over the same 9,263 valid pairs from tiers 02–09. PSIR, PSDRg, and NPCS∣D use the judge-specific divergent denominator shown in the first row. Conditional NPCS has the same negative sign across variants; this is a sensitivity check, not ground-truth adjudication.
Predictor (z-scored)
β
OR per SD
p
Selected-correct rate
+5.94
≈380
3.1⋅10−11
Uncond. improvement-event rate
+1.40
4.08
0.002
Routing slack (oracle − argmax)
+1.08
2.96
9.0⋅10−4
Tier difficulty
+0.42
1.52
2.9⋅10−4
Selected-invalid rate
+0.60
1.81
9.1⋅10−6
Divergence rate (PSDR)
+0.19
1.21
0.10
Appendix
Table 18: Related decomposition specification with selected-correct rate and routing slack entered separately. Selected-correct rate is mechanically closer to the broadcast outcome than pool-solvable rate. McFadden pseudo- R2 is 0.802 here and 0.798 in the collapsed specification. Both use same-trace summaries and are interpreted descriptively.
math
bio
bio
math
Actor
gpt-oss
gpt-oss
gemma
gemma
Valid poll cand.
23,391
2,912
4,852
14,129
Candidate acc.
0.518
0.593
0.491
0.491
Mean stated conf.
0.790
0.729
0.879
0.839
ECE raw
0.278
0.167
0.438
0.443
ECE isotonic
0.008
0.023
0.008
0.001
Appendix
Table 19: Descriptive point estimates over the 2×2 actor × benchmark grid. The first column is the primary result. Routing rows are per-block success over all-routeable blocks; net commitment shift is the unconditional poll-to-speak correctness shift in percentage points. Correctness labels in all cells use the same fixed gpt-oss-120b equivalence judge. Per-cell confidence intervals are not reproduced in this preprint.
Mode
Pass rate
Avg. tokens
Avg. calls
Avg. wall (s)
baseline_llm
0.568
18.4K
9.69
43.6
single_agent
0.788
48.1K
18.0
74.3
broadcast
0.892
616.5K
134.6
294.9
Appendix
Table 20: Task-level context from the same broader verifier-gated math evaluation over the same 4,181 problems. Here baseline_llm is a single direct LLM solve, single_agent is one LLM with multi-turn self-revision, and broadcast is the multi-agent protocol audited in the body. Broadcast outperforms the same-task single_agent comparator in final pass rate (0.892 vs. 0.788), but at much higher average cost: 12.8 × tokens, 7.5 × model calls, and 4.0 × wall time. These rows are contextual protocol comparisons, not equal-compute frontiers for the broadcast trace audit. Aggregates are weighted across tier-level summaries from that evaluation.
Field
R/O
Description
Protocol identity
protocol_name
R
Short name for the deliberation protocol.
protocol_version
R
Version tag.
paper_of_record
O
Citation for protocol definition.
Inference setup
agent_model
R
Model id(s); homogeneous or heterogeneous.
Appendix
Table 21: Trace-Audit Card v0.1 schema. The card describes the measurement properties of a deliberation protocol; it does not describe a single model (model cards do that) or a dataset (data cards do that).
Multiagent language model systems, such as collaborative reasoning and debate, produce multiple correlated candidate answers and confidence signals. However, these signals are usually calibrated only at the individual agent level, and provide no principled confidence estimate for the system's final answer. We formulate this as a confidence composition problem where combining confidence across agents and reasoning stages while preserving both selective utility and probabilistic reliability. We study confidence-aware routing and log-odds pooling protocols that select among candidate answers and output a system-level confidence. Across five benchmarks, 30 heterogeneous and homogeneous model pairs, and two confidence estimators, our gated-fusion methods improve AUARC and reduce Brier score over single agent, standard debate, and selective debate baselines, while retaining competitive weighted F1-score as a correctness metric. We further show that our log-odds fusion is overconfident due to correlated intermediate signals. We propose a shared dependence discount that substantially improves reliability while preserving predictions.
A deployed LLM agent emits tool calls, queries, and code that can be silently wrong -- by the time the error surfaces, the action has run. Frontier chat APIs hide the model's token probabilities; the agent's stated confidence barely beats chance on the mistakes that matter; and resampling does not help, since frontier models are highly repetitive, reproducing the same call across samples. We recover the missing signal from a low-cost open-weight surrogate run in parallel. It reads the same context, schema, and proposed action as the agent, then scores the call from its own log-probabilities through a family of complementary readouts: teacher forcing and request-PMI weigh the likelihood of each argument value, a discriminative verdict judges the call as a whole, and tool-choice competition tests the function against its siblings. One principle says which to trust: a generative likelihood localizes wrong argument values, while the verdict catches holistically wrong calls. When the error type is unknown, an ensemble is the low-regret default. The readout is training-free, needs no access to the agent's internals, and costs one prefill pass alongside the tool call. On difficult coding tasks it reaches AUROC 0.825 where the actor's stated confidence is near chance (0.598), and the generative readouts beat it by +0.07 to +0.28 across three further actors. Against self-consistency it gains +0.14 to +0.19 on near-deterministic actors, at 1/K the cost. The signal drives two deployment modes: a real-time gate escalating the least-trustworthy calls for review (+0.05 to +0.30 accepted-action accuracy at 50% coverage), and confidence feedback, returning the tool result with the score so the agent adapts its next step -- lifting task success on live-execution benchmarks (+0.119 and +0.137, p <= 1e-4) and beating a random-value control where step errors are silent (+0.078, p = 0.003).
When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads. This paper presents MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a runtime calibration method that learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set. MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate. The corrected scores weight candidate answers in a collective decision. Evaluation covers code generation, question answering, and mathematics, using an 18-model pool and a nine-model subset for distribution-shift experiments. On BigCodeBench, model-mean confidence is negatively related to accuracy; among correct/incorrect response pairs, choosing the more confident responder performs below chance. Against five online calibration baselines receiving identical feedback and retaining their learned state across each transition, MARGIN achieves lower post-shift expected calibration error than all five in two code-generation transitions and than four in a question-answering transition; the remaining question-answering comparison is inconclusive. In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting. These results support model-specific runtime calibration for coordination under changing workloads when correctness feedback is available for the participating responders.