Measuring Collapse and Correction in Homogeneous-Panel LLM Debate

Xin Li; Mengbing Liu; Chau Yuen

cs.CLSep 28, 2026

Measuring Collapse and Correction in Homogeneous-Panel LLM Debate

Authors: Xin Li, Mengbing Liu, Chau Yuen

Organizations: Nanyang Technological University

Abstract

Multi-agent large language model (LLM) debate is often evaluated by whether final answers improve, but movement is not necessarily improvement: the same discussion can rescue an initially wrong majority or destroy an initially correct one. Standard final-accuracy evaluations conflate these opposing mechanisms. We introduce an auditable protocol for homogeneous debate on multiple-choice questions (MCQs) that records each run as a transition ledger over collapse, correction, onset, and signed intervention utility. On 6,925 MMLU-Pro debates, the protocol identifies 253 collapses and a parallel correction ledger that changes how interventions should be judged. Replay experiments reveal the central tradeoff: a leave-one-model-out probe-gated freeze prevents 29 collapses but loses 108 corrections under equal weights, so collapse prevention alone can recommend the wrong policy. A compact pre-debate 8-probe screen is a triage signal: its unadjusted family-level association with conditional-collapse risk is high (G=7, Spearman rho=0.893, exact two-sided p=0.0123), but initial-majority accuracy is a close comparator (rho=0.821; family partial rho=0.767, p=0.0877), so we do not treat it as calibrated or capability-adjusted prediction. Round-level traces localize many collapses to the first debate round, where early disagreement can precede both harmful cascades and useful recovery. We release replayable schemas, coders, audits, cost cards, and zero-API rebuild scripts so future model-scaffold rows can be compared under the same denominators and signed utility ledger.

Figures & tables

Figure 1 : Debate movement is not necessarily improvement. Left: the same debate scaffold can collapse an initially correct majority or correct an initially wrong one. Middle: the audit combines a pre-debate screen with round-level traces to localize where transition risk appears. Right: signed replay scores an intervention by both collapses prevented and corrections lost, so collapse prevention alone can select the wrong policy.

Figure 2 : Eight-probe instrument and triage workflow. Left: a target agent first answers the multiple-choice question (MCQ). Middle: the same question is re-asked under an $8$ -probe factorial crossing two social-channel settings with four argument-channel strengths. Right: $\alpha_{\text{tot}}$ is the total flip rate used for family-level risk ranking, while splitting rows by initial correctness yields adversarial flips ( $\alpha_{\text{adv}}$ ) and corrective flips ( $\alpha_{\text{cor}}$ ).

Layer	New-row input	Rebuilt outputs and boundary
Selection	Model/backend, pool id, $8$ -probe rows	$\alpha_{\mathrm{tot}}$ , channel split, parser/cost audit, and capability comparison; aggregates open, tuned convince-wrong text gated
Accounting	Debate JSON Lines (JSONL) with Round $0$ and final answers	Initial/final accuracy, transition counts, conditional denominators, and intervals; no LLM judge labels
Localization	Round traces or derived Round $1$ feature matrix	Onset and Round $1$ diagnostics; transcript-derived matrices may be gated, but missingness is explicit
Replay	Gate decisions, held-out split, weights	$(\mathrm{prevented},\mathrm{lost},\mathrm{net})$ ledger, break-even ratio, and provenance; oracle and collapse-only scores are separated from held-out replay

Table 1 : Reusable audit surface. A new model–scaffold row can be added without changing the measurement definitions; release flags state which fields are open versus gated.

Quantity	Value	Scope
Pre-debate screening
split-half / test–retest / intraclass corr. (ICC)	$0.873$ ; $0.760$ ; $0.630$ – $0.633$	probe
$\alpha_{\text{tot}}$ vs. $C^{\mathrm{cond}}$ Spearman	$0.893$ ( $p{=}0.0123$ )	$G{=}7$
initial-accuracy comparator; partial after init-acc	$0.821$ ; $0.767$ ( $p{=}0.0877$ )	$G{=}7$
Runtime diagnosis and localization
disagreement / probe-FR / S/A AUC	$0.55$ – $0.86$ ; $0.46$ – $0.74$ ; $0.39$ – $0.58$	per question

Table 2 : Roadmap of the main numerical results. The table groups the pre-debate screening, runtime localization, signed replay, and transfer stress-test quantities that are interpreted in the corresponding result sections. Scope gives the analysis unit or denominator; the

G{=}7

association is the primary analysis, and GPQA rows are post-hoc stress checks.

Family	Models	$\boldsymbol{\alpha_{\text{tot}}}$	$\boldsymbol{C^{\text{cond}}\%}$
DeepSeek	$1$	$0.113$	$\phantom{0}0.00$
Google	$2$	$0.288$	$\phantom{0}0.29$
OpenAI	$2$	$0.346$	$\phantom{0}1.52$
Anthropic	$1$	$0.450$	$\phantom{0}2.15$
Phi	$1$	$0.469$	$10.24$
Qwen	$6$	$0.699$	$19.75$

Table 3: Family means for the primary

G{=}7

rank analysis. Qwen contributes

6/14

model rows, so related variants are averaged before the exact Spearman test; the Meta row is the visible upper-tail inversion. Unit: model family;

C^{\text{cond}}

uses initially correct majorities as its denominator; primary analysis.

Predictor	Role	Model-row $\rho$	Family $\rho$
$8$ -probe $\alpha_{\mathrm{tot}}$	pre-debate screen	$+0.829$	$+0.893$
Initial-majority accuracy	capability proxy	$-0.803$	$-0.821$
Capability pressure ( $1{-}$ init acc)	free risk proxy	$+0.803$	$+0.821$
Raw debate-revision proxy	revision-quantity baseline	$+0.456$	$+0.714$
Social-over-solo lift	narrow conformity proxy	$+0.026$	$-0.071$
Final debate accuracy	post-debate descriptive control	$-0.851$	$-0.679$

Table 4 : Comparator and confound bake-off against conditional collapse on the realized

N{=}14

cohort. Family columns collapse related variants before ranking; post-debate rows are descriptive controls, not inputs to the pre-debate triage claim. Units: model row (

N{=}14

) and family (

G{=}7

); sensitivity analysis.

Predictor	Gemini	Haiku	GPT	Pooled
	( $N{=}990$ )	( $N{=}500$ )	( $N{=}494$ )	( $N{=}994$ )
Mean probe FR	0.455	0.626	0.735	0.658
S/A ratio	0.394	0.473	0.576	0.529
Initial disagreement	0.765	0.550	0.858	0.573
Combined (FR+Diff+Disagree)	0.845	0.719	0.931	0.762
Combined (+S/A)	0.843	0.719	0.933	0.765

Table 5 : Per-question diagnostic AUCs after initial answers are available. Initial disagreement dominates the probe-derived quantities, and adding S/A changes AUC by at most

0.004

; this bounds the screen as model-level triage rather than a row-level oracle. Unit: question; diagnostic analysis.

Model	$N_{\text{debate}}$	Collapses	Round $1$	Round $2$	Round $3$
Llama-3.1-8B	2,003	86	43	21	22
Phi-4-mini	2,161	112	67	17	28
Qwen3-4B	2,161	50	35	9	6
Qwen3-8B	200	3	3	0	0
Gemini 3-flash	200	1	1	0	0
GPT-5.4-mini	200	1	0	0	1

Table 6 : Collapse onset by debate round. Across

6{,}925

debates and

253

collapses,

58.9\%

of collapses originate in Round

1

. Unit: collapse (denominator

253

); descriptive.

Policy	Cohort	Valid.	$\Delta$ acc	Prev.	Lost	Net	$r^{\star}$
Probe-gated freeze	$6{,}525$ OSS	LOMO	$-1.21$ pp	$29$	$108$	$-79$	$3.72$
Round $1$ majority change	$1{,}255$ traces	fixed replay	$-1.04$ pp	$56$	$69$	$-13$	$1.23$
Learned Round $1$ stump	$1{,}255$ traces	strict LOMO	$-2.31$ pp	$2$	$31$	$-29$	$15.50$

Table 7 : Freeze-style replay. All rows are net-negative at equal weights;

r^{\star}

is the break-even value ratio. Unit: debates in each row’s cohort; the cohorts differ, so rows are not a matched comparison.

Task	Model (mode)	$n$	$C/n_{+}$	$K/n_{-}$	Accuracy (%)	Private $-$ peer
MMLU-Pro	Qwen3-8B (thinking)	1,000	15/716	17/264	71.60 / 72.50	$-8\;[-20,4]$
MMLU-Pro	Qwen3-32B (thinking)	1,000	9/754	7/212	75.40 / 76.30	$-13\;[-26,0]$
MMLU-Pro	R1-Distill-Llama-8B †	1,000	87/472	42/344	47.20 / 47.20	$-44\;[-70,-18]$
MMLU-Pro	gpt-oss-20b (low effort)	2,000	43/1384	78/488	69.20 / 73.25	$-26\;[-48,-4]$
MMLU-Pro	gpt-oss-20b (medium effort) †	2,000	33/1491	49/402	74.55 / 76.95	$-22\;[-42,-2]$
MATH-L5	R1-Distill-Llama-8B	1,324	72/1037	22/80	78.32 / 81.42	$-29\;[-56,-3]$

Table 8 : Reasoning-mode rows from the post-hoc follow-up (peer debate with shared Round

0

; ties coded as undecided).

C/n_{+}

and

K/n_{-}

are collapses and corrections with their initial-state denominators; accuracy is Round

0

/ Round

3

. Private

-

peer is in correct-answer counts with a paired

95\%

interval.

\dagger

: amended-only QC. All eleven rows are in Table 44 .

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Term	Meaning	Role in this paper
$\alpha_{\text{tot}}$	Total $8$ -probe flip rate	Locked pre-debate screen used for the rank audit
$\alpha_{\text{adv}}$ / $\alpha_{\text{cor}}$	Adversarial / corrective probe flip rates	Channel decomposition; not a standalone safety leaderboard
$\alpha_{\text{neutral}}$	Non-adversarial or neutral probe lane where available	Diagnostic channel, not the headline predictor
S/A	Social-over-argument balance or ratio	Construct diagnostic; weak as a per-question oracle
FR	Flip rate	Generic answer-revision rate in probes or debates, depending on table context
AA / AS / D	Anti-argument / anti-social / default prompt conditions	Channel-separability manipulation

Appendix

Table 9 : Compact glossary for notation, cohorts, and short labels used across the paper.

Tier	Contents	License/access	Maintenance note
Open code and derived tables	Evaluation scripts, parser, aggregate tables, figures, zero-API rebuild outputs, reuse-card schema, and Round $1$ derived-matrix schema	MIT for code; CC-BY 4.0 for tables/schema	Immutable release tag plus DOI archive
Open low-risk probes	Initial-answer prompts and non-social weak/moderate/strong probe templates	CC-BY 4.0 subject to upstream benchmark terms	Versioned as alpha-tot-v1.0
Gated probe/trace bundle	Convince-wrong templates, tuned phrasings, full debate transcripts; exact Round $1$ matrix until a transcript-free derived matrix is staged	Research-use click-through license with no redistribution or fine-tuning for user-facing persuasion systems	Access log and snapshot metadata retained
Metadata and documentation	Datasheet-style card, Croissant/RAI metadata, provenance hashes, parser-failure notes, cost card	Open with the aggregate artifact	Updated only by new versioned release, not in-place edits

Appendix

Table 10: Artifact release tiers. Licenses and access rules differ by tier because code, aggregate tables, and gated traces carry different reuse risks.

Mode	Generations on 200 questions	Use and caveat
Alpha-lite triage	$200$ initial answers $+$ $1{,}600$ probe completions	Cheapest screening pass; estimates $\alpha_{\text{tot}}$ under one condition/agent and should be followed by richer logging if high.
Paper-full screen	$1{,}800$ initial answer rows $+$ $14{,}400$ probe completions	Full $3$ -condition $\times$ $3$ -agent psychometric estimate used for the reported stress rows; more stable, but not always cheaper than a small debate sweep.
Direct debate audit	Approximately $3{,}000$ debate completions	Measures $C^{\text{cond}}$ and correction directly on the same 200-question slice, but only after paying for multi-agent traces.

Appendix

Table 11: Operational cost card for instrumenting one new model on a 200-question MCQ slice. “Completion” counts one model generation; dollar cost depends on backend and output length.

Card field	Example value	Source / release note
Identity and pool	Phi-4-mini, local vLLM, homogeneous 3-agent standard debate; MMLU-Pro high-FR pool	Aggregate row open; model-specific pool selected outcome-blind by probe behavior.
Selection screen	$\alpha{=}0.437$ , $\alpha_{\mathrm{adv}}{=}0.356$ , $\alpha_{\mathrm{cor}}{=}0.092$ , S/A $=+0.23$ over $1{,}800$ probe rows	Probe templates and aggregate fields open; tuned convince-wrong phrasing is gated.
Transition table	$2{,}161$ debates; initial accuracy $50.6\%$ , final accuracy $53.5\%$ , $C^{\mathrm{cond}}{=}10.24\%$ , correction $16.31\%$	Full transcripts are gated; aggregate collapse/correction counts are open.
Signed utility	LOMO probe-gated freeze at $\tau{=}0.70$ , $w_{\mathrm{coll}}{=}w_{\mathrm{corr}}{=}1$ : $4$ prevented, $13$ lost, net $-9$ ( $-0.42$ pp)	Policy row is open; the strict matched- $\tau$ DG/DRS matrix is not in the open checkout.
Release flags	Aggregates open; full transcripts restricted for public redistribution; exact Round $1$ matrix gated unless a transcript-free derived matrix is supplied	State missing or gated matrices explicitly before reuse.

Appendix

Table 12: Populated reuse-card example for one sealed OSS row. This compact paper-form row mirrors the machine-readable reuse-card schema.

Trace family	Rows / states	None count	Rate / note
DeepSeek-v4-flash debate states	$3{,}000$	$264$	41 init-None debates; 18 final-None debates
Gemma-4-31B alpha post-answers	$14{,}400$	$864$	$6.00\%$
Gemini 3.1 Flash-Lite alpha post-answers	$14{,}400$	$677$	$4.70\%$
Qwen3.5-4B alpha post-answers	$14{,}320$	$576$	$4.02\%$
Qwen3.6-35B-A3B debate states	$3{,}000$	$15$	8 answer/final-tag mismatches

Appendix

Table 13: Parser audit for realized-cohort traces. Counts are state-level answer-extraction failures in debate traces and post-answer extraction failures in alpha traces.

Trace	Rows	Timestamp / metadata	SHA256 prefix
Gemini 3.1 Flash-Lite debate	$200$	2026-04-26T23:53:42–23:56:02; Gemini API model recorded	2cfa125250f6
Qwen3-32B local-vLLM debate	$200$	2026-04-27T09:22:56–11:21:36; local vLLM, qwen3-32b-a7-local	9a3c0db8c1a7
Mistral Small 4 OpenRouter debate	$20$	2026-04-27T08:59:28–09:04:00; OpenRouter, Mistral Small 4	991e3baae9e8
Gemma-4-31B alpha lane	$1{,}800$	older local lane; partial metadata	eb7fcb5a6f27

Appendix

Table 14: Representative provenance entries from the reproducibility artifact. SHA prefixes support trace matching without exposing host metadata.

Predictor of $C^{\text{cond}}$	Spearman $\rho$	Fisher-z $95\%$ CI	$p_{\text{raw}}$	$p_{\text{Holm}}$
Probe $\alpha_{\text{tot}}$ (8-probe total) ⋆	$+0.78$	$[+0.25,\,+0.95]$	$0.013$	n/a
$\alpha_{\text{adv}}$ (right ${\to}$ wrong $\mid$ init correct)	$+0.62$	$[-0.08,\,+0.91]$	$0.077$	$0.385$
$\alpha_{\text{cor}}$ (wrong ${\to}$ right $\mid$ init wrong)	$-0.20$	$[-0.76,\,+0.54]$	$0.606$	$0.606$
Debate FR (revision quantity in debate)	$+0.33$	$[-0.43,\,+0.81]$	$0.391$	$0.783$
Initial accuracy	$-0.73$	$[-0.94,\,-0.13]$	$0.025$	$0.175$

Appendix

Table 15: Sealed pre-A6

N{=}9

rank-correlation cross-check. The total-

\alpha

row is the original sealed primary test; secondary rows use Holm–Bonferroni correction across

m{=}7

diagnostics. The realized

N{=}14

headline is in Table 23 .

Model	N ${}_{\text{deb}}$	Init. Acc.	Debate FR	Probe $\alpha$	Raw Coll. %	Cond. Coll. % [ $95\%$ CI]	S/A
Sonnet 4.5	120	77.5%	0.705	0.428	1.67	2.15 [0.59, 7.51]	$+0.15$
Haiku 4.5	500	82.2%	0.512	0.491	9.20	11.19 [8.50, 14.61]	$-0.03$
GPT-4o-mini	498	68.5%	0.458	0.337	1.61	2.35 [1.19, 4.56]	$+0.05$
Gemini 2.5 Flash ∗	200	36.5%	0.648	n/a	0.50	1.37 [0.24, 7.36]	$+0.14$
GPT-5.4-mini †	200	71.5%	0.350	0.346	0.50	0.70 [0.12, 3.85]	$+0.05$
Gemini 3-flash †	200	85.0%	0.249	0.273	0.50	0.59 [0.10, 3.26]	$-0.08$

Appendix

Table 16: Sealed pre-A6

N{=}9

per-model cross-check. Conditional-collapse values include Wilson

95\%

intervals, while raw collapse percentages mix initial accuracy with revision. The realized

N{=}14

model-row view is in Table 22 .

Model	S/A (default)	95% CI
Haiku 4.5	$-0.030$	[ $-0.100$ , $+0.030$ ]
Sonnet 4.5	$+0.154$	[ $+0.092$ , $+0.215$ ]
Sonnet 4.6	$-0.074$	[ $-0.186$ , $+0.038$ ]
Opus 4.5	$-0.040$	[ $-0.117$ , $+0.039$ ]
Opus 4.6	$+0.113$	[ $+0.020$ , $+0.206$ ]
GPT-4o-mini	$+0.046$	[ $-0.034$ , $+0.126$ ]

Appendix

Table 17: S/A ratio bootstrap intervals. Intervals are

95\%

CIs from

10{,}000

bootstrap iterations on the default condition, using the same high-FR-question S/A values as Table 16 .

Model	$\alpha$	S ${}_{\text{sens}}$	A ${}_{\text{sens}}$	S/A	InitAcc
Opus 4.5	0.266	$-0.019$	$+0.108$	$-0.152$	85.2%
Opus 4.6	0.273	$+0.038$	$+0.124$	$+0.232$	85.8%
Sonnet 4.5	0.428	$+0.056$	$+0.189$	$+0.230$	83.6%
Sonnet 4.6	0.127	$-0.012$	$+0.052$	$-0.192$	83.8%
Gemini 3-flash	0.271	$+0.035$	$+0.100$	$+0.263$	84.3%
Gemini 3.1-flash-lite	0.293	$+0.031$	$+0.098$	$+0.241$	81.9%

Appendix

Table 18: Shared-pool S/A on the 195-question model intersection.

\alpha

is pooled probe flip rate, S

{}_{\text{sens}}

and A

{}_{\text{sens}}

are pre-normalization contrasts, and InitAcc is initial-correct rate on the shared pool.

Model	$\alpha$	$\alpha_{\text{adv}}$	$\alpha_{\text{cor}}$	$\alpha_{\text{neutral}}$	$\alpha$ (soc)	$\alpha_{\text{adv}}$ (soc)
GPT-4o-mini	0.337	0.266	0.113	0.289	0.348	0.267
GPT-5.4	0.121	0.057	0.163	0.220	0.139	0.066
GPT-5.4-mini	0.346	0.273	0.115	0.393	0.354	0.273
GPT-5.4-nano	0.287	0.176	0.134	0.336	0.354	0.230
Gemini 3-flash	0.273	0.263	0.067	0.258	0.256	0.240
Gemini 3.1-flash-lite	0.310	0.296	0.099	0.281	0.327	0.309

Appendix

Table 19: Probe flip-rate decomposition by initial correctness.

\alpha_{\text{adv}}

is right-to-wrong revision conditional on initial correctness, and

\alpha_{\text{cor}}

is wrong-to-right revision conditional on initial error; the final columns repeat the social-condition values.

Model	$N$	FR(D)	FR(AS)	FR(AA)	Order	$d$ (AA $-$ AS)	$p$
Anthropic (5 models, mean $d=0.208$ )
Haiku 4.5	450	.178	.175	.234	$\checkmark$	0.202	9.2e-2
Sonnet 4.5	1,800	.443	.355	.501	$\checkmark$	0.430	2.5e-13 ⋆
Sonnet 4.6	1,799	.136	.102	.149	$\checkmark$	0.188	2.4e-4 ⋆
Opus 4.5	1,800	.319	.234	.257	$\sim$	0.069	1.1e-1
Opus 4.6	1,745	.293	.240	.287	$\sim$	0.151	8.8e-3 ⋆

Appendix

Table 20: S/A channel manipulation across 16 models. D = default, AS = anti-social, and AA = anti-argument;

N

counts valid agent-condition records. A

\checkmark

denotes strict AA

>

>

AS,

\sim

denotes AA

>

AS without strict ordering, and

\star

marks Holm–Bonferroni significance over 16 paired tests.

Lane	Family	$N$	post-None	FR(D)	FR(AS)	FR(AA)	$d$ (AA $-$ AS)	$p$	Status
Mistral Small 4	Mistral	1,800	$2.18\%$	.421	.066	.645	2.549	$2.0{\cdot}10^{-264}$	near-complete
Llama 3.3 70B Instruct	Meta	1,797	$4.00\%$	.470	.192	.580	1.306	$2.4{\cdot}10^{-131}$	near-complete
Llama 4 Scout	Meta	1,800	$9.99\%$	.434	.426	.574	0.540	$3.1{\cdot}10^{-35}$	near-complete; parse QC
Llama 4 Maverick	Meta	1,794	$9.17\%$	.573	.622	.680	0.234	$1.7{\cdot}10^{-8}$	near-complete; parse QC
Grok 4.1 Fast	xAI	1,799	$0.23\%$	.183	.068	.184	0.497	$1.3{\cdot}10^{-30}$	near-complete
HY3 Preview Free	Tencent	1,785	$0.93\%$	.312	.083	.402	1.037	$1.4{\cdot}10^{-95}$	near-complete; preview

Appendix

Table 21: Post-hoc OpenRouter alpha-channel extension. D = default, AS = anti-social, and AA = anti-argument;

N

counts valid agent-condition records out of a nominal

1{,}800

. The post-None column reports parsed post-answer failures, and these rows are excluded from the headline association.

Model	Family	$\alpha_{\text{tot}}$	$C^{\text{cond}}\%$ [Wilson 95%]
sonnet-4.5	Anthropic	$0.4504$	$\phantom{0}2.15\,[0.59,\,7.51]$
deepseek-v4-flash §	DeepSeek	$0.1129$	$\phantom{0}0.00\,[0.00,\,2.51]$
gemini-3-flash	Google	$0.2644$	$\phantom{0}0.59\,[0.10,\,3.26]$
gemma-4-31b-it-awq	Google	$0.3106$	$\phantom{0}0.00\,[0.00,\,2.34]$
llama-3.1-8b	Meta	$0.7737$	$\phantom{0}8.46\,[6.91,\,10.34]$
gpt-4o-mini	OpenAI	$0.3424$	$\phantom{0}2.35\,[1.19,\,4.56]$

Appendix

Table 22 : Realized

N{=}14

model rows.

C^{\text{cond}}

includes Wilson

95\%

intervals; these rows are the model-row sensitivity view, not the primary inferential unit.

Statistic	Value
Family mean / median aggregation	$\rho{=}{+}0.893$
Family exact permutation $p$	two-sided $0.0123$
Family LoFO worst case	$\rho{=}{+}0.829$
Meta+Qwen joint-drop check	$\rho{=}{+}1.000$ ( $G{=}5$ , descriptive)
$N{=}14$ model-row estimate	$\rho{=}{+}0.829$
Family-clustered bootstrap on model rows	$95\%$ CI $[+0.386,+0.919]$

Appendix

Table 23: Family rank analysis and model-row sensitivity checks.

Grouping rule	Groups	Spearman $\rho$	Notes
Canonical family mean	$7$	$+0.893$	two-sided exact $p{=}0.0123$
Canonical family median	$7$	$+0.893$	same ranks as mean
Worst leave-one-family-out	$6$	$+0.829$	drop Anthropic/DeepSeek/Google/OpenAI tie
Drop Qwen family	$6$	$+0.943$	Qwen not required for positive association
Drop Meta and Qwen	$5$	$+1.000$	joint upper-tail leverage sensitivity; two-sided exact $p{=}0.0167$
Qwen split by generation	$9$	$+0.883$	Qwen-3, Qwen-3.5, Qwen-3.6

Appendix

Table 24: Family-level sensitivity details. Rows are recomputed after collapsing related models by the stated grouping rule; alternative taxonomies are descriptive sensitivity checks.

Held-out family	$\boldsymbol{\alpha_{\text{tot}}}$	Observed	Pred.	$80\%$ PI	$95\%$ PI	Rank error
Anthropic	$0.4504$	$2.15\%$	$6.77\%$	$[-1.96,+15.50]$	$[-9.03,+22.58]$	$1$
DeepSeek	$0.1129$	$0.00\%$	$-4.13\%$	$[-16.01,+7.75]$	$[-25.64,+17.38]$	$0$
Google	$0.2875$	$0.29\%$	$2.70\%$	$[-7.03,+12.43]$	$[-14.92,+20.31]$	$0$
Meta	$0.7737$	$8.46\%$	$18.94\%$	$[+9.97,+27.92]$	$[+2.69,+35.20]$	$2$
OpenAI	$0.3461$	$1.52\%$	$4.02\%$	$[-5.39,+13.43]$	$[-13.02,+21.07]$	$0$
Phi	$0.4686$	$10.24\%$	$5.93\%$	$[-2.90,+14.76]$	$[-10.06,+21.92]$	$2$

Appendix

Table 25: Post-hoc leave-one-family predictive check on family means. “Pred.” is fit on the other six families, and prediction intervals are unbounded percentage points of

C^{\mathrm{cond}}

Held-out family	Observed	EIV $80\%$ PI	EIV $95\%$ PI	Covered at $95\%$
Anthropic	$2.15\%$	$[0.00,+15.88]$	$[0.00,+23.25]$	yes
DeepSeek	$0.00\%$	$[0.00,+8.32]$	$[0.00,+17.62]$	yes
Google	$0.29\%$	$[0.00,+12.60]$	$[0.00,+21.01]$	yes
Meta	$8.46\%$	$[+10.18,+28.39]$	$[+2.31,+35.95]$	yes
OpenAI	$1.52\%$	$[0.00,+13.89]$	$[0.00,+21.77]$	yes
Phi	$10.24\%$	$[0.00,+15.01]$	$[0.00,+22.60]$	yes

Appendix

Table 26: Jeffreys-propagated held-out-family predictive sensitivity. Conditional-collapse uncertainty is propagated through the six-family LoFO fit; displayed intervals are clipped to

[0,100]\%

and remain diagnostic rather than calibrated.

Runtime diagnostic	$n$	Spearman $\rho$ vs. $C^{\mathrm{cond}}$	Interpretation
Initial disagreement rate	$6$	$+0.928$	non-unanimous initial panels collapse more often
Initial unanimity rate	$6$	$-0.928$	unanimous initial panels are safer in this slice
Normalized initial-answer entropy	$6$	$+0.928$	same signal expressed as answer diversity
At-risk initial disagreement	$6$	$+0.928$	diversity remains visible after conditioning on initial majority correctness
Round $1$ majority-change rate	$6$	$+0.986$	strong runtime trajectory score, measured after debate has started

Appendix

Table 27: Trace-available initial-diversity diagnostics. These runtime diagnostics are computed only after initial answers exist, and family coverage is too small for a family-level primary test.

Predictor	Partial $\rho$	$p_{\text{raw}}$	$p_{\text{Holm}}$
$\alpha$ partial $\rho_{\alpha\mid\text{init\_acc}}$	$+0.64$	$0.062$	$0.372$
$\alpha_{\text{adv}}$ partial $\rho_{\alpha_{\text{adv}}\mid\text{init\_acc}}$	$+0.52$	$0.148$	$0.592$
$\alpha_{\text{cor}}$ partial $\rho_{\alpha_{\text{cor}}\mid\text{init\_acc}}$	$-0.36$	$0.349$	$0.698$

Appendix

Table 28: Partial Spearman residualized on rank initial accuracy. At

N{=}9

, no row survives Holm–Bonferroni correction at family-wise

\alpha{=}0.05

; the

\alpha_{\text{adv}}

-only partial is positive but weaker than the pooled

\alpha

signal.

Lane	Family	Qids	$\boldsymbol{\alpha_{\text{tot}}}$	AS	AA	Init $\rightarrow$ R4 acc.	R3 $C^{\mathrm{cond}}$	R4 $C^{\mathrm{cond}}$	R4 correction	None
Gemini 3 Flash	Google	$200$	$.1856$	$.0321$	$.3844$	$83.0\%\rightarrow 84.0\%$	$1/166$ ( $0.60\%$ [ $0.1,3.3$ ])	$1/166$ ( $0.60\%$ [ $0.1,3.3$ ])	$3/34$ ( $8.82\%$ [ $3.0,23.0$ ])	$0.03\%$
Gemini 3.1 Flash-Lite	Google	$200$	$.3017$	$.2746$	$.3242$	$82.0\%\rightarrow 84.5\%$	$0/164$ ( $0.00\%$ [ $0.0,2.3$ ])	$0/164$ ( $0.00\%$ [ $0.0,2.3$ ])	$5/36$ ( $13.89\%$ [ $6.1,28.7$ ])	$0.03\%$
Gemini 3.1 Pro	Google	$199$	$.2013$	$.1330$	$.3197$	$72.9\%\rightarrow 78.9\%$	$0/145$ ( $0.00\%$ [ $0.0,2.6$ ])	$0/145$ ( $0.00\%$ [ $0.0,2.6$ ])	$7/37$ ( $18.92\%$ [ $9.5,34.2$ ])	$10.42\%$
Mistral Small 4	Mistral	$200$	$.3772$	$.0660$	$.6448$	$70.0\%\rightarrow 73.0\%$	$4/140$ ( $2.86\%$ [ $1.1,7.1$ ])	$7/140$ ( $5.00\%$ [ $2.4,10.0$ ])	$13/60$ ( $21.67\%$ [ $13.1,33.6$ ])	$0.07\%$
Grok 4.1 Fast	xAI	$200$	$.1452$	$.0683$	$.1845$	$83.5\%\rightarrow 84.5\%$	$0/167$ ( $0.00\%$ [ $0.0,2.2$ ])	$1/167$ ( $0.60\%$ [ $0.1,3.3$ ])	$3/33$ ( $9.09\%$ [ $3.1,23.6$ ])	$0.13\%$

Appendix

Table 29: Post-hoc matched holdout stress tests. Rows are excluded from the headline Spearman test, family aggregation, and

N{=}14

scatter. Only strict alpha/debate question overlap is summarized; brackets are Wilson

95\%

intervals, and R4 is the stored final endpoint for these traces.

Run	Init $\to$ final acc.	$\boldsymbol{C^{\mathrm{cond}}}$	Correction	Signed	$\alpha$ rows	$\alpha\to C$	Scope note
Gemini 3.1 Flash-Lite	$74.2\%\to 77.8\%$	$2/147$ ( $1.4\%$ [ $0.4,4.8$ ])	$9/51$ ( $17.6\%$ [ $9.6,30.3$ ])	$+7$	$198/198$	$\rho{=}+.039$ , $p{=}0.640$	low-collapse boundary
Mistral Small 4	$60.6\%\to 61.1\%$	$20/120$ ( $16.7\%$ [ $11.1,24.3$ ])	$21/78$ ( $26.9\%$ [ $18.3,37.7$ ])	$+1$	$196/198$	$\rho{=}+.236$ , $p{=}0.0096$	dense stress row
Llama 3.3 70B	$51.5\%\to 55.1\%$	$6/102$ ( $5.9\%$ [ $2.7,12.2$ ])	$13/96$ ( $13.5\%$ [ $8.1,21.8$ ])	$+7$	$192/198$	$\rho{=}+.061$ , $p{=}0.544$	provider-routed stress row
DeepSeek V4 Flash	$56.1\%\to 74.7\%$	$10/111$ ( $9.0\%$ [ $5.0,15.8$ ])	$47/87$ ( $54.0\%$ [ $43.6,64.1$ ])	$+37$	$161/198$	$\rho{=}+.029$ , $p{=}0.783$	parser-caveated utility stress row

Appendix

Table 30 : Post-hoc full-GPQA Diamond boundary/stress rows. Rows are excluded from the MMLU-Pro headline family association. Denominators condition on initial-majority correctness; brackets are Wilson

95\%

intervals and signed utility uses equal collapse/correction weights.

Quantity	Value	Notes
Alpha rows / questions	$1{,}800$ / $200$	complete $3{\times}3$ design
$\alpha_{\text{tot}}$	$0.6767$	anti-argument $0.7002$ , anti-social $0.6754$ , default $0.6546$
Debates	$200$	matched high-FR slice; $200/200$ overlap
Initial $\rightarrow$ final majority accuracy	$36.5\%\rightarrow 52.0\%$	net positive
Conditional collapse	$8/73$ ( $10.96\%$ )	initially-correct majority at risk
Conditional correction	$39/127$ ( $30.71\%$ )	initially-wrong majority

Appendix

Table 31: Post-hoc Qwen3-32B local-vLLM holdout. This corrected-pool expansion row is excluded from the sealed

N{=}14

headline table and has complete 200-question alpha/debate overlap.

Coefficient	Posterior mean	$95\%$ HDI	ICC
Round $1$ majority changed (z)	$\mathbf{+0.90}$	$\mathbf{[+0.68,\,+1.12]}$	n/a
Round $1$ agent flips (z)	$\mathbf{+0.43}$	$\mathbf{[+0.23,\,+0.62]}$	n/a
Round $1$ unanimous (z)	$\mathbf{-0.98}$	$\mathbf{[-1.23,\,-0.76]}$	n/a
Model-level $\alpha$ (z, conditional)	$-0.07$	$[-0.97,\,+0.81]$	n/a
$\sigma_{\text{question}}$	$1.44$	$[0.76,\,2.21]$	$0.344$ $[0.10,0.56]$
$\sigma_{\text{model}}$	$0.71$	$[0.00,\,2.07]$	$0.044$ $[0.00,0.54]$

Appendix

Table 32: Bayesian crossed random-effects MLM on the

4

-OSS-trace cohort. Round

1

trajectory features are strong runtime correlates; the residual model-level

\alpha

slope is weakly identified, so this is a localization check rather than causal mediation evidence.

Model	$\hat{u}_{m}$
Phi- $4$ -mini	$-0.003$
Llama- $3.1$ - $8$ B	$-0.053$
Qwen3- $4$ B	$-0.331$
Qwen3- $8$ B	$-0.371$

Appendix

Table 33 : Per-model posterior random intercepts for the Bayesian Round

1

multilevel model, on the logit scale.

Model	Condition	Acc.%	Collapse%	Correct.%	$p$ (McNemar)
Haiku	Default debate	80.0	15.0	3.0	0.593
Haiku	Independent debate	73.0	17.0	1.0	0.593
GPT	Default debate	73.0	1.0	3.0	1.000
GPT	Independent debate	70.0	0.0	2.0	1.000

Appendix

Table 34: Independence-prompt ablation. Results are shown for Claude Haiku as the primary row and GPT-4o-mini as a replication.

Model	Method	Acc.%	Col.%	Corr.%	Corr./Col.	Net
Haiku	No debate	77.5	0.0	0.0	n/a	0
	Standard debate	72.0	9.5	4.0	0.42	$-$ 11
	Shielded	77.5	2.0	2.0	1.00	0
	ACC	78.0	1.5	2.0	1.33	+1
	Oracle upper bound	79.0	0.5	2.0	4.00	+3
Phi-4-mini ‡	No debate	57.8	0.0	0.0	n/a	0

Appendix

Table 35: Development-set question-level intervention baselines. ACC is the strongest deployable policy on the development split; the oracle detect-revert row is an upper bound.

Model	Policy	$N$	Acc.%	Collapse%	Correct.%	Acc. $\Delta$ vs Std
Haiku	Standard	800	81.8	1.4	2.5	+0.0
	Shielded	800	80.6	1.3	1.8	$-$ 1.1
	ACC	800	80.6	1.4	1.9	$-$ 1.1
Phi-4-mini (800q)	Standard	800	56.9	5.8	9.1	+0.0
	Shielded	800	57.5	5.3	6.9	+0.6
	ACC	800	58.1	4.6	6.9	+1.3

Appendix

Table 36: Frozen deployable policy evaluation on disjoint held-out test sets. ACC is selected on pooled development data, then evaluated without retuning; the two Phi-4-mini tests use disjoint question sets.

Held-out model	$\tau^{*}$	Prevented	Lost	Net	$\Delta$ acc (pp)	Oracle $\Delta$ acc
Llama-3.1-8B (86 col / 98 corr)	0.70	10/86	7/98	$+3$	$+0.15$ [ $-0.25,+0.55$ ]	$+0.80$
Phi-4-mini (112 col / 174 corr)	0.70	4/112	13/174	$-9$	$-0.42$ [ $-0.79,-0.05$ ]	$-0.32$
Qwen3-4B (50 col / 461 corr)	0.05	15/50	87/461	$-72$	$-3.33$ [ $-4.26,-2.45$ ]	$-0.14$
Qwen3-8B (3 col / 71 corr)	0.70	0/3	1/71	$-1$	$-0.50$ [ $-1.50,+0.00$ ]	$-0.50$
Pooled (frozen LOMO policy)	n/a	29/251	108/804	$-79$	$-1.21$	n/a

Appendix

Table 37: LOMO pilot-gated freeze policy on

6{,}525

local-model debates. The classifier, threshold

\tau

, and freeze mode are selected on training models only; rowwise thresholds are held-in fold choices, not model optima. The oracle column is non-deployable and quantifies selection bias.

Control	$r{=}0.5$	$r{=}1$	$r{=}2$	$r{=}4$	$r{=}8$	Breakeven $r$
Probe-gated freeze	$-1.43$	$-1.21$	$-0.77$	$+0.12$	$+1.90$	$3.72$
Round $1$ majority changed	$-3.27$	$-1.04$	$+3.43$	$+12.35$	$+30.20$	$1.23$
Learned Round $1$ stump	$-2.39$	$-2.31$	$-2.15$	$-1.83$	$-1.20$	$15.50$

Appendix

Table 38 : Accuracy-delta sensitivity as utility weights vary. Values are percentage points;

r{=}1

gives equal collapse/correction weights, and breakeven is the smallest non-negative replay ratio. Any non-equal utility ratio should be fixed before policy selection and reported with this ladder.

Aggregates: median $\tilde{\rho}{=}\mathbf{0.253}$ ; Page’s $L{=}723.0$ ( $p_{\text{one-sided, analytic}}{=}0.0957$ ).
Model	$\rho_{k,p}$	exact perm. $p$	max inv. drop	F $3$ floor $\Delta p$	McNemar $p$	disc. $b/c$
Qwen3.5-4B	$+0.840$	$0.0030$	$0.08$	$+0.20$	$0.0032$	$11/1$
Qwen3.5-9B	$\mathbf{-0.295}$	$0.7817$	$0.06$	$-0.02$	$1.000$	$0/1$
Qwen3.6-27B-FP $8$	$+0.253$	$0.2467$	$0.06$	$+0.10$	$0.0313$	$5/0$

Appendix

Table 39: B-pilot per-model results across the

9

-rung ladder.

\rho_{k,p}

is the within-model rung/refusal Spearman correlation, exact permutation

p

is one-sided over

9!

rung orderings, and bold entries mark the two pre-registered §

3

Fail triggers.

Model	F $3$ floor $\Delta p$	$\geq{+}0.02$	McNemar $p$	$\leq 0.05$	Wilson LB	LB $>0.5$
Qwen3.5-4B	$+0.20$	✓	$0.0032$	✓	$0.699$	✓
Qwen3.5-9B	$-0.02$	n/a	$1.000$	n/a	$0.000$	n/a
Qwen3.6-27B-FP $8$	$+0.10$	✓	$0.0313$	✓	$0.649$	✓

Appendix

Table 40: F

3

floor usability per model. Bold “Yes” entries are the two pilot models that clear both gates and constitute the F

3

usable verdict.

Pair	$p$	Holm $p$	Reject	Observed order follows
llama-3.1-8b vs. qwen3.5-9b	$<0.0001$	$<0.0001$	yes	initial accuracy
llama-3.1-8b vs. qwen3.5-4b	$<0.0001$	$<0.0001$	yes	initial accuracy
qwen3.5-9b vs. qwen3.6-27b-fp8	$<0.0001$	$<0.0001$	yes	initial accuracy
qwen3.5-4b vs. qwen3.6-27b-fp8	$<0.0001$	$<0.0001$	yes	initial accuracy
qwen3-8b vs. qwen3.5-9b	$<0.0001$	$<0.0001$	yes	screen
qwen3-8b vs. qwen3.5-4b	$<0.0001$	$<0.0001$	yes	screen

Appendix

Table 41 : All

26

model-row pairs that the screen and initial-majority accuracy order differently (post hoc, in-sample).

p

: two-sided Fisher exact test for the difference in conditional collapse; Holm: adjusted over the

26

tests; the last column names the ordering that observed

C^{\text{cond}}

follows.

Task	Panel or model	$n$	Acc. (%)	$C/n_{+}$	$K/n_{-}$	Net	Note
MMLU-Pro	Mistral + Llama-3.3-70B + DeepSeek	200	74.0 / 81.0	5/148	19/52	$+14$	26 ties
	Mistral + DeepSeek + Llama-4-Scout	200	78.5 / 81.5	3/157	9/43	$+6$	20 ties
	Mistral + DeepSeek + Qwen3-4B	200	71.5 / 71.5	7/143	7/57	$0$	28 ties
	GLM-4.6 (reasoning disabled)	200	76.0 / 75.5	6/152	5/48	$-1$	homogeneous
GSM8K	Mistral	300	96.3 / 97.0	1/289	3/11	$+2$	sparse
	Llama-4-Scout	300	96.3 / 95.7	2/289	0/11	$-2$	sparse

Appendix

Table 42 : Post-hoc feasibility rows (Round

3

, alphabetic tie rule).

C/n_{+}

and

K/n_{-}

are collapses and corrections with their initial-state denominators; “ties” counts initial states without a unique plurality. Mistral is Mistral Small 4 and DeepSeek is DeepSeek V4 Flash. None of these rows enters the primary analysis.

Task	Model	$\Delta C_{\rm wrong}/n$	$\Delta C_{\rm tie}/n$	$\Delta$ binary accuracy	$\Delta$ expected tie score
MMLU-Pro	GLM-4-9B	$-0.59$	$+0.88$	$-2.14\;[-2.82,-1.47]$	$-0.30\;[-0.87,0.28]$
	Gemma-3-4B	$-2.82$	$+2.53$	$-3.28\;[-4.29,-2.30]$	$+0.38\;[-0.57,1.33]$
	Granite-4.1-8B	$-0.73$	$+0.44$	$-2.45\;[-3.20,-1.73]$	$-0.89\;[-1.54,-0.24]$
	Phi-4	$-0.23$	$+0.45$	$-1.52\;[-2.11,-0.94]$	$-0.51\;[-1.03,0.01]$
	Phi-4-mini †	$-0.80$	$+2.65$	$-4.81\;[-5.73,-3.90]$	$-1.27\;[-2.07,-0.44]$
	Qwen3-0.6B †	$-2.51$	$+0.89$	$-3.33\;[-4.28,-2.35]$	$+0.35\;[-0.49,1.23]$

Appendix

Table 43 : Private revision minus peer debate on twelve multiple-choice settings with shared Round

0

(six peer and four private runs per setting,

n{=}2{,}000

). All entries are percentage points of the question pool. The first two columns are point estimates; brackets are paired

95\%

bootstrap intervals over question-level run means.

\dagger

: amended-only quality control.

Task	Model (mode)	Tokens	$n$	$C/n_{+}$	$K/n_{-}$	Accuracy (%)	Private $-$ peer
MMLU-Pro	Qwen3-4B (thinking)	16,384	1,000	18/656	16/317	65.60 / 66.20	$-6\;[-17,5]$
	Qwen3-8B (thinking)	16,384	1,000	15/716	17/264	71.60 / 72.50	$-8\;[-20,4]$
	Qwen3-14B (thinking)	16,384	1,000	24/737	17/241	73.70 / 73.80	$-13\;[-26,-1]$
	Qwen3-32B (thinking)	16,384	1,000	9/754	7/212	75.40 / 76.30	$-13\;[-26,0]$
	R1-Distill-Llama-8B †	16,384	1,000	87/472	42/344	47.20 / 47.20	$-44\;[-70,-18]$
	gpt-oss-20b (low effort)	16,384	2,000	43/1384	78/488	69.20 / 73.25	$-26\;[-48,-4]$

Appendix

Table 44 : Reasoning-mode rows of the follow-up (peer debate, shared Round

0

). Tokens is the generation limit.

C/n_{+}

and

K/n_{-}

are collapses and corrections with their initial-state denominators; accuracy is Round

0

/ Round

3

under binary scoring. The last column is private minus peer in correct-answer counts with a paired

95\%

bootstrap interval from the base paired run of each arm (original answer coding).

\dagger

: amended-only quality control.

Explore similar work

The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate
Apr 29, 2026Blaž Bertalanič, Carolina FortunaMulti-Agent Debate Debate
Jožef Stefan Institute Ljubljana, Slovenia
Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols
May 10, 2026Julia Hu, Alfred Shen, Kumar LakshmipathiDebate Multi-Agent Debate
Amazon Web Services
Detection Without Correction: A Two-Parameter Decomposition of Multi-Stage LLM Pipelines
May 26, 2026Prashanti Nilayam, Kiran Ramanna, Prashil TumbadeIterative Self-Correction Multi-Agent Debate
Servicenow CA, USA

Apr 29, 2026, cs.MAJ/K move · Enter open · S save

The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate

Blaž Bertalanič, Carolina Fortuna

Jožef Stefan Institute Ljubljana, Slovenia

Multi-Agent Debate Debate Iterative Self-Correction Free-Form Textual Rationales

Open paper AskPDF arXiv

Multi-agent debate, where teams of LLMs iteratively exchange rationales and vote on answers, is widely deployed under the assumption that peer review filters hallucinations. Yet the failure dynamics of homogeneous debate remain poorly understood, therefore we report findings from a controlled empirical study of teams of

N{=}10

homogeneous agents (Qwen2.5-7B, Llama-3.1-8B, Ministral-3-8B) across

R{=}3

debate rounds on two high-difficulty benchmarks (GSM-Hard and MMLU-Hard). We compare peer debate against isolated self-correction and a stochastic noise control that injects rationales from unrelated problems. We decompose debate failure into three model-dependent pathways: sycophantic conformity, where agents uncritically adopt majority answers (modal adoption up to 85.5%); contextual fragility, where peer rationales destabilize previously correct reasoning (vulnerability rate up to 70.0%); and consensus collapse, where plurality voting discards correct answers already present in the generation pool (oracle gap up to 32.3 percentage points). Ablations over communication density (

K \in \{2,4,9\}

) and sampling temperature (

T \in \{0.4, 0.7\}

) show that conformity reaches high levels at minimal peer exposure (

K{=}2

) and intensifies with greater initial diversity. Across all configurations, debate consumes 2.1-3.4

\times

more tokens (up to 28,631 tokens per problem) than self-correction for equal or lower accuracy. Our results indicate that, within the 7-8B parameter class, homogeneous teams without structured roles do not benefit from unguided peer exchange, and that isolated self-correction consistently offers a more favorable cost-accuracy tradeoff.

Abstract

Explore similar work

The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate

Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols

Detection Without Correction: A Two-Parameter Decomposition of Multi-Stage LLM Pipelines