Multi-agent debate can improve reasoning, yet often fails to beat simple majority voting. We argue that successful debate requires two distinct mechanisms: proposal supply must surface a correct answer, and readout must identify that answer when voting misses it. We formalize the first requirement through recoverable headroom, which measures cases where a correct proposal is available but the majority answer is wrong. For the second, we develop Latent Verification Debate (LVD), an accounting model in which candidate proposals receive answer-specific verification evidence before final generation. Controlled fixed-proposal interventions estimate this latent effect in equivalent peer-support units and show that correct evidence changes answer probabilities and generated decisions while proposal supply remains fixed. To improve proposal supply, we construct societies from neural-thicket agents using labeled and label-free coverage objectives. Across two backbones and matched-budget reasoning benchmarks, coverage-selected societies increase complementary proposal supply and improve aggregate accuracy in repeated stochastic evaluations. Round-level controls further show that interaction provides gains beyond applying the same finalizer directly to the initial proposals. These results identify proposal coverage and truth-sensitive evidence use as complementary conditions for debate to outperform voting. Code is available at https://github.com/Wang-ML-Lab/when-debate-helps.
Figures & tables
Figure 1: Overview of the supply-readout view. Transparent regions denote questions for which individual sampled agents can surface the correct answer; solid regions denote questions recovered by the final readout. Top: Overlapping sampled agents provide redundant proposal coverage, so majority vote keeps a small shared region and can reinforce a repeated wrong answer. Bottom: Diverse agents provide complementary proposal coverage; after debate, LVD uses verification-aware evidence to recover a surfaced correct minority answer.
Figure 2: Motivating observations. (a) Qwen2.5-3B-Instruct shows null-like debate dynamics with little gain over vote, while Qwen3-4B-Instruct shows positive gain across rounds. (b) Under the same neural-thicket supply regime, Debate readout improves over vote , while the proposal-hit upper bound remains higher, indicating recoverable headroom. (c) Final debate lift is positively correlated with initial proposal-hit headroom, suggesting that debate is supply-constrained. Together, these observations motivate a supply-readout account: debate helps when correct minority proposals are surfaced and the readout can recover them through verification-aware dynamics.
Method
Reasoning
MMLU-Redux
Avg
Avg Ranking
AMC12
MATH500
GPQA
Moral
Chem.
CS
Acct.
Med.
Backbone: Qwen3-4B-Instruct
Label-Free Methods
SC@20
91.20
92.33
49.13
30.00
70.00
81.67
75.00
83.33
71.58
2.38
Vanilla Debate
92.69
91.67
52.31
50.00
71.67
80.00
76.67
83.33
74.79
1.81
SC-Greedy Debate (Ours)
92.29
92.67
51.45
63.33
68.33
81.67
73.33
86.67
76.22
1.81
Table 1: Accuracy (%) on Qwen3-4B-Instruct and OLMo-3-7B-Instruct under a unified computation budget. SC and Neural thicket vote rows use vote20 ; debate rows use round-3 (R3) majority vote from matched five-round, four-agent traces. MMLU-Redux is split into five 60-example subjects. Avg is the unweighted mean over the eight displayed columns, and Avg Ranking is the mean within-block method rank. Boldface and underline denote the best and second-best result per column within each method block and model. Gray rows mark our proposed agent pool construction methods.
Method
N
Init Vote
R3 Vote
R5 Vote
Final
Init Hit
R3 Hit
R5 Hit
Lift
Backbone: Qwen3-4B-Instruct, AMC12 → MMLU-Redux
SC-Greedy Debate (Ours)
300
72.33
75.67
76.33
76.33
84.00
77.67
76.33
+4.00
Top-4 Debate
300
72.67
75.00
74.67
75.33
84.00
77.67
76.67
+2.67
AC-Greedy Debate (Ours)
300
75.33
77.33
77.00
76.67
83.67
78.00
78.00
+1.33
Backbone: OLMo-3-7B-Instruct, AMC12 → MMLU-Redux
SC-Greedy Debate (Ours)
300
55.33
56.33
58.33
58.33
70.00
59.67
59.33
+3.00
Table 2: AMC12-selected seed transfer to MMLU-Redux. Init, R3, and R5 report pre-debate, round-3, and round-5 metrics over the same four agents. Final is the R5 vote-then-judge readout; Lift is Final minus Init Vote. Values are percentages except N . Gray rows mark our greedy construction methods.
Figure 3: Label-based five-round dynamics. Solid lines show round-level accuracy before finalization; dashed hollow legend entries marked “Oracle” show proposal-hit coverage. Later rounds can improve vote accuracy while reducing answer diversity, so final gains depend on recovering correct proposals before coverage collapses. The label-free counterpart is in Fig. 4 .
Backbone
Dataset
Δmtrace [95% CI]
Median W
Answer-stripped T−F
Δ Acc at d=3
Qwen3-4B
GPQA
8.15 [4.96, 11.41]
2.23
4.59
+.14
Qwen3-4B
MMLU-Redux
11.40 [9.02, 13.97]
>3
5.50
+.04
OLMo3-7B
GPQA
4.57 [3.65, 5.50]
1.92
2.01
+.56
OLMo3-7B
MMLU-Redux
5.47 [4.52, 6.43]
>3
2.10
+.58
Table 3: Controlled fixed-proposal evidence for verification-aware readout. Trace effect is the answer-margin shift from an original correct peer response relative to neutral context; W is its median equivalent controlled support over recovery cases. T−F is the gold-margin difference between answer-stripped correct and wrong rationales. Δ Acc is correct-support accuracy minus neutral accuracy at d=3 . All entries use the main readout configuration; secondary configuration controls are reported in the appendix.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Experiment family
Workers per run
Observed wall time and compute
Qwen seed profiling (500 agents per dataset)
6–8 GPUs
3.3–18.4 hours per dataset; approximately 27–110 GPU-hours
OLMo seed profiling (300 retained agents per dataset)
8 GPUs
1.3–6.6 hours per dataset; about 10–53 GPU-hours
Five-round downstream and prompt-control evaluation
1 GPU per job
0.2–4.9 hours per job
Five-run stochastic robustness experiment
1 GPU per job; 120 jobs
156.8 GPU-hours in total
Controlled mechanism, readout, and transfer diagnostics
1 GPU per job
Within the same 0.2–4.9-hour single-job envelope
Appendix
Table 4: Compute used by the reported experiment families. Profiling times vary with dataset size and output length; downstream times vary with the backbone, dataset, and number of debate rounds.
Asset and version
License or terms
Use in this work
Qwen3-4B-Instruct-2507
Apache-2.0
Main 4B backbone; research evaluation only
OLMo-3-7B-Instruct
Apache-2.0; Ai2 Responsible Use Guidelines
Main 7B backbone; research evaluation only
Qwen2.5-3B-Instruct
Qwen Research License
Motivating experiment; non-commercial research evaluation
GPQA
CC BY 4.0
Gated benchmark; examples are not redistributed
MMLU-Redux 2.0
CC BY 4.0
Benchmark evaluation
OlympiadBench
Apache-2.0
Benchmark evaluation
Appendix
Table 5: Licenses and stated terms for existing model and dataset assets.
Backbone
Profiling split
σ
Seeds
Base ref.
Seeds > base
Mean
Best
Used
Qwen3-4B
MATH500 first200
0.0005
160
90.00
59/160
89.82
92.50
yes
Qwen3-4B
MATH500 first200
0.001
174
90.00
51/174
89.58
92.50
yes
Qwen3-4B
MATH500 first200
0.002
166
90.00
11/166
88.10
91.00
yes
Qwen3-4B
AMC12 train360
0.0005
160
84.17
5/160
82.40
85.83
yes
Qwen3-4B
AMC12 train360
0.001
174
84.17
7/174
81.96
86.11
yes
Qwen3-4B
AMC12 train360
0.002
166
84.17
0/166
78.97
83.33
yes
Appendix
Table 7: Seed-quality diagnostics for the main-table backbones. Values are percentages on the profiling split. Base ref. is the same-split unperturbed pass@1 reference. The Used column marks scales retained for downstream construction.
Backbone
Dataset
Δmtrace [95% CI]
Median W
Correct − wrong margin, d=1/2/3
Qwen3-4B
GPQA
8.15 [4.96, 11.41]
2.23
2.59 / 5.94 / 13.63
Qwen3-4B
MMLU-Redux
11.40 [9.02, 13.97]
>3
1.03 / 5.36 / 11.76
OLMo3-7B
GPQA
4.57 [3.65, 5.50]
1.92
3.33 / 5.06 / 5.79
OLMo3-7B
MMLU-Redux
5.47 [4.52, 6.43]
>3
3.59 / 5.18 / 5.53
Appendix
Table 8: Original peer-response effects and equivalent conclusion-only support over recovery cases. Trace effects are means with 95% confidence intervals; W is the median interpolated equivalent support. The final column reports the correct-support minus wrong-support answer margin under the Verify prompt at each dose.
Backbone / dataset
Help-over-harm asymmetry A
Correct-over-wrong gap T−F
Qwen3-4B / GPQA
7.34 [2.14, 12.45] / 6.84 [1.29, 12.33]
1.90 [.54, 3.39] / 4.59 [2.45, 6.96]
Qwen3-4B / MMLU-Redux
8.70 [3.50, 13.72] / 6.36 [2.34, 10.42]
5.98 [3.78, 8.32] / 5.50 [3.46, 7.69]
OLMo3-7B / GPQA
1.92 [.46, 3.35] / 1.39 [.17, 2.58]
1.70 [1.16, 2.31] / 2.01 [1.35, 2.77]
OLMo3-7B / MMLU-Redux
2.84 [.87, 4.85] / 2.24 [.78, 3.78]
1.93 [1.26, 2.63] / 2.10 [1.41, 2.83]
Appendix
Table 9: Truth sensitivity of answer-stripped trace rationales. Entries are mean margin changes with 95% confidence intervals, shown as Standard / Verify.
Backbone
Dataset
Prompt
Neutral
Correct support
Answer withheld
Shuffled rationale
Qwen3-4B
GPQA
Standard
.32
.50
.32
.28
Qwen3-4B
GPQA
Verify
.32
.46
.34
.14
Qwen3-4B
MMLU-Redux
Standard
.28
.58
.26
.28
Qwen3-4B
MMLU-Redux
Verify
.38
.42
.32
.26
OLMo3-7B
GPQA
Standard
.30
.80
.30
.30
OLMo3-7B
GPQA
Verify
.30
.86
.30
.30
Appendix
Table 10: Free-generation accuracy after three peer messages. Each entry evaluates the same 50 questions. Correct support is bolded.
Round
Verify Vote
Standard Vote
Δ Vote
Verify Hit
Standard Hit
Δ Hit
R0
76.04
76.04
+0.00
83.95
83.95
+0.00
R1
77.95
77.81
-0.14
85.38
84.89
-0.49
R2
78.19
78.12
-0.08
82.20
83.38
+1.18
R3
78.37
78.16
-0.21
80.44
82.98
+2.54
R4
78.10
78.43
+0.32
79.56
82.04
+2.48
R5
78.21
78.13
-0.08
79.26
81.62
+2.37
Appendix
Table 11: Round-level prompt-style dynamics averaged over the 15 matched dataset-method cells. R0 is identical because both controls use the same seed answers before debate. Δ is Standard minus Verify. All values are percentages.
Dataset
Method
N
Verify Final
Standard Final
Δ
Standard R5 Vote
Standard R5 Hit
AMC12
Vanilla Debate
739
92.96
92.42
-0.54
92.29
93.78
AC-Greedy Debate
739
91.61
91.34
-0.27
91.75
94.18
SC-Greedy Debate
739
91.88
92.02
+0.14
91.88
93.50
MATH500
Vanilla Debate
300
91.33
92.00
+0.67
91.67
93.00
AC-Greedy Debate
300
91.67
92.67
+1.00
93.33
94.00
SC-Greedy Debate
300
92.00
91.67
-0.33
91.67
93.33
Appendix
Table 12: Prompt-style control for the five-round new-seed debate runs. Verify Final is the completed verify_then_answer debate prompt; Standard Final repeats the same datasets, seed banks, verifier settings, and vote_then_judge_tiebreak finalizer with the normal standard debate prompt. Δ is Standard Final minus Verify Final. All accuracy values are percentages.
Scope
Qwen3-4B
OLMo3-7B
Four shared datasets, pooled
r=.770,ρ=.787
r=.548,ρ=.648
Five MMLU subjects, pooled
r=.568[.125,.787]
r=.635[.219,.805]
Appendix
Table 14: Association between initial headroom H=P0−V0 and final lift G=F−V0 on the two backbones. The MMLU row reports a 95% paired question-bootstrap interval for Pearson r .
Backbone
Method
R0 vote
Headroom
Recovery rate
Recovered
New correct
Damage
Gain
Final
Qwen3-4B
SC-Greedy
72.33
11.67
45.71
5.33
0.67
2.00
+4.00
76.33
Qwen3-4B
AC-Greedy
75.33
8.33
48.00
4.00
0.33
3.00
+1.33
76.67
OLMo3-7B
SC-Greedy
55.33
14.67
36.36
5.33
0.67
3.00
+3.00
58.33
OLMo3-7B
AC-Greedy
54.00
15.33
26.09
4.00
0.33
2.00
+2.33
56.33
Appendix
Table 15: AMC12-to-MMLU-Redux transfer decomposition. Recovery rate is conditional on R0 headroom; the contribution columns are unconditional percentage points.
Selector
Pool N=50→100→200→500
Construction m=25→50→100→150
AC-Greedy
84.06→84.08→84.29→84.12
83.22→83.59→84.12→84.38
SC-Greedy
83.95→84.03→84.01→83.81
83.30→83.36→83.81→83.85
Appendix
Table 17: Sensitivity to candidate-pool size N and construction-set size m . Each arrow follows the increasing values shown in the column header.
Selector
Construction signal
Requires
R0 hit@4
R0 vote
R5 final
AC-Greedy
Correct-answer coverage
Labels
84.67
74.67
75.67
SC-Greedy
Answer-cluster coverage
Clusters
82.00
68.67
75.33
Hidden-Prefix
Representation-cluster coverage
Neither
84.30±0.69
75.07±1.03
77.00±1.25
TopAcc
Individual accuracy
Labels
83.60±0.62
73.93±1.07
75.40±0.80
Appendix
Table 18: Fixed-society Qwen3-4B MMLU-Redux results. Hidden-Prefix and TopAcc report mean ± SD across 10 paired generation seeds; AC-Greedy and SC-Greedy are the submitted fixed runs and are included only as reference points.
Figure 4: Label-free five-round dynamics. This appendix counterpart to Fig. 3 shows methods that do not use labeled training diagnostics for society selection. Solid lines show round-level majority-vote accuracy before finalization; dashed hollow legend entries marked “Oracle” show proposal-hit coverage.
Method
Round
Vote
Hit
Final
H
G
ρ
R
I
L
Vanilla Debate
R0
75.65
83.03
77.92
7.39
2.27
0.31
53.49
9.94
5.00
R1
77.85
85.05
77.92
7.20
0.07
0.01
43.10
1.23
4.25
R2
77.84
81.86
77.92
4.02
0.08
0.02
24.33
0.14
1.58
R3
78.05
80.07
77.92
2.03
-0.13
-0.06
43.39
0.60
1.37
R4
78.13
79.04
77.92
0.92
-0.21
-0.23
38.89
0.13
1.08
R5
77.99
78.98
77.92
0.99
-0.08
-0.08
10.00
0.13
0.33
Appendix
Table 19: Round-level recovery diagnostics averaged over the five benchmarks. For each round, Vote and Hit are the round-specific majority-vote and proposal-hit accuracies, while Final is the end-of-run vote_then_judge_tiebreak accuracy. We define H=P−V , G=D−V , ρ=G/H , R=Pr(D=1∣P=1,V=0) , I=Pr(D=1∣P=0,V=0) , and L=Pr(D=0∣V=1) , where D is the final decision. All values except ρ are percentages; averages are unweighted over benchmarks.
Dataset
Method
N
Init Vote
R5 Vote
Final
Init Hit
R5 Hit
Lift
Judge Ties
MATH500
Vanilla Debate
300
91.33
91.33
91.33
93.33
91.67
+0.00
1
MATH500
AC-Greedy Debate
300
91.33
91.33
91.67
94.33
91.67
+0.33
2
MATH500
SC-Greedy Debate
300
92.00
92.00
92.00
93.67
92.67
+0.00
0
OlympiadBench
Vanilla Debate
474
80.38
81.65
81.22
83.54
82.28
+0.84
7
OlympiadBench
AC-Greedy Debate
474
79.11
81.43
81.43
82.91
81.86
+2.32
7
OlympiadBench
SC-Greedy Debate
474
80.59
80.59
80.80
83.33
81.22
+0.21
5
Appendix
Table 20: Five-round debate results for the new seed teams. Final is accuracy after vote_then_judge_tiebreak ; Lift is Final minus Init Vote. Bold marks the best final accuracy within each dataset. Average rows report the unweighted mean across the five benchmarks.
When should a language model answer directly, sample and vote, or engage in multi-agent debate? Recent work shows voting often explains much of the gain attributed to debate, while selective-debate systems activate deliberation only on uncertain examples. We ask: under a matched ceiling on generated tokens (960 per example), how much per-example routing headroom exists, and how much is recoverable from cheap pre-deliberation signals? We evaluate greedy decoding, three-sample voting, and a two-agent critique-revise debate on MuSiQue and GSM8K using Llama 3.1 8B Instruct and Ministral 3 8B Instruct. On MuSiQue, an oracle selecting the correct protocol per example gains +14.0 and +13.7 pp over the best fixed one. The best fixed protocol is model- and dataset-dependent: each (model, dataset) cell has a different winner. This headroom is hard to recover from cheap ex-ante signals. A vote-entropy threshold is the only controller that directionally beats the best fixed protocol on both models (+1.3 and +1.7 pp), though individual paired-bootstrap CIs include zero. A joint analysis (meta-analysis +1.6 pp, p=0.125; Bayesian P(both>0)=0.59) is directionally consistent but not significant. Learned controllers (LR, GBT) do not outperform the threshold. The key finding is structural: vote entropy predicts where debate is safe, not where debate is needed. High entropy sharply reduces debate backfire, but 66% of debate-helpful examples (31/47) occur when voting is unanimous but wrong. A single-prompt self-critique probe on Llama flips the answer in 127/127 unanimous cases, yielding zero mutual information with the debate-helpful label; we cannot rule out a prompt-compliance artifact, but either interpretation disqualifies the probe as a router. Recovering the remaining headroom requires behavioral probes that avoid format-compliance confounds at the 8B scale.
Multi-agent debate systems are typically evaluated only on whether the final answer is correct, overlooking the quality of the intermediate reasoning that debate is designed to produce. This paper studies the relationship between three signals in multi-agent debate: token-level log-probability distributions over reasoning tokens, LLM-as-judge rubric scores assigned to those tokens, and final task accuracy. We examine whether internal confidence signals predict externally evaluated reasoning quality, and whether either signal aligns with task correctness, across three domains: rubric-based scoring, mathematical reasoning, and factual question answering. Our framework pairs a two-agent debate architecture -- a Constructor and an Auditor -- with an LLM-as-judge that scores each agent's reasoning along instruction following, justification quality, and evidence grounding, together with a critical-failure flag. Experiments in the rubric-scoring domain reveal a consistent four-phase confidence trajectory and a substantial role asymmetry: confidence aligns with judged reasoning quality roughly twice as strongly for the Constructor as for the Auditor, and confidence-based detection of critical reasoning failures is markedly more reliable for the Constructor (AUROC 0.804) than for the Auditor (0.634). These findings motivate the broader cross-domain investigation proposed in this paper.
Multi-agent debate can improve large language model reasoning by eliciting diverse hypotheses and critiques, yet its performance is often constrained by weak moderation. Common pipelines rely on fixed budgets, agreement-based stopping, or untrained judges, leading to redundant deliberation and unreliable evidence aggregation. We cast moderation as a meta-cognitive process, monitoring debate utility, controlling deliberation, and adjudicating a final answer, and introduce Meta-Moderator, a learnable framework that dynamically regulates debate and decides when to finalize an answer. Meta-Moderator is trained independently of the debaters via outcome-driven policy optimization, making debate regulation an explicit capability rather than an incidental effect of prompting. Across five benchmarks, Meta-Moderator outperforms widely used decision layers and transfers across tasks and system configurations. Further analyses show that it allocates debate more selectively and reduces mis-aggregation after informative hypotheses appear.
Wentao Hu, Zhuoyue Wan, Jinhao Shen +3
The Hong Kong Polytechnic University · Sichuan University