Post-training language models for reasoning typically rely on gold answers, task-specific verifiers, or learned reward models, limiting scalability when such signals are scarce. Multi-agent debate offers a self-supervised alternative, but converting debate into reliable training supervision remains challenging: majority voting weights agreeing responses equally, lacks a principled tie-breaking rule, and is ill-defined for open-ended tasks without a canonical answer; likelihood-based scoring can conflate fluency with correctness; and homogeneous committees may reinforce correlated errors. We introduce Alignment by Confident Debate (AbCD), which converts debate traces into preference data without gold answer labels. A sampling-diverse committee debates each problem, after which final answers are clustered by task-level semantic equivalence. We use verifiable tasks (mathematical reasoning, multiple-choice quantitative QA, and executable code-generation) as a controlled test bed: their answer keys or executable checkers let us evaluate the quality of debate-derived supervision against ground truth, even though the oracle itself never uses gold labels. Post-training on the resulting preferences outperforms inference-only ensembling and debate, as well as SFT on the same traces, across five model families spanning 1B to 8B parameters and four datasets, improving consensus accuracy, with a maximum gain of 21.5pp. Ablations show that the approach transfers to a larger model and to code generation, while also revealing limitations of heterogeneous committees and agreement-based weighting. Overall, the results suggest that debate traces can yield useful preference supervision when answer equivalence is combined with confidence-weighted disagreement.
Figures & tables
Figure 1: Committee behavior for Qwen-2.5-1.5B-it on GSM8K. The committee size varies from one to five agents under independent sampling, after which the number of debate rounds is varied with five agents. Additional rounds increase agreement but do not improve consensus accuracy beyond the first exchange and can reduce correctness.
Figure 2: AbCD instantiates N sampling-diverse agents from the base policy πθ and lets them debate for M generations. Final traces are clustered by task-level answer equivalence, and the cluster with the largest aggregate confidence on the claimed-answer span is selected. Traces in the winning cluster are preferred, while the remaining traces are dispreferred (green ≻ red). The resulting preference supervision updates πθ without gold answers in the oracle.
Figure 3: Committee behavior for Qwen-2.5-1.5B-it on GSM8K. Subfigures show accuracy by committee size (a), accuracy by debate rounds (b), agreement-rate across rounds (c), and correctness across rounds (d).
Figure 4: Change in AUROC from the initial-response round to the first exchange round for full-trace and claimed answer-span confidence. Each bar is one model–dataset pair across GSM8K, SVAMP, and MathQA. Positive values indicate improved discrimination after debate.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Datasets
What is aggregated
Confidence
Discrete math
GSM8K, SVAMP, MathQA
The same extracted answer
Span confidence
Executable code
MBPP
Programs with the same public outputs
Public tests passed
Checklist text
RaR
Identical satisfied-criteria patterns
Weighted hit score, penalties subtracted
Appendix
Table 7: How agent answers are clustered and scored for the training label.
Figure 5: Committee behavior for Llama-3.1-8B-it model on GSM8K. The consensus accuracy by committee size beyond 5 agents shows diminishing returns.
Figure 6: Effect of additional debate rounds on GSM8K across model families. Left: Consensus accuracy at a fixed committee size as the number of exchange rounds increases. Right: Oracle precision stratified by agreement rate, where k/5 denotes that k of five agents produce parseable answers in the selected answer-equivalence cluster. The first exchange round often improves consensus accuracy, while subsequent rounds can increase agreement without improving correctness and can reduce oracle precision at several agreement levels. Reference answers are used only for evaluation.
Figure 7: Effect of additional debate rounds on Qwen-2.5-1.5B-it across datasets. Left: Consensus accuracy at a fixed committee size as the number of exchange rounds increases. Right: Oracle precision stratified by agreement rate, where k/5 denotes that k of five agents produce parseable answers in the selected answer-equivalence cluster. Reference answers are used only for evaluation.
Committee
Initial response accuracy
Avg. agent performance
Consensus correctness
Llama-3.2-1B-it
Clones
3.2
4.6
6.4
Temperature-diverse
4.4
7.4
17.0
Role-diverse
5.6
7.3
20.6
Seed-diverse
3.2
4.3
6.6
Qwen-2.5-1.5B-it
Appendix
Table 8: Inference-only GSM8K accuracy (%) at N=5 , M=2 . Initial response accuracy is single-sample accuracy before any ensemble or debate; Avg. agent performance is average per-agent accuracy after M rounds of debate; Consensus correctness is aggregated response accuracy.
Model
Dataset
CoT R1
CoT R2
Δ CoT
Box R1
Box R2
Δ Box
Llama-3.2-1B-it
GSM8K
0.60
0.50
-0.10
0.63
0.72
0.10
SVAMP
0.55
0.56
0.01
0.59
0.77
0.18
MathQA
0.64
0.54
-0.10
0.63
0.65
0.02
Qwen-2.5-1.5B-it
GSM8K
0.74
0.66
-0.08
0.71
0.80
0.09
SVAMP
0.66
0.57
-0.09
0.68
0.76
0.08
MathQA
0.75
0.68
-0.07
0.73
0.77
0.04
Appendix
Table 9: Full-trace (CoT) versus claimed-answer-span (Box) AUROC before and after one debate round. Δ=R2−R1 .
Parameter
SFT
GRPO
DPO
KTO
Learning rate
1×10−5
1×10−5
1×10−5
1×10−5
Weight decay
1×10−2
1×10−2
1×10−2
1×10−2
Batch size
8
8
8
8
Epochs
3
3
3
3
Gradient accumulation steps
4
4
4
4
Optimizer
AdamW
AdamW
AdamW
AdamW
Appendix
Table 10: Training parameters. We report the hyperparameter ranges explored across objectives; the main results use r=32 , α=64 , and objective-specific learning rates and batch sizes given in the text.
Parameter
Value
Temperature
0.5
Top- p sampling
0.9
Maximum new tokens
256 (GSM8K, SVAMP, MATHQA); 1024 (MBPP)
Do sample
True
Pad token ID
EOS token ID
Appendix
Table 11: Generation parameters used for all debate and evaluation runs unless otherwise noted.
Parameter
Value
Number of agents
5
Debate rounds
2
Batch debate size
32–256
Majority vote computed
True (diagnostics only)
Include debate context
True
Appendix
Table 12: Multi-agent debate configuration. Default settings are N=5 agents and M=2 rounds; sweeps in Section 4.2 vary these. “Majority vote” indicates that majority consensus is computed for diagnostics and baselines, not as the training label.
Table 14: Mean per-agent accuracy (%) after M=2 debate rounds. “Ensemble” and “Debate” are inference-only baselines; “SFT”, “DPO”, “KTO”, and “GRPO” denote post-training with the corresponding objective on the same debate traces. Column definitions match Table 1 . Best post-training objective per row is highlighted in bold.
Aggregation
Method
Initial
Exchange
Consensus
Ensemble
24.2
-
48.6
Sum
Debate
24.0
44.5
66.0
DPO
35.4
59.7
71.6
Mean
Debate
23.8
44.0
63.2
DPO
38.9
60.6
67.6
Appendix
Table 15: Oracle-protocol ablation on GSM8K using Qwen-2.5-1.5B-it, N=5 agents, and M=2 debate rounds. “Initial” denotes accuracy (%) before debate, “Exchange” denotes accuracy (%) after exchange, and “Consensus” denotes the accuracy (%) of the aggregated committee answer.
Figure 8: Comparison of confidence-sum and confidence-mean aggregation for answer-equivalence clusters on GSM8K with Qwen-2.5-1.5B-it and five agents. The agreement rate is the fraction of parseable agent answers assigned to the selected cluster. Left: frequency with which each aggregation rule selects an answer at each agreement rate. Right: correctness of the selected answer within each agreement-rate bin, measured against GSM8K reference answers for evaluation only. Confidence-sum aggregation retains the total confidence mass contributed by the committee, whereas confidence-mean aggregation normalizes away cluster size. Across agreement-rate bins, the sum rule selects more accurate oracle answers than the mean rule.
Method
Accuracy
Consensus
Avg. Agent
L1
Q1
Q2
P4
G4
Ensemble
78.2
30.9
3.6
26.0
1.4
76.8
46.8
Debate
79.2
36.8
16.6
57.6
3.2
71.6
35.2
SFT
79.8
37.4
17.8
54.2
4.2
75.6
35.4
DPO
87.0
68.2
71.0
75.8
41.8
81.2
71.4
KTO
86.6
59.2
52.0
71.6
22.6
83.6
66.0
Appendix
Table 17: Accuracy (%) of a heterogeneous committee on GSM8K. The committee consists of L1 (Llama-3.2-1B-it), Q1 (Qwen-2.5-1.5B-it), Q2 (Qwen3-1.7B-it), P4 (Phi-4-mini-3.8B-it), and G4 (Gemma-3-4B-it). “Consensus” denotes accuracy of aggregate solution. “Avg. Agent” denotes mean per-agent performance. The Δ row reports the per-agent change from Ensemble to DPO.
Method
Training oracle
Majority
Span-confidence
Gap
DPO
Span-confidence
71.2 (356)
71.6 (358)
+2
KTO
Span-confidence
68.4 (342)
70.0 (350)
+8
GRPO
Span-confidence
68.6 (343)
69.0 (345)
+2
DPO
Majority voting
68.6 (343)
70.0 (350)
+7
KTO
Majority voting
70.8 (354)
71.6 (358)
+4
GRPO
Majority voting
67.0 (335)
68.6 (343)
+8
Appendix
Table 22: Factorial evaluation of Qwen-2.5-1.5B-it on GSM8K test ( N=5 ). Rows are the training oracle. Columns are the test-time consensus, recomputed on the post-training debate. Entries are accuracy over 500 questions, with the correct count in parentheses. Span-confidence sums boxed-answer probabilities inside an answer cluster.
Training traces
Ensemble
Debate
SFT
DPO
KTO
GRPO
Independent samples
44.5
–
47.0
64.4
61.2
56.4
One exchange round
44.5
66.0
66.3±0.71
71.4±0.28
69.2±1.13
68.9±0.14
Gap
–
–
+19.3
+7.0
+8.0
+12.5
Appendix
Table 23: GSM8K consensus accuracy (%) for Qwen-2.5-1.5B-it. Ensemble and Debate are the instruction-tuned checkpoint before either training run. The one-exchange row is the GSM8K entry of Table 1 . The independent-sample row is a single run. The gap row subtracts that run from the one-exchange mean.
Model
Dataset
Incorrect (%)
Correct (%)
Unparseable
Wrong
GPT-4o
SVAMP
1.7
5.0
93.3
GSM8K
17.8
4.0
78.2
MathQA
51.4
4.6
44.0
Qwen-3.8-27B
SVAMP
55.3
13.7
31.0
GSM8K
87.0
8.6
14.4
Appendix
Table 25: Zero-shot single-agent performance of models on the evaluation splits at temperature 0 with a maximum completion budget of 256 tokens. Values are percentages of evaluated examples. Accuracy is the percentage of correct responses; wrong and unparseable responses are both counted as incorrect.
Dataset
Majority
GPT-4o judge
Gold cluster
GSM8K
65.6
71.6
72.8
MathQA
41.0
47.2
49.6
Appendix
Table 26: Test accuracy of the selected answer before training. Each entry is correct questions out of 500. The gold cluster is the best answer already present in the debate.
Selection
Debate
After DPO
Majority
65.6
71.2
GPT-4o judge
71.6
78.2
Gold cluster
72.8
78.6
Appendix
Table 27: GSM8K test accuracy before and after DPO. DPO trains on the GPT-4o split for 3 epochs. The after-DPO majority column is a vote over that trained committee, not a model trained on majority labels.
Model
Dataset
Trace mean
Trace median
Boxed mean
Boxed median
Llama-3.2-1B
GSM8K
0.926→0.807
0.931→0.819
0.989→0.966
1.000→0.999
Qwen-2.5-1.5B
0.864→0.694
0.873→0.718
0.989→0.968
0.999→0.999
Phi-4-mini
0.902→0.798
0.938→0.802
0.999→0.996
1.000→1.000
Gemma-3-4B
0.880→0.857
0.882→0.857
1.000→1.000
1.000→1.000
Llama-3.2-1B
SVAMP
0.924→0.787
0.931→0.804
0.993→0.984
1.000→1.000
Qwen-2.5-1.5B
0.836→0.659
0.844→0.678
0.985→0.974
0.999→0.999
Appendix
Table 28: Training-split confidence from temperature 0.01 to 1.0 .
Temperature
0.01
0.25
0.50
0.75
1.0
Boxed mean
0.989
0.989
0.984
0.982
0.968
Boxed median
0.999
0.999
0.999
0.999
0.999
Trace mean
0.864
0.855
0.834
0.790
0.694
Trace median
0.873
0.865
0.844
0.801
0.718
Appendix
Table 29: Qwen-2.5-1.5B-it on the 1,500 GSM8K training questions. Trace uses all 1,500 replies at each temperature.
Figure 9: Pass rate of the winning cluster on MBPP. Majority voting (blue) is a plurality over public-output clusters. Execution confidence (green) keeps the cluster with the most public cases passed. Hidden success-rate is the all-or-nothing check over public and challenge assertions. Public pass-rate pools the visible cases of the selected program. Debate is the shared pre-training trace. DPO is three epochs on that selector’s label.
Full success rate
Rubric hit rate
Oracle
Debate
DPO
Debate
DPO
Majority Voting
4.2
3.2
30.5
29.3
Hit Confidence
5.6
6.0
41.9
42.8
Appendix
Table 30: RaR-Medicine test split success rate and hit rate. Each cell is the mean fraction of checklist items satisfied.
Cluster
Agents n
Criteria met c
Score
Vote ( n )
3
2
3
Vote ( n )
1
8
1
Hit count ( n⋅c )
3
2
6
Hit count ( n⋅c )
1
8
8
Appendix
Table 31: Vote score and hit-count score on one question with K=10 criteria. The vote selects the repeated partial checklist. The hit-count score selects the checklist that meets more criteria.
Multi-agent debate can improve large language model reasoning by eliciting diverse hypotheses and critiques, yet its performance is often constrained by weak moderation. Common pipelines rely on fixed budgets, agreement-based stopping, or untrained judges, leading to redundant deliberation and unreliable evidence aggregation. We cast moderation as a meta-cognitive process, monitoring debate utility, controlling deliberation, and adjudicating a final answer, and introduce Meta-Moderator, a learnable framework that dynamically regulates debate and decides when to finalize an answer. Meta-Moderator is trained independently of the debaters via outcome-driven policy optimization, making debate regulation an explicit capability rather than an incidental effect of prompting. Across five benchmarks, Meta-Moderator outperforms widely used decision layers and transfers across tasks and system configurations. Further analyses show that it allocates debate more selectively and reduces mis-aggregation after informative hypotheses appear.
Wentao Hu, Zhuoyue Wan, Jinhao Shen +3
The Hong Kong Polytechnic University · Sichuan University
When should a language model answer directly, sample and vote, or engage in multi-agent debate? Recent work shows voting often explains much of the gain attributed to debate, while selective-debate systems activate deliberation only on uncertain examples. We ask: under a matched ceiling on generated tokens (960 per example), how much per-example routing headroom exists, and how much is recoverable from cheap pre-deliberation signals? We evaluate greedy decoding, three-sample voting, and a two-agent critique-revise debate on MuSiQue and GSM8K using Llama 3.1 8B Instruct and Ministral 3 8B Instruct. On MuSiQue, an oracle selecting the correct protocol per example gains +14.0 and +13.7 pp over the best fixed one. The best fixed protocol is model- and dataset-dependent: each (model, dataset) cell has a different winner. This headroom is hard to recover from cheap ex-ante signals. A vote-entropy threshold is the only controller that directionally beats the best fixed protocol on both models (+1.3 and +1.7 pp), though individual paired-bootstrap CIs include zero. A joint analysis (meta-analysis +1.6 pp, p=0.125; Bayesian P(both>0)=0.59) is directionally consistent but not significant. Learned controllers (LR, GBT) do not outperform the threshold. The key finding is structural: vote entropy predicts where debate is safe, not where debate is needed. High entropy sharply reduces debate backfire, but 66% of debate-helpful examples (31/47) occur when voting is unanimous but wrong. A single-prompt self-critique probe on Llama flips the answer in 127/127 unanimous cases, yielding zero mutual information with the debate-helpful label; we cannot rule out a prompt-compliance artifact, but either interpretation disqualifies the probe as a router. Recovering the remaining headroom requires behavioral probes that avoid format-compliance confounds at the 8B scale.
Multi-agent debate (MAD) can improve large language model reasoning, but fixed debate pipelines often waste computation and can amplify correlated errors among similar agents. We propose ARMOR-MAD, a training-free heterogeneous MAD framework that treats debate as conditional computation. ARMOR-MAD combines three components: Pre-debate Agreement Routing (PAR) decides whether independently generated Round-0 answers require debate; Early Agreement Stopping Evaluator (EASE) stops debate after convergence; and Semantic Outlier Detection (SOD) down-weights abnormal final answers during aggregation. Across MATH Level 5, GSM8K, MMLU, and MMLU-Pro, ARMOR-MAD consistently improves over fixed-round heterogeneous debate with the same model pool, reaching 65.5%, 96.5%, 90.0%, and 81.5% accuracy, respectively. The results suggest that genuine model heterogeneity and agreement-based control are both important for making MAD more accurate and efficient.
Fuqiang Niu, Bowen Zhang
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China