Alignment by Confident Debate: Preference Learning from Multi-Agent Debate

Nikhil Verma; Dingxian Wang; Mohammad Reza Rezaei; Andrew Rabinovich; Zhao Chen

cs.AINov 6, 2026

Alignment by Confident Debate: Preference Learning from Multi-Agent Debate

Authors: Nikhil Verma, Dingxian Wang, Mohammad Reza Rezaei, Andrew Rabinovich, Zhao Chen

Organizations: Upwork Inc.

Abstract

Post-training language models for reasoning typically rely on gold answers, task-specific verifiers, or learned reward models, limiting scalability when such signals are scarce. Multi-agent debate offers a self-supervised alternative, but converting debate into reliable training supervision remains challenging: majority voting weights agreeing responses equally, lacks a principled tie-breaking rule, and is ill-defined for open-ended tasks without a canonical answer; likelihood-based scoring can conflate fluency with correctness; and homogeneous committees may reinforce correlated errors. We introduce Alignment by Confident Debate (AbCD), which converts debate traces into preference data without gold answer labels. A sampling-diverse committee debates each problem, after which final answers are clustered by task-level semantic equivalence. We use verifiable tasks (mathematical reasoning, multiple-choice quantitative QA, and executable code-generation) as a controlled test bed: their answer keys or executable checkers let us evaluate the quality of debate-derived supervision against ground truth, even though the oracle itself never uses gold labels. Post-training on the resulting preferences outperforms inference-only ensembling and debate, as well as SFT on the same traces, across five model families spanning 1B to 8B parameters and four datasets, improving consensus accuracy, with a maximum gain of $21.5_{pp}$ . Ablations show that the approach transfers to a larger model and to code generation, while also revealing limitations of heterogeneous committees and agreement-based weighting. Overall, the results suggest that debate traces can yield useful preference supervision when answer equivalence is combined with confidence-weighted disagreement.

Figures & tables

Figure 1: Committee behavior for Qwen-2.5-1.5B-it on GSM8K. The committee size varies from one to five agents under independent sampling, after which the number of debate rounds is varied with five agents. Additional rounds increase agreement but do not improve consensus accuracy beyond the first exchange and can reduce correctness.

Figure 2: AbCD instantiates $N$ sampling-diverse agents from the base policy $\pi_{\theta}$ and lets them debate for $M$ generations. Final traces are clustered by task-level answer equivalence, and the cluster with the largest aggregate confidence on the claimed-answer span is selected. Traces in the winning cluster are preferred, while the remaining traces are dispreferred (green $\succ$ red). The resulting preference supervision updates $\pi_{\theta}$ without gold answers in the oracle.

Panel of Figure 3 — Figure 3: Committee behavior for Qwen-2.5-1.5B-it on GSM8K. Subfigures show accuracy by committee size (a), accuracy by debate rounds (b), agreement-rate across rounds (c), and correctness across rounds (d).

Figure 4: Change in AUROC from the initial-response round to the first exchange round for full-trace and claimed answer-span confidence. Each bar is one model–dataset pair across GSM8K, SVAMP, and MathQA. Positive values indicate improved discrimination after debate.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Datasets	What is aggregated	Confidence
Discrete math	GSM8K, SVAMP, MathQA	The same extracted answer	Span confidence
Executable code	MBPP	Programs with the same public outputs	Public tests passed
Checklist text	RaR	Identical satisfied-criteria patterns	Weighted hit score, penalties subtracted

Appendix

Table 7: How agent answers are clustered and scored for the training label.

Figure 5: Committee behavior for Llama-3.1-8B-it model on GSM8K. The consensus accuracy by committee size beyond 5 agents shows diminishing returns.

Panel of Figure 6 — Figure 6: Effect of additional debate rounds on GSM8K across model families. Left: Consensus accuracy at a fixed committee size as the number of exchange rounds increases. Right: Oracle precision stratified by agreement rate, where $k/5$ denotes that $k$ of five agents produce parseable answers in the selected answer-equivalence cluster. The first exchange round often improves consensus accuracy, while subsequent rounds can increase agreement without improving correctness and can reduce oracle precision at several agreement levels. Reference answers are used only for evaluation.

Panel of Figure 7 — Figure 7: Effect of additional debate rounds on Qwen-2.5-1.5B-it across datasets. Left: Consensus accuracy at a fixed committee size as the number of exchange rounds increases. Right: Oracle precision stratified by agreement rate, where $k/5$ denotes that $k$ of five agents produce parseable answers in the selected answer-equivalence cluster. Reference answers are used only for evaluation.

Committee	Initial response accuracy	Avg. agent performance	Consensus correctness
	Llama-3.2-1B-it
Clones	3.2	4.6	6.4
Temperature-diverse	4.4	7.4	17.0
Role-diverse	5.6	7.3	20.6
Seed-diverse	3.2	4.3	6.6
	Qwen-2.5-1.5B-it

Appendix

Table 8: Inference-only GSM8K accuracy (%) at

N{=}5

M{=}2

. Initial response accuracy is single-sample accuracy before any ensemble or debate; Avg. agent performance is average per-agent accuracy after

M

rounds of debate; Consensus correctness is aggregated response accuracy.

Model	Dataset	CoT R1	CoT R2	$\bm{\Delta}$ CoT	Box R1	Box R2	$\bm{\Delta}$ Box
Llama-3.2-1B-it	GSM8K	0.60	0.50	-0.10	0.63	0.72	0.10
	SVAMP	0.55	0.56	0.01	0.59	0.77	0.18
	MathQA	0.64	0.54	-0.10	0.63	0.65	0.02
Qwen-2.5-1.5B-it	GSM8K	0.74	0.66	-0.08	0.71	0.80	0.09
	SVAMP	0.66	0.57	-0.09	0.68	0.76	0.08
	MathQA	0.75	0.68	-0.07	0.73	0.77	0.04

Appendix

Table 9: Full-trace (CoT) versus claimed-answer-span (Box) AUROC before and after one debate round.

\Delta=\mathrm{R2}-\mathrm{R1}

Parameter	SFT	GRPO	DPO	KTO
Learning rate	$1\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-5}$
Weight decay	$1\times 10^{-2}$	$1\times 10^{-2}$	$1\times 10^{-2}$	$1\times 10^{-2}$
Batch size	8	8	8	8
Epochs	3	3	3	3
Gradient accumulation steps	4	4	4	4
Optimizer	AdamW	AdamW	AdamW	AdamW

Appendix

Table 10: Training parameters. We report the hyperparameter ranges explored across objectives; the main results use

r{=}32

\alpha{=}64

, and objective-specific learning rates and batch sizes given in the text.

Parameter	Value
Temperature	0.5
Top- $p$ sampling	0.9
Maximum new tokens	256 (GSM8K, SVAMP, MATHQA); 1024 (MBPP)
Do sample	True
Pad token ID	EOS token ID

Appendix

Table 11: Generation parameters used for all debate and evaluation runs unless otherwise noted.

Parameter	Value
Number of agents	5
Debate rounds	2
Batch debate size	32–256
Majority vote computed	True (diagnostics only)
Include debate context	True

Appendix

Table 12: Multi-agent debate configuration. Default settings are

N{=}5

agents and

M{=}2

rounds; sweeps in Section 4.2 vary these. “Majority vote” indicates that majority consensus is computed for diagnostics and baselines, not as the training label.

Parameter	Value
Base models	Qwen2.5-1.5B-it, Llama-3.2-1B-it, Phi-4-mini-3.8B-it, Gemma-3-4B-it, Llama-3.1-8B-it
Training datasets	GSM8K, SVAMP, MathQA, MBPP
Test datasets	GSM8K, MathQA, SVAMP, MBPP
Training size	1500 samples (GSM8K, MathQA); 700 samples (SVAMP); 374 samples (MBPP)
Test size	500 samples (GSM8K, MathQA); 300 samples (SVAMP); 200 samples (MBPP)
Use quantization	4-bit (BitsAndBytesConfig)

Appendix

Table 13: Base models and dataset splits.

Model	Dataset	Baselines			Post-training
Model	Dataset	Ensemble	Debate	SFT	DPO	KTO	GRPO
Llama-3.2-1B-it	SVAMP	4.5	5.1	$8.6_{\pm 0.85}$	$\textbf{18.3}_{\pm 1.48}$	$15.3_{\pm 2.69}$	$10.2_{\pm 0.14}$
	GSM8K	4.2	5.4	$5.8_{\pm 0.21}$	$\textbf{13.2}_{\pm 2.26}$	$9.85_{\pm 2.47}$	$5.4_{\pm 1.41}$
	MathQA	1.8	1.7	$3.8_{\pm 1.98}$	$\textbf{3.5}_{\pm 0.49}$	$2.6_{\pm 0.42}$	$2.3_{\pm 0.58}$
Qwen-2.5-1.5B-it	SVAMP	50.9	60.9	$63.6_{\pm 0.64}$	$\textbf{70.6}_{\pm 1.91}$	$64.5_{\pm 3.75}$	$59.4_{\pm 0.14}$
	GSM8K	24.2	44.5	$45.3_{\pm 0.64}$	$\textbf{60.8}_{\pm 1.56}$	$54.1_{\pm 6.08}$	$54.7_{\pm 0.35}$

Appendix

Table 14: Mean per-agent accuracy (%) after

M{=}2

debate rounds. “Ensemble” and “Debate” are inference-only baselines; “SFT”, “DPO”, “KTO”, and “GRPO” denote post-training with the corresponding objective on the same debate traces. Column definitions match Table 1 . Best post-training objective per row is highlighted in bold.

Aggregation	Method	Initial	Exchange	Consensus
	Ensemble	24.2	-	48.6
Sum	Debate	24.0	44.5	66.0
Sum	DPO	35.4	59.7	71.6
Mean	Debate	23.8	44.0	63.2
Mean	DPO	38.9	60.6	67.6

Appendix

Table 15: Oracle-protocol ablation on GSM8K using Qwen-2.5-1.5B-it,

N{=}5

agents, and

M{=}2

debate rounds. “Initial” denotes accuracy (%) before debate, “Exchange” denotes accuracy (%) after exchange, and “Consensus” denotes the accuracy (%) of the aggregated committee answer.

Panel of Figure 8 — Figure 8: Comparison of confidence-sum and confidence-mean aggregation for answer-equivalence clusters on GSM8K with Qwen-2.5-1.5B-it and five agents. The agreement rate is the fraction of parseable agent answers assigned to the selected cluster. Left: frequency with which each aggregation rule selects an answer at each agreement rate. Right: correctness of the selected answer within each agreement-rate bin, measured against GSM8K reference answers for evaluation only. Confidence-sum aggregation retains the total confidence mass contributed by the committee, whereas confidence-mean aggregation normalizes away cluster size. Across agreement-rate bins, the sum rule selects more accurate oracle answers than the mean rule.

Method	Accuracy
Method	Consensus	Avg. Agent	L1	Q1	Q2	P4	G4
Ensemble	78.2	30.9	3.6	26.0	1.4	76.8	46.8
Debate	79.2	36.8	16.6	57.6	3.2	71.6	35.2
SFT	79.8	37.4	17.8	54.2	4.2	75.6	35.4
DPO	87.0	68.2	71.0	75.8	41.8	81.2	71.4
KTO	86.6	59.2	52.0	71.6	22.6	83.6	66.0

Appendix

Table 17: Accuracy (%) of a heterogeneous committee on GSM8K. The committee consists of L1 (Llama-3.2-1B-it), Q1 (Qwen-2.5-1.5B-it), Q2 (Qwen3-1.7B-it), P4 (Phi-4-mini-3.8B-it), and G4 (Gemma-3-4B-it). “Consensus” denotes accuracy of aggregate solution. “Avg. Agent” denotes mean per-agent performance. The

\Delta

row reports the per-agent change from Ensemble to DPO.

Method	Training oracle	Majority	Span-confidence	Gap
DPO	Span-confidence	71.2 (356)	71.6 (358)	+2
KTO	Span-confidence	68.4 (342)	70.0 (350)	+8
GRPO	Span-confidence	68.6 (343)	69.0 (345)	+2
DPO	Majority voting	68.6 (343)	70.0 (350)	+7
KTO	Majority voting	70.8 (354)	71.6 (358)	+4
GRPO	Majority voting	67.0 (335)	68.6 (343)	+8

Appendix

Table 22: Factorial evaluation of Qwen-2.5-1.5B-it on GSM8K test (

N{=}5

). Rows are the training oracle. Columns are the test-time consensus, recomputed on the post-training debate. Entries are accuracy over 500 questions, with the correct count in parentheses. Span-confidence sums boxed-answer probabilities inside an answer cluster.

Training traces	Ensemble	Debate	SFT	DPO	KTO	GRPO
Independent samples	44.5	–	47.0	64.4	61.2	56.4
One exchange round	44.5	66.0	$66.3_{\pm 0.71}$	$\textbf{71.4}_{\pm 0.28}$	$69.2_{\pm 1.13}$	$68.9_{\pm 0.14}$
Gap	–	–	$+19.3$	$+7.0$	$+8.0$	$+12.5$

Appendix

Table 23: GSM8K consensus accuracy (%) for Qwen-2.5-1.5B-it. Ensemble and Debate are the instruction-tuned checkpoint before either training run. The one-exchange row is the GSM8K entry of Table 1 . The independent-sample row is a single run. The gap row subtracts that run from the one-exchange mean.

Model	Dataset	Incorrect (%)		Correct (%)
Model	Dataset	Unparseable	Wrong	Correct (%)
GPT-4o	SVAMP	1.7	5.0	93.3
	GSM8K	17.8	4.0	78.2
	MathQA	51.4	4.6	44.0
Qwen-3.8-27B	SVAMP	55.3	13.7	31.0
	GSM8K	87.0	8.6	14.4

Appendix

Table 25: Zero-shot single-agent performance of models on the evaluation splits at temperature

0

with a maximum completion budget of

256

tokens. Values are percentages of evaluated examples. Accuracy is the percentage of correct responses; wrong and unparseable responses are both counted as incorrect.

Dataset	Majority	GPT-4o judge	Gold cluster
GSM8K	65.6	71.6	72.8
MathQA	41.0	47.2	49.6

Appendix

Table 26: Test accuracy of the selected answer before training. Each entry is correct questions out of 500. The gold cluster is the best answer already present in the debate.

Selection	Debate	After DPO
Majority	65.6	71.2
GPT-4o judge	71.6	78.2
Gold cluster	72.8	78.6

Appendix

Table 27: GSM8K test accuracy before and after DPO. DPO trains on the GPT-4o split for 3 epochs. The after-DPO majority column is a vote over that trained committee, not a model trained on majority labels.

Model	Dataset	Trace mean	Trace median	Boxed mean	Boxed median
Llama-3.2-1B	GSM8K	$0.926\to 0.807$	$0.931\to 0.819$	$0.989\to 0.966$	$1.000\to 0.999$
Qwen-2.5-1.5B		$0.864\to 0.694$	$0.873\to 0.718$	$0.989\to 0.968$	$0.999\to 0.999$
Phi-4-mini		$0.902\to 0.798$	$0.938\to 0.802$	$0.999\to 0.996$	$1.000\to 1.000$
Gemma-3-4B		$0.880\to 0.857$	$0.882\to 0.857$	$1.000\to 1.000$	$1.000\to 1.000$
Llama-3.2-1B	SVAMP	$0.924\to 0.787$	$0.931\to 0.804$	$0.993\to 0.984$	$1.000\to 1.000$
Qwen-2.5-1.5B		$0.836\to 0.659$	$0.844\to 0.678$	$0.985\to 0.974$	$0.999\to 0.999$

Appendix

Table 28: Training-split confidence from temperature

0.01

1.0

	Temperature
	$0.01$	$0.25$	$0.50$	$0.75$	$1.0$
Boxed mean	$0.989$	$0.989$	$0.984$	$0.982$	$0.968$
Boxed median	$0.999$	$0.999$	$0.999$	$0.999$	$0.999$
Trace mean	$0.864$	$0.855$	$0.834$	$0.790$	$0.694$
Trace median	$0.873$	$0.865$	$0.844$	$0.801$	$0.718$

Appendix

Table 29: Qwen-2.5-1.5B-it on the

1{,}500

GSM8K training questions. Trace uses all

1{,}500

replies at each temperature.

Figure 9: Pass rate of the winning cluster on MBPP. Majority voting (blue) is a plurality over public-output clusters. Execution confidence (green) keeps the cluster with the most public cases passed. Hidden success-rate is the all-or-nothing check over public and challenge assertions. Public pass-rate pools the visible cases of the selected program. Debate is the shared pre-training trace. DPO is three epochs on that selector’s label.

	Full success rate		Rubric hit rate
Oracle	Debate	DPO	Debate	DPO
Majority Voting	4.2	3.2	30.5	29.3
Hit Confidence	5.6	6.0	41.9	42.8

Appendix

Table 30: RaR-Medicine test split success rate and hit rate. Each cell is the mean fraction of checklist items satisfied.

Cluster	Agents $n$	Criteria met $c$	Score
Vote ( $n$ )	3	2	3
Vote ( $n$ )	1	8	1
Hit count ( $n\cdot c$ )	3	2	6
Hit count ( $n\cdot c$ )	1	8	8

Appendix

Table 31: Vote score and hit-count score on one question with

K{=}10

criteria. The vote selects the repeated partial checklist. The hit-count score selects the checklist that meets more criteria.

Preference Learning for LLMs

See all 14 papers

Meta-Moderator: Empowering Multi-Agent Debate with Meta-Cognition
Aug 24, 2026Wentao Hu, Zhuoyue Wan, Jinhao Shen +3Multi-Agent Debate Multi-Agent Deliberation
The Hong Kong Polytechnic University · Sichuan University
Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols
May 10, 2026Julia Hu, Alfred Shen, Kumar LakshmipathiLLM Evaluation Multi-Agent Debate
Amazon Web Services
ARMOR-MAD: Adaptive Routing for Heterogeneous Multi-Agent Debate in Large Language Model Reasoning
Jun 11, 2026Fuqiang Niu, Bowen ZhangMulti-Agent Debate Multi-Agent Consensus
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China

Aug 24, 2026J/K move · Enter open · S save

Meta-Moderator: Empowering Multi-Agent Debate with Meta-Cognition

Wentao Hu, Zhuoyue Wan, Jinhao Shen, Chen Jason Zhang+2

The Hong Kong Polytechnic University · Sichuan University

Multi-Agent Debate Multi-Agent Deliberation Metacognition in Language Models RL for Language Model Reasoning

Open paper AskPDF arXiv

Multi-agent debate can improve large language model reasoning by eliciting diverse hypotheses and critiques, yet its performance is often constrained by weak moderation. Common pipelines rely on fixed budgets, agreement-based stopping, or untrained judges, leading to redundant deliberation and unreliable evidence aggregation. We cast moderation as a meta-cognitive process, monitoring debate utility, controlling deliberation, and adjudicating a final answer, and introduce Meta-Moderator, a learnable framework that dynamically regulates debate and decides when to finalize an answer. Meta-Moderator is trained independently of the debaters via outcome-driven policy optimization, making debate regulation an explicit capability rather than an incidental effect of prompting. Across five benchmarks, Meta-Moderator outperforms widely used decision layers and transfers across tasks and system configurations. Further analyses show that it allocates debate more selectively and reduces mis-aggregation after informative hypotheses appear.

Abstract

Preference Learning for LLMs

Explore similar work

Meta-Moderator: Empowering Multi-Agent Debate with Meta-Cognition

Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols

ARMOR-MAD: Adaptive Routing for Heterogeneous Multi-Agent Debate in Large Language Model Reasoning