From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation

Mengdie Flora Wang; Haochen Xie; Guanghui Wang; Aijing Gao; Guang Yang; Ziyuan Li; Qucy Wei Qiu; Fangwei Han

cs.AIApr 9, 2026

From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation

Authors: Mengdie Flora Wang, Haochen Xie, Guanghui Wang, Aijing Gao, Guang Yang, Ziyuan Li, Qucy Wei Qiu, Fangwei Han, +4 more

Organizations: AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China

Abstract

Multi-agent debate improves LLM reasoning, yet agreement among agents is not evidence of correctness. When agents converge on a wrong answer through social reinforcement, consensus-based stopping commits that error to an automated action with no recourse. We introduce Conformal Social Choice, a post-hoc decision layer that converts debate outputs into calibrated act-versus-escalate decisions. Verbalized probability distributions from heterogeneous agents are aggregated via a linear opinion pool and calibrated with split conformal prediction, yielding prediction sets with a marginal coverage guarantee: the correct answer is included with probability ${\geq}\,1{-}α$ , without assumptions on individual model calibration. A hierarchical action policy maps singleton sets to autonomous action and larger sets to human escalation. On eight MMLU-Pro domains with three agents (Claude Haiku, DeepSeek-R1, Qwen-3 32B), coverage stays within 1--2 points of the target. The key finding is not that debate becomes more accurate, but that the conformal layer makes its failures actionable: 81.9% of wrong-consensus cases are intercepted at $α{=}0.05$ . Because the layer refuses to act on cases where debate is confidently wrong, the remaining conformal singletons reach 90.0--96.8% accuracy (up to 22.1pp above consensus stopping) -- a selection effect, not a reasoning improvement. This safety comes at the cost of automation, but the operating point is user-adjustable via $α$ .

Figures & tables

Figure 1: The cost of acting without calibrated refusal. Consensus stopping never escalates (0% escalation rate) but commits to high error rates (up to 32.1%). Conformal stopping trades automation for safety: by escalating uncertain cases to human review, it dramatically reduces error among the cases it does act on. At $\alpha{=}0.05$ , 81.9% of wrong-consensus cases are escalated instead of acted upon. Each arrow connects the same domain under the two stopping rules.

Figure 2: From debate output to deployment decision. Heterogeneous LLM agents debate for $T$ rounds, producing verbalized probability distributions (Stage 1). These are aggregated via a linear opinion pool into social probabilities (Stage 2), calibrated using a held-out set to compute the conformal threshold $\hat{q}$ (Stage 3), and used to construct prediction sets with a hierarchical action policy (Stage 4). The pipeline adds a calibrated refusal layer: $\mathcal{C}(x)$ satisfies a marginal coverage guarantee ( $\Pr[y\in\mathcal{C}(x)]\geq 1{-}\alpha$ ); singletons trigger autonomous action under the calibrated policy, while larger sets trigger escalation.

Figure 3: Coverage maintenance across debate rounds. At

\alpha{=}0.05

, coverage remains near 95% across all rounds for most domains. At

\alpha{=}0.10

, coverage clusters around the 90% target.

		Engineering		Law		Chemistry		Physics		Math		Economics		Health		Psychology
Metric	Rd	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10
$\hat{q}$	0	.975	.952	.980	.945	.962	.784	.950	.769	.750	.605	.937	.764	.970	.904	.954	.890
	1	.987	.960	.990	.967	.982	.702	.923	.651	.718	.359	.974	.744	.984	.924	.967	.933
	2	.990	.963	.994	.977	.987	.669	.957	.492	.704	.229	.980	.823	.990	.930	.978	.944
	3	.995	.980	.997	.984	.990	.733	.973	.537	.740	.126	.980	.852	.992	.940	.983	.950
Coverage (%)	0	93.8	87.6	97.6	89.5	95.0	89.0	95.1	89.4	94.5	88.6	96.0	90.5	96.3	91.4	94.7	88.7

Table 1: Conformal prediction results with early stopping on MMLU-Pro (Haiku + DeepSeek-R1 + Qwen-3 32B). Two sub-columns per domain:

\alpha{=}0.05

(left) and

\alpha{=}0.10

(right).

\hat{q}

= calibration threshold, Coverage = empirical coverage (%),

|\mathcal{C}|

= average prediction set size, Singleton Rate = cumulative fraction of samples resolved as singletons (

|\mathcal{C}|{=}1

) by that round (%), Singleton Acc. = accuracy among samples whose conformal set reached singleton at that round (%). Rows 0–3 correspond to debate rounds. Coverage remains near the target across all domains, while set size shrinks across rounds—with slower shrinkage on harder domains (Math

\approx

1 vs. Law

\approx

7). Entropy-based weighting yields near-identical results (Table 6 ; mean

|\Delta\text{Coverage}|=0.6\%

Figure 4: Calibrated refusal in action ( $\alpha{=}0.05$ ). In both examples, agents unanimously converge to the wrong answer through social reinforcement. Consensus alone would commit this error to an automated action; conformal prediction keeps the correct answer inside the set and forces escalation to human review.

Domain	$n$	Consensus	Conformal	Consensus	Conformal	Consensus	Conformal	$\Delta$ Acc
		Avg Round		Singleton %		Singleton Acc.
Engineering	485	1.80	1.67	90.1	52.6	82.6	95.5	+12.9
Law	551	1.81	2.24	87.7	6.2	67.9	90.0	+22.1
Chemistry	566	1.63	1.57	95.2	78.8	90.0	96.8	+6.8
Physics	650	1.55	1.53	95.1	87.2	90.9	95.7	+4.8
Math	676	1.43	1.29	97.8	98.2	94.9	94.6	$-$ 0.3

Table 2: Safer but more conservative stopping (

\alpha{=}0.05

, round 3). Compared with consensus stopping, conformal stopping resolves fewer cases automatically but yields substantially higher accuracy on the cases it does resolve. The accuracy gain is a selection effect explained by Table 3 .

		Wrong-Consensus Convergence			Wrong Consensus		Correct Consensus
Domain	$n$	Disagree	$\to$ WC	WC%	Count	Rejected%	Count	Rejected%
Engineering	485	312	75	24.0	83	88.0	375	41.1
Law	551	388	135	34.8	162	97.5	344	95.9
Chemistry	566	288	55	19.1	63	82.5	494	18.8
Physics	650	313	50	16.0	58	70.7	570	8.4
Math	676	256	31	12.1	35	11.4	631	0.6

Table 3: Wrong-consensus convergence and conformal interception at round 3 (

\alpha{=}0.05

\to

WC = converge to unanimous wrong consensus. Rejected% = fraction where

|\mathcal{C}|>1

prevents automated action. Conformal prediction intercepts 81.9% of wrong-consensus errors while over-rejecting 31.9% of correct ones.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Model(s)	Engineering	Law	Chemistry	Physics	Math	Economics	Health	Psychology
Greedy	Claude Haiku	70.1	60.5	83.2	84.8	90.2	83.8	75.9	80.8
	DeepSeek-R1	58.4	63.9	77.2	78.4	88.4	82.8	76.8	79.7
	Qwen-3 32B	50.5	39.4	62.4	62.0	66.8	76.4	68.8	70.3
Self-Reflection (Rd 3)	Claude Haiku	72.7	59.5	86.6	86.2	92.1	85.2	76.0	80.6
	DeepSeek-R1	78.3	66.3	88.7	87.4	93.0	85.6	80.1	82.1
	Qwen-3 32B	56.8	39.4	69.5	72.2	78.5	78.1	68.6	72.6

Appendix

Table 4: Point-estimate baselines (%) on MMLU-Pro. H =Claude Haiku, D =DeepSeek-R1, Q =Qwen-3 32B. Self-reflection and debate report final-round (round 3) accuracy. Rd 0 corresponds to static majority voting (no debate). Debate + majority voting is the strongest point-estimate method, but point estimates provide no mechanism to flag incorrect predictions; the set-valued conformal approach of Table 1 addresses this gap.

Domain	R0	R1	R2	R3
Engineering	36.3%	40.9%	14.3%	8.5%
Law	34.2%	38.3%	14.7%	12.8%
Chemistry	53.2%	35.9%	7.2%	3.7%
Physics	52.3%	35.2%	8.2%	4.4%
Math	60.8%	31.8%	4.8%	2.6%
Economics	69.2%	21.7%	5.5%	3.7%

Appendix

Table 5: Distribution of samples across stopping rounds for consensus-based early stopping. The majority of samples stop at rounds 0–1, with domain difficulty reflected in the fraction reaching the final round.

		Engineering		Law		Chemistry		Physics		Math		Economics		Health		Psychology
Metric	Rd	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10	.05	.10
$\hat{q}$	0	.992	.979	.985	.960	.986	.850	.973	.815	.785	.507	.957	.796	.980	.917	.962	.911
	1	.989	.970	.992	.970	.986	.696	.943	.641	.769	.343	.980	.793	.987	.928	.971	.938
	2	.992	.967	.995	.978	.990	.663	.962	.488	.703	.178	.983	.824	.990	.935	.980	.944
	3	.995	.980	.997	.984	.990	.734	.968	.535	.801	.111	.984	.852	.992	.940	.983	.950
Coverage (%)	0	95.3	86.6	96.2	90.4	95.2	89.2	94.8	88.8	93.9	89.2	96.0	90.8	97.3	90.5	94.7	89.0

Appendix

Table 6: Conformal prediction results with entropy-based weighting (

\lambda{=}1

) and early stopping on MMLU-Pro. Format and metric names identical to Table 1 (uniform weighting). Differences are negligible: mean

|\Delta\text{Coverage}|=0.6\%

, mean

|\Delta\text{Set Size}|=0.06

, mean

|\Delta\text{Singleton Acc.}|=0.4\%

(see Table 7 for per-domain breakdown).

Domain	$\boldsymbol{\alpha}$	$\Delta$ Coverage	$\Delta$ Set Size	$\Delta$ Sing. Acc.
Engineering	.05	$-$ 0.4	$-$ 0.03	$-$ 0.2
	.10	+0.2	+0.00	$-$ 0.2
Law	.05	$-$ 0.5	$-$ 0.37	$-$ 0.7
	.10	$-$ 0.9	$-$ 0.11	$-$ 0.7
Chemistry	.05	$-$ 0.5	$-$ 0.03	+0.3
	.10	$-$ 1.1	+0.00	$-$ 0.7

Appendix

Table 7: Final-round differences between entropy and uniform weighting (entropy

-

uniform) in coverage (%), average set size, and singleton accuracy (%). Differences are small: mean

|\Delta\text{Coverage}|=0.6\%

, mean

|\Delta\text{Set Size}|=0.06

, mean

|\Delta\text{Singleton Acc.}|=0.4\%

, max

|\Delta\text{Coverage}|=2.1\%

. The debate consensus mechanism dominates the aggregation weighting strategy.

		Engineering	Law	Chemistry	Physics	Math	Economics	Health	Psychology
$\hat{q}$	0	.997	.994	.997	.997	.990	.993	.997	.989
	1	.997	1.00	.999	.000	.997	.997	1.00	.997
	2	1.00	1.00	1.00	1.00	.999	1.00	1.00	1.00
	3	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Coverage (%)	0	99.0	99.3	99.3	99.4	99.0	99.3	99.3	99.8
	1	98.6	100.0	99.3	99.4	99.6	99.0	100.0	99.5

Appendix

Table 8: Conformal prediction results at

\alpha=0.01

(99% target coverage) across debate rounds 0–3 on MMLU-Pro. Unlike

\alpha\in\{0.05,0.10\}

, prediction set sizes increase with debate rounds due to calibration threshold saturation : as debate sharpens confidence, the few remaining errors produce extreme non-conformity scores that push

\hat{q}\to 1.0

at the 99th percentile, collapsing all labels into the prediction set.

		$\to$ Wrong Consensus (%)
Domain	Init. Disagr.	Rd 1	Rd 2	Rd 3
Engineering	312	37 (11.9)	65 (20.8)	75 (24.0)
Law	388	90 (23.2)	125 (32.2)	135 (34.8)
Chemistry	288	29 (10.1)	45 (15.6)	55 (19.1)
Physics	313	31 (9.9)	44 (14.1)	50 (16.0)
Math	256	19 (7.4)	30 (11.7)	31 (12.1)

Appendix

Table 9: Wrong-consensus convergence among initially disagreeing cases. Init. Disagr. = inference samples without unanimous consensus at round 0. Columns show the count (and rate) of samples converging to unanimous wrong consensus by each round. The monotonically increasing trend confirms that extended debate amplifies sycophantic convergence: by round 3, nearly one in four initially-disputed questions (23.9%) ends in wrong consensus.

	$\alpha=0.05$				$\alpha=0.10$
Domain	Rd 0	Rd 1	Rd 2	Rd 3	Rd 0	Rd 1	Rd 2	Rd 3
Engineering	0	0	0	0	0	0	0	0
Law	0	0	0	0	1	0	0	0
Chemistry	1	0	0	0	14	19	8	2
Physics	0	0	0	0	24	19	12	6
Math	17	9	2	2	31	3	0	0

Appendix

Table 10: New wrong singletons introduced by conformal prediction without prior unanimous wrong consensus: cases where agents do not all agree on the wrong answer, but the conformal set collapses to

|\mathcal{C}|{=}1

containing the wrong answer. Counts per round; % of

n

relative to total inference samples (

n{=}4{,}158

). At

\alpha{=}0.05

round 3, conformal catches 480 wrong-consensus errors while introducing only 2 new wrong singletons (ratio 240:1). At

\alpha{=}0.10

the ratio is 41:1.

Law

Health

Economics

Psychology

Chemistry

Engineering

Math

Physics

Overall

Model

Fail

Rate

Fail

Rate

Fail

Rate

Fail

Rate

Fail

Rate

Fail

Rate

Fail

Rate

Fail

Rate

Claude-Haiku

1/4,404

0.02%

1/3,272

0.03%

0/3,376

0.00%

0/3,192

0.00%

2/4,528

0.04%

0/3,876

0.00%

2/5,404

0.04%

1/5,196

0.02%

DeepSeek-R1

128/4,404

2.91%

41/3,272

1.25%

54/3,376

1.60%

37/3,192

1.16%

118/4,528

2.61%

129/3,876

3.33%

67/5,404

1.24%

108/5,196

2.08%

2.05%

Qwen-32B

2/4,404

0.05%

4/3,272

0.12%

5/3,376

0.15%

0/3,192

0.00%

18/4,528

0.40%

21/3,876

0.54%

16/5,404

0.30%

9/5,196

0.17%

0.23%

All Models: 764 / 99,744

0.77%

Appendix

Table 11: Response parsing failure rates across models and domains (all rounds combined). Each cell shows failures / total responses (%).

Explore similar work

Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate
May 30, 2026Xiqi Hao, Zengqing Wu, Yu-Xuan Qiu +4LLM Agent Evaluation Multi-Agent Debate
Beijing Institute of Technology, Zhuhai · University of Osaka · Shenzhen University
Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate
Aug 4, 2026Hao Wu, Shoucheng Song, Chang Yao +4Multi-Agent Debate RL for LLM Agents
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing, China
The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate
Apr 29, 2026Blaž Bertalanič, Carolina FortunaLLM Self-Correction AI Agent Evaluation
Jožef Stefan Institute Ljubljana, Slovenia

May 30, 2026J/K move · Enter open · S save

Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

Xiqi Hao, Zengqing Wu, Yu-Xuan Qiu, Chuan Xiao+3

Beijing Institute of Technology, Zhuhai · University of Osaka · Shenzhen University

LLM Agent Evaluation Multi-Agent Debate Multi-Agent LLMs

Open paper AskPDF arXiv

Multi-agent debate (MAD) is a promising strategy for improving LLM reasoning, but when agents converge on a shared answer, it is unclear whether that convergence reflects genuine deliberation or social compliance. We show that the conventional answer flip rate conflates three distinct mechanisms: spontaneous instability, stance-induced conformity, and reasoning-induced persuasion. Our three-source decomposition framework isolates each through controlled counterfactual conditions. In the primary MMLU-Pro setting, 37% of agent-question observations change under self-reflection alone, while robustness tests show substantial model-dependent instability across GPQA-Diamond and three model families; strict conformity is 29% in the primary setting and remains predominantly harmful across model replications (57-77% correct-to-wrong). A controlled information-gradient experiment reveals that even vacuous reasoning is associated with 20-39% error adoption among resistant agents, with reasoning-like presentation carrying substantial persuasive weight. Harmful conformity can be predicted from Round 0 features (AUC = 0.79), and risk-targeted intervention reduces it by 13.6 percentage points (p < 0.001). However, without correctness labels or self-reflection controls, reducing peer adoption does not improve accuracy, because harmful and beneficial influence cannot be distinguished.

Abstract

Explore similar work

Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate

The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate