The impact of multi-agent debate protocols on debate quality: a controlled case study

Ramtin Zargari Marandi

cs.MAMar 28, 2026

The impact of multi-agent debate protocols on debate quality: a controlled case study

Authors: Ramtin Zargari Marandi

Abstract

In multi-agent debate (MAD) systems, performance gains are often reported; however, because the debate protocol (e.g., number of agents, rounds, and aggregation rule) is typically held fixed while model-related factors vary, it is difficult to disentangle protocol effects from model effects. To isolate these effects, we compare three main protocols, Within-Round (WR; agents see only current-round contributions), Cross-Round (CR; full prior-round context), and novel Rank-Adaptive Cross-Round (RA-CR; dynamically reorders agents and silences one per round via an external judge model), against a No-Interaction baseline (NI; independent responses without peer visibility). In a controlled macroeconomic case study (20 diverse events, five random seeds, matched prompts/decoding), RA-CR achieves faster convergence than CR, WR shows higher peer-referencing, and NI maximizes Argument Diversity (unaffected across the main protocols). These results reveal a trade-off between interaction (peer-referencing rate) and convergence (consensus formation), confirming protocol design matters. When consensus is prioritized, RA-CR outperforms the others.

Figures & tables

Figure 1: Controlled workflow for dataset construction, matched protocol execution, metric computation, and paired inference. Protocol is the manipulated factor, while prompts, decoding settings, model assignment, and judge-guided candidate selection are held fixed across conditions.

Panel of Figure 2 — Figure 2: Protocol comparison under a shared two-round timeline for WR, CR, RA-CR, and NI.

Metric	Operational range	Better direction	Interpretation
PRR	$[0,1]$	Higher	More explicit response to peers
AD	$[0,1]$	Context-dependent	More lexical variation across responses
CF	$[0,1]$	Higher	More agreement by the final round

Table 1: Primary metric set used in the main analysis.

Figure 3: Aggregate comparison of the three primary debate protocols plus the No-Interaction baseline from the SBERT top-20, five-seed aggregate (100 matched units per condition): (a) PRR, (b) AD, and (c) CF. Bars show condition means with 95% bootstrap confidence intervals. NI is omitted from panel (a) because PRR is structurally zero under no peer visibility. The figure highlights the central trade-off of the study: WR is stronger on interaction-oriented metrics, whereas RA-CR is stronger on convergence-oriented metrics. Significance markers: $*$ $p<0.05$ , $**$ $p<0.01$ , $***$ $p<0.001$ after Holm–Bonferroni adjustment.

Comparison	Metric	$\Delta$	$p_{\mathrm{HB}}$
WR vs RA-CR	PRR	$+0.064$	$0.0056$
WR vs RA-CR	AD	$+0.024$	$0.1045$
WR vs RA-CR	CF	$-0.212$	$0.0064$
WR vs CR	PRR	$+0.038$	$0.0691$
WR vs CR	AD	$+0.023$	$0.1596$
WR vs CR	CF	$+0.075$	$0.7989$

Table 2: Pairwise comparisons for the primary three-metric set after Holm–Bonferroni adjustment.

\Delta

is the paired mean difference (first minus second). NI contrasts for PRR are omitted because PRR is structurally zero under no peer visibility.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Metric	Mean	95% CI
Within-Round	Peer Reference Rate	0.320	[0.298, 0.343]
Cross-Round	Peer Reference Rate	0.282	[0.263, 0.298]
Rank-Adaptive Cross-Round	Peer Reference Rate	0.256	[0.230, 0.280]
Within-Round	Argument Diversity	0.663	[0.646, 0.679]
Cross-Round	Argument Diversity	0.640	[0.625, 0.656]
Rank-Adaptive Cross-Round	Argument Diversity	0.639	[0.622, 0.656]

Appendix

Table 1: Supplementary Table 1. Condition means with bootstrap

95\%

confidence intervals from the SBERT top-20, five-seed main aggregate (

n=100

per condition).

Date	Inflation (%)	Major world event	Inflation relation confirmed?
2016-02	2.54	The World Health Organization declares the Zika virus outbreak a Public Health Emergency of International Concern.	No confirmed correlation with US sticky price movements.
2016-05	2.57	A massive wildfire in Fort McMurray, Alberta, forces the largest evacuation in Canadian history and halts oil sands production.	Minimal; energy is excluded from this index, and the production halt was temporary.

Appendix

Table 2: Supplementary Table 2. Example rows from the event-annotated dataset.

Event	Relevant comment	Irrelevant comment
Fed raises policy rate by 75 bps amid persistent core inflation	The rate hike should cool demand-sensitive sectors over 6–12 months, but sticky services inflation may remain elevated due to wage persistence.	I like hiking and the weather is nice this week. This event reminds me of my vacation photos.
Major shipping disruptions in the Red Sea increase global freight costs	Higher freight costs can pass through to goods prices and import channels, adding near-term upside pressure to US sticky core inflation.	My favorite movie soundtrack is underrated and has great violin solos.

Appendix

Table 3: Supplementary Table 3. Representative judge-validation examples used for the pre-run sanity check.

Multi-Agent Debate

Multi-Agent Debate studies how language-model agents exchange arguments, critique answers, and combine judgments for collective reasoning.

See all 145 papers

Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges
Jul 28, 2026Quim Motger, Marc Oriol, Jordi Marco +1Multi-Agent Debate Multi-Agent Consensus
Dpt. of Service and Information System Engineering, Universitat Politècnica de Catalunya, Barcelona, Spain · Dpt. of Computer Science, Universitat Politècnica de Catalunya, Barcelona, Spain
When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?
Oct 23, 2025Yongqiang Chen, Gang Niu, James Cheng +2Multi-Agent Debate Multi-Agent Collaboration
1The Chinese University of Hong Kong · 2RIKEN Center for Advanced Intelligence Project · 3Hong Kong Baptist University +1
Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate
Aug 4, 2026Hao Wu, Shoucheng Song, Chang Yao +3Multi-Agent Debate RL for LLM Agents
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing, China

Jul 28, 2026J/K move · Enter open · S save

Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

Quim Motger, Marc Oriol, Jordi Marco, Xavier Franch

Dpt. of Service and Information System Engineering, Universitat Politècnica de Catalunya, Barcelona, Spain · Dpt. of Computer Science, Universitat Politècnica de Catalunya, Barcelona, Spain

Multi-Agent Debate Multi-Agent Consensus Multi-Agent Communication

Open paper AskPDF arXiv

Multi-Agent Debate (MAD) is a promising paradigm for improving the accuracy and robustness of Large Language Model (LLM)-based agentic systems. It enables multiple agents to exchange arguments, critique each other's outputs, and iteratively converge towards a solution. However, research remains fragmented, with inconsistent terminology and no rigorous synthesis of MAD design dimensions. We present a systematic literature review characterizing 141 primary studies on MAD. We derive a three-dimensional taxonomy covering debate participants, the interaction mechanisms structuring the exchange, and the agreement protocols governing debate resolution, supported by formal notations to render MAD configurations. Our analysis reveals that the field has implicitly converged on a narrow design pattern - static, fully connected topologies, verbatim exchange, short-term memory and voting resolution strategies - adopted by convention rather than systematic comparison, while promising alternatives remain marginal. Because any MAD setting reflects roughly a dozen interacting design decisions, cross-study comparison is unreliable when these are left implicit. We position the taxonomy as a descriptive map of the research landscape, a framework for controlled benchmarking, and potentially as a schema for machine-readable MAD specifications. As future work, we propose formalizing it into an executable specification, enabling cost-aware benchmarking and automated tuning of debate configurations.

Abstract

Multi-Agent Debate

Explore similar work

Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?

Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate