Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification

Masnun Nuha Chowdhury; Nusrat Jahan Beg; Umme Hunny Khan; Syed Rifat Raiyan; Md Kamrul Hasan; Hasan Mahmud

cs.CLMar 30, 2026

Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification

Authors: Masnun Nuha Chowdhury, Nusrat Jahan Beg, Umme Hunny Khan, Syed Rifat Raiyan, Md Kamrul Hasan, Hasan Mahmud

Organizations: Systems and Software Lab (SSL), Department of Computer Science and Engineering, Islamic University of Technology, Dhaka, Bangladesh

Abstract

Large language models (LLMs) remain unreliable for high-stakes claim verification due to hallucinations and shallow reasoning. While retrieval-augmented generation (RAG) and multi-agent debate (MAD) address this, they are limited by one-pass retrieval and unstructured debate dynamics. We propose a courtroom-style multi-agent framework, PROClaim, that reformulates verification as a structured, adversarial deliberation. Our approach integrates specialized roles (e.g., Plaintiff, Defense, Judge) with Progressive RAG (P-RAG) to dynamically expand and refine the evidence pool during the debate. Furthermore, we employ evidence negotiation, self-reflection, and heterogeneous multi-judge aggregation to enforce calibration, robustness, and diversity. In zero-shot evaluations on the Check-COVID benchmark, PROClaim achieves 81.7% accuracy, outperforming standard multi-agent debate by 10.0 percentage points, with P-RAG driving the primary performance gains (+7.5 pp). We ultimately demonstrate that structural deliberation and model heterogeneity effectively mitigate systematic biases, providing a robust foundation for reliable claim verification. Our code and data are publicly available at https://github.com/mnc13/PROClaim.

Figures & tables

Figure 1: Overview of the courtroom-style claim-verification pipeline in PROClaim .

Criterion	Threshold	Rationale
Novelty filter	$<0.20$	Rejects near-duplicates
Redundancy ratio	$>70\%$	Indicates saturation
Relevance gain	$<0.05$	Diminishing returns
Iteration cap	$10$	Limits compute cost

Table 1: P-RAG stopping criteria.

Panel of Table 2 — Table 2: Role-wise models. Providers (Prov.): OR =OpenRouter, OA =OpenAI. T.=Temperature.

Run	Acc	m-F1	$\bar{\kappa}$	Agr.	Unan.	Split
Run-0	0.950	0.950	0.429	0.617	0.442	0.558
Run-1	0.817	0.817	0.549	0.700	0.558	0.442
Run-2	0.790	0.790	0.474	0.652	0.496	0.504
W. Total	0.841	0.841	0.484	0.656	0.499	0.501
Maj. Vote	0.817	0.817	0.468	0.648	0.489	0.511
Oracle	0.958	0.958	0.438	0.622	0.450	0.550

Table 3: Performance statistics. m-F1 : MacroF1.

\bar{\kappa}

: Mean

\kappa

. Agr. : Raw Agreement. Unan. : Unanimity (3–0 consensus). Split : Divided vote. W. Total : Weighted Total. Oracle : Best-of-3. Per-judge

\kappa_{\text{GT}}

in App. E.1 .

Panel of Figure 2 — Figure 2: Termination distribution and convergence speed across 360 debate instances.

Panel of Figure 3 — Figure 3: P-RAG evidence novelty across debate rounds.

Figure 4: Reflection score trajectories across plateau, judicial, and critic resolution patterns.

System	Acc	MacroF1	Ev
Single-call DeepSeek-v3.2 + RAG	0.8000	0.7972	18.5
Standard MAD	0.7167	0.7068	12.1
PROClaim (Majority Vote)	0.8167	0.8165	67.5

Table 4: PROClaim vs. baselines on Check-COVID. Ev denotes average evidence pool size per claim.

Component	Full	A1	A2	A3	A4
Evidence Negotiation	✓	✓	✓	✓	✓
P-RAG	✓	✓	✓	$\mathbf{\times}$	✓
Expert Witnesses	✓	✓	✓	✓	✓
Self-Reflection	✓	✓	✓	✓	$\mathbf{\times}$
Reflection $\to$ P-RAG	✓	✓	✓	$\mathbf{\times}$	$\mathbf{\times}$
Critic Agent	✓	✓	✓	✓	✓

Table 5: Component ablation matrix. ✓ = active;

\mathbf{\times}

= disabled. Note: A1: no role-switch; A2: 1-judge; A3: no P-RAG; A4: no self-reflection.

System	Acc	MacroF1	$\boldsymbol{\Delta}$ Acc	$\bar{\kappa}$	Avg. Rounds	Ev	Tok (K)
PROClaim (Ours)	0.8167	0.8165	—	0.468	5.47	67.5	210.9
w/o P-RAG	0.7417	0.7408	$-$ 7.5 pp	0.599	6.00	37.5	188.9
w/o Role-Switching	0.7750	0.7750	$-$ 4.2 pp	0.513	2.88	54.0	147.3
w/o Three-Judge Panel	0.7833	0.7818	$-$ 3.3 pp	–	5.29	68.8	195.9
w/o Self-Reflection	0.8083	0.8080	$-$ 0.8 pp	0.591	7.06	81.5	247.3
Standard MAD	0.7167	0.7068	$-$ 10.0 pp	–	2.00	12.1	18.9

Table 6: Ablation results on 120 claims.

\Delta

Acc is measured relative to the full PROClaim pipeline. Ev denotes the average evidence pool size per claim. Each round count reflects the sum of primary and role-switched debate rounds per claim.

Dataset	Claim	Acc	F1	Tok(K)
HealthVer	100	0.720	0.713	223
FEVEROUS	60	0.783	0.772	236
Check-COVID	120	0.817	0.817	211

Table 7: Generalization results (single run)

System	Acc	$\boldsymbol{\Delta}$ (pp)	Odds Ratio
Single-call DeepSeek-v3.2 + RAG	63.0%	$+9.0$	2.50
Standard MAD	56.0%	$+16.0$	3.67 ‡
PROClaim	72.0%	—	—

Table 8: Paired comparison on the identical 100-claim HealthVer subset.

\Delta

is PROClaim ’s accuracy advantage; the odds ratio

\mathrm{OR}=b/c

counts claims PROClaim resolves correctly where the baseline fails (

b

) against the converse (

c

), so

\mathrm{OR}>1

means PROClaim wins the contested cases. ‡ Statistically significant (McNemar’s

\chi^{2}=8.04

p=0.0046

). Protocol details in Appendix E.2.2 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Model	Selection rationale
Plaintiff Counsel	GPT-5-mini	Strong instruction-following under advocacy constraints
Defense Counsel	DeepSeek-V3.2	Distinct pretraining lineage from Plaintiff
The Court	Qwen3-235B	Reliable query refinement and procedural control
Judge 1	DeepSeek-R1	Analytical rigor (deep CoT tracing)
Judge 2	Hermes-3-405B	Distinct reasoning style; diversity of adjudication
Judge 3	Qwen3-235B	Balanced abstention behaviour

Appendix

Table 9: Functional rationale for the heterogeneous role–model assignment. Counsels operate at

T{=}0.5

for argumentative diversity; judges at

T{=}0.3

for deterministic adjudication (validated in Section 5.5 ).

Hyperparameter	Value
Max debate rounds	10
Initial retrieval top- $k$	5
Per-round retrieval top- $k$	3
Novelty threshold	0.20
Admissibility floor	$>$ 0.5 admitted, $>$ 0.1 disputed
Redundancy similarity threshold	0.85

Appendix

Table 10: Debate and retrieval hyperparameters.

Mapping Policy	Accuracy	Macro F1
Inconclusive $\to$ Supported (ours)	81.7%	0.817
Inconclusive $\to$ Refuted	80.8%	0.808
Inconclusive excluded	83.8%	0.838

Appendix

Table 11: Verdict-mapping sensitivity on Check-COVID (120 majority-voted claims).

Variant	N	Accuracy	ECE
$W_{\text{consensus}}=0.6$	118	95.76%	0.1802
$W_{\text{consensus}}=0.8$ (Final)	118	95.76%	0.0340

Appendix

Table 12: Impact of Consensus Weighting on Calibration Error.

Confidence Bin	N	Observed Acc.	Mean Conf.
$[0.6,0.7)$	4	75.00%	0.6685
$[0.7,0.8)$	4	100.00%	0.7365
$[0.8,0.9)$	20	95.00%	0.8639
$[0.9,1.0]$	90	96.67%	0.9768

Appendix

Table 13: Bucket Analysis for

W_{\text{consensus}}=0.8

Run	$\kappa_{\text{GT}}$ (J1)	$\kappa_{\text{GT}}$ (J2)	$\kappa_{\text{GT}}$ (J3)	Mean $\kappa_{\text{GT}}$
	DeepSeek-R1	Hermes-3-405B	Qwen3-235B
Run-0	0.442	0.413	0.414	0.423
Run-1	0.402	0.367	0.347	0.372
Run-2	0.452	0.353	0.348	0.384
Majority Voting	0.452	0.431	0.468	0.450

Appendix

Table 14: Per-judge ground-truth agreement (

\kappa_{\text{GT}}

) across runs and aggregation modes. Mean

\kappa_{\text{GT}}

is the unweighted average across the three judges.

Agent	Logic	Novelty	Rebuttal	Total	Discovery Need (summary)
Plaintiff	0.78	0.45	0.62	0.633	Controlled studies comparing myocarditis rates in COVID-19 vs. non-COVID respiratory illness
Defense	0.70	0.50	0.40	0.550	Studies demonstrating SARS-CoV-2 viral presence in cardiomyocytes

Appendix

Table 15: Self-Reflection Scores after Phase 1.

Agent	Logic	Novelty	Rebuttal	Total
Plaintiff	0.85	0.60	0.70	0.730
Defense	0.70	0.30	0.60	0.550

Appendix

Table 20

Panel of Table 16 — Table 16: Judicial panel evaluation for the example debate.

System	Avg Tokens/Claim (K)	120 Claims (M)	1K Claims (M)	10K Claims (B)	Acc.
Standard MAD	18.9	2.3	18.9	0.19	71.7%
PROClaim w/o Self-Refl.	247.3	29.7	247.3	2.47	80.8%
PROClaim w/o Role-Switch	147.3	17.7	147.3	1.47	77.5%
PROClaim w/o P-RAG	188.9	22.7	188.9	1.89	74.2%
PROClaim (Full)	210.9	25.3	210.9	2.11	81.7%

Appendix

Table 17: Token usage per claim across system configurations and projected totals at scale. Projections assume linear scaling with no batching discount.

Panel of Figure 5 — Figure 5: Cost–accuracy Pareto front across system configurations. PROClaim (Full) lies on the efficient frontier; the dashed line traces Pareto-optimal points. w/o P-RAG is strictly dominated: it saves fewer tokens than w/o Role-Switch while incurring a larger accuracy penalty.

Claim Verification

Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.

See all 59 papers

A Multi-Agent Approach for Claim Verification from Tabular Data Documents
Apr 19, 2026Rudra Ranajee Saha, Laks V. S. Lakshmanan, Raymond T. NgClaim Verification Multi-Agent LLMs
University of British Columbia
Contestable Multi-Agent Debate with Arena-based Argumentative Computation for Multimedia Verification
May 14, 2026Truong Thanh Hung Nguyen, Vo Thanh Khang Nguyen, Hoang-Loc Cao +3Multi-Agent Debate Computational Argumentation
University of New Brunswick Fredericton, New Brunswick, Canada · FPT Software Quy Nhon, Vietnam · University of Science, VNU-HCM Ho Chi Minh City, Vietnam
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Oct 1, 2026Caiqi Zhang, Rujun Han, Zifeng Wang +4LLM Agent Verification Long-Horizon LLM Agents
University of Cambridge · Google Cloud AI Research

Apr 19, 2026J/K move · Enter open · S save

A Multi-Agent Approach for Claim Verification from Tabular Data Documents

Rudra Ranajee Saha, Laks V. S. Lakshmanan, Raymond T. Ng

University of British Columbia

Claim Verification Multi-Agent LLMs

Open paper AskPDF arXiv

We present a novel approach for claim verification from tabular data documents. Recent LLM-based approaches either employ complex pretraining/fine-tuning or decompose verification into subtasks, often lacking comprehensive explanations and generalizability. To address these limitations, we propose a Multi-Agentic framework for Claim verification (MACE) consisting of three specialized agents: Planner, Executor, and Verifier. Instead of elaborate finetuning, each agent employs a zero-shot Chain-of-Thought setup to perform its tasks. MACE produces interpretable verification traces, with the Planner generating explicit reasoning strategies, the Executor providing detailed computation steps, and the Verifier validating the logic. Experiments demonstrate that MACE achieves state-of-the-art (SOTA) performance on two datasets and performs on par with the best models on two others, while achieving 80--100% of best performance with substantially smaller models: 27--92B parameters versus 235B. This combination of competitive performance, memory efficiency, and transparent reasoning highlights our framework's effectiveness.

Abstract

Claim Verification

Explore similar work

A Multi-Agent Approach for Claim Verification from Tabular Data Documents

Contestable Multi-Agent Debate with Arena-based Argumentative Computation for Multimedia Verification

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks