Multi-Agent Reasoning with Consistency Verification Improves Uncertainty Calibration in Medical MCQA

John Ray B. Martinez

cs.AIMar 25, 2026

Multi-Agent Reasoning with Consistency Verification Improves Uncertainty Calibration in Medical MCQA

Authors: John Ray B. Martinez

Organizations: Department of Data Science and Analytics, Harrisburg University of Science and Technology

Abstract

Miscalibrated confidence scores are a practical obstacle to deploying AI in clinical settings. A model that is always overconfident offers no useful signal for deferral. We present a multi-agent framework that combines domain-specific specialist agents with Two-Phase Verification (Wu et al., 2024) and S-Score Weighted Fusion to improve both calibration and discrimination in medical multiple-choice question answering. Four specialist agents (respiratory, cardiology, neurology, gastroenterology) generate independent diagnoses using Qwen2.5-7B-Instruct. Each diagnosis undergoes a two-phase self-verification process that measures internal consistency and produces a Specialist Confidence Score (S-score). The S-scores drive a weighted fusion strategy that selects the final answer and calibrates the reported confidence. We evaluate on high-disagreement subsets of MedQA-USMLE and MedMCQA (100 and 250 questions). All results are specific to this filtered regime. On MedQA-250, the full system achieves ECE = 0.091 (74.4% reduction over the single-specialist baseline) and AUROC = 0.630 (+0.056) at 59.2% accuracy. Calibration gains of 49-74% hold across all four settings. Ablation analysis reveals that Two-Phase Verification drives ECE reduction while multi-agent reasoning drives AUROC improvement, suggesting that consistency checking and ensemble aggregation address different failure modes of LLM uncertainty. Whether the resulting confidence signal is sufficient to support clinical deferral decisions in practice remains a direction for future investigation.

Figures & tables

Panel of Figure 1 — Figure 1: The MARC framework pipeline. A medical question is distributed to four domain-specific specialist agents (Component 1), each producing an answer and reasoning chain. Two-Phase Consistency Verification (Component 2; Wu et al. 2024 ) extracts factual claims, answers them independently and with the original reasoning in context, and measures cross-condition inconsistency to derive an S-score $S_{k}$ per specialist. S-Score Weighted Fusion (Component 3) selects the final answer by vote-count-weighted mean S-score and derives calibrated confidence $\hat{C}$ from the vote fraction and S-score distribution.

Dataset	Configuration	Acc	$\boldsymbol{\Delta}$ Acc	ECE	$\boldsymbol{\Delta}$ ECE	AUROC	$\boldsymbol{\Delta}$ AUROC	Avg Conf	Time (min)
MedQA-100	C1: Single Specialist	52.0%	—	0.374	—	0.537	—	0.886	39
	Avg. 4 Specialists	54.2%	—	0.360	—	0.552	—	—	—
	C2: Single + Two-Phase	52.0%	+0.0	0.185	$-$ 51%	0.678	$+$ 0.141	0.430	67
	C3: Multi + S-Score (No 2P)	55.0%	+3.0pp	0.356	$-$ 5%	0.578	$+$ 0.041	0.896	152
	C4: Full System	59.0%	+7.0pp	0.098	$-$ 73%	0.645	$+$ 0.108	0.549	249
MedQA-250	C1: Single Specialist	54.4%	—	0.355	—	0.574	—	0.899	116

Table 1: Performance comparison across all four datasets and configurations. Best value in each column group is bold .

\Delta

columns show change relative to Config 1 baseline. The italic Avg. 4 Specialists row reports the mean performance across all four individual specialists running independently (see Table 4 for per-specialist details). Avg Conf = mean predicted confidence score. Time = wall-clock inference time in minutes on a single NVIDIA RTX 5090 GPU.

Figure 2: Accuracy (top row), ECE (middle row), and AUROC (bottom row) for each of the four datasets and all four configurations. Green-outlined bars indicate the best value in each panel. Config 4 (Full System, dark blue) achieves the best ECE across all four datasets, and the best accuracy on the MedQA subsets. On MedMCQA, Config 3 achieves the best accuracy at the cost of calibration, illustrating the knowledge-recall challenge for verification-based methods.

Figure 3: Reliability diagrams for all four datasets. Each panel shows calibration curves for all four configurations against the perfect-calibration diagonal. Across all datasets, Config 4 (solid blue squares) lies closest to the diagonal, confirming that Two-Phase Verification is the dominant calibration driver regardless of question type or dataset size.

Figure 4: ROC curves for all four datasets. Each panel plots the four configuration curves with the random baseline (diagonal). Config 4 achieves the highest AUROC on MedQA-250 and MedMCQA-250. On MedQA-100, Config 2 (single specialist with verification) achieves the highest AUROC, and on MedMCQA-100, Config 3 (multi-agent without verification) is best, illustrating that discrimination benefits from multi-agent fusion scale more reliably with dataset size and question type.

Figure 5: Calibration analysis of Qwen2.5-7B across all four evaluation sets. Columns represent configurations C1–C4; each dataset occupies two sub-rows. Top sub-row : stacked confidence frequency histogram (blue = correct answer, red = wrong answer). Bottom sub-row : calibration histogram with 5%-wide bins for visual resolution, where bar height represents observed accuracy within that bin; the dashed diagonal represents perfect calibration. Config 4 (Full System) consistently produces the most evenly spread confidence distributions and aligns most closely with the perfect-calibration diagonal across all datasets.

Component Added	$\boldsymbol{\Delta}$ Acc	$\boldsymbol{\Delta}$ ECE	$\boldsymbol{\Delta}$ AUROC
Two-Phase Verif. (C1 $\to$ C2)	0.0 pp	$-$ 0.177 ( $-$ 49.9%)	$-$ 0.018
Multi-Agent (C1 $\to$ C3)	+2.8 pp	$-$ 0.019 ( $-$ 5.4%)	+0.013
Two-Phase added to Multi (C3 $\to$ C4)	+2.0 pp	$-$ 0.245 ( $-$ 72.9%)	+0.043
Full System (C1 $\to$ C4)	+4.8 pp	$-$ 0.264 ( $-$ 74.4%)	+0.056

Table 2: Ablation summary on MedQA-250 (

n=250

). Each row reports the marginal effect of adding one component.

Figure 6: (A) ECE reduction percentage and (B) AUROC change for each component transition across all four datasets. Adding the full system (C1 $\to$ C4, dark blue) consistently delivers the largest ECE reduction (49–74%) across all settings. Multi-agent reasoning (C1 $\to$ C3, orange) drives the largest AUROC gains on MedQA subsets, while verification components dominate calibration improvement throughout.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Config	Accuracy 95% CI	ECE 95% CI	AUROC 95% CI
MedQA-100	C1: Single Specialist	[42.0, 62.0]	[0.269, 0.463]	[0.470, 0.605]
	C2: Single + Two-Phase	[42.0, 62.0]	[0.103, 0.272]	[0.572, 0.781]
	C3: Multi + S-Score (No 2P)	[45.0, 65.0]	[0.244, 0.438]	[0.502, 0.652]
	C4: Full System	[49.0, 69.0]	[0.050, 0.194]	[0.531, 0.755]
MedQA-250	C1: Single Specialist	[48.4, 60.4]	[0.291, 0.412]	[0.533, 0.616]
	C2: Single + Two-Phase	[48.4, 60.4]	[0.133, 0.245]	[0.486, 0.626]

Appendix

Table 3: Bootstrap confidence intervals (95%,

n{=}10{,}000

resamples) for all metrics across all four datasets and configurations (C1–C4).

	MedQA-100			MedQA-250			MedMCQA-100			MedMCQA-250
Specialist	Acc.	ECE	AUROC	Acc.	ECE	AUROC	Acc.	ECE	AUROC	Acc.	ECE	AUROC
Respiratory	52.0%	0.374	0.537	54.4%	0.355	0.574	45.0%	0.469	0.501	42.8%	0.469	0.536
Cardiology	57.0%	0.343	0.541	57.6%	0.322	0.537	47.0%	0.451	0.557	42.8%	0.469	0.562
Neurology	55.0%	0.343	0.587	58.4%	0.309	0.555	48.0%	0.418	0.514	44.8%	0.448	0.543
Gastroenterology	53.0%	0.380	0.543	58.4%	0.315	0.552	51.0%	0.368	0.589	48.4%	0.399	0.570
Average	54.2%	0.360	0.552	57.2%	0.325	0.555	47.8%	0.426	0.540	44.7%	0.446	0.553

Appendix

Table 4: Per-specialist performance (Accuracy, ECE, AUROC) across all four evaluation sets. Each specialist runs independently with no verification and no fusion (equivalent to Config 1 setup). The Average row is the mean across all four specialists and serves as an alternative single-specialist baseline. Respiratory corresponds to the Config 1 baseline in Table 1 .

Uncertainty Calibration

Uncertainty calibration aligns model confidence and predictive intervals with observed correctness and outcome frequencies.

See all 264 papers

Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA
Jun 25, 2026Eren Senoglu, Federico Toschi, Nicolo Brunello +2Medical VQA Language Model Calibration
Politecnico di Milano
Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
Jun 18, 2026Yuetian Du, Yucheng Wang, Ming Kong +4Multimodal QA Language Model Calibration
College of Computer Science and Technology, Zhejiang University · School of Computer Science and Technology, Xidian University · Zhihui Medical Technology (Shanghai) Co., Ltd., Shanghai, China +1
The Consistency Illusion: How Multi-Agent Debate Hides Reasoning Misalignment
Jun 7, 2026Xiaoyang Wang, Christopher C. YangLLM Agent Evaluation Multi-Agent Debate
1Drexel University

Jun 25, 2026J/K move · Enter open · S save

Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA

Eren Senoglu, Federico Toschi, Nicolo Brunello, Andrea Sassella+1

Politecnico di Milano

Medical VQA Language Model Calibration Medical VLMs VLM Calibration Uncertainty Calibration

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) applied to Medical Visual Question Answering (VQA) tend to produce overconfident outputs regardless of actual correctness, and existing verbalized confidence calibration methods, developed primarily for text only LLMs, do not account for the multimodal nature of medical image understanding. This work proposes a training based framework that finetunes MLLMs to improve their calibration using a composite loss function combining a Brier style calibration term, an anchor regularizer that prevents confidence collapse toward extreme values, a contrastive image text alignment term, and a KL based model stabilization term. The alignment signal is derived from a

2 \times 2

factorial perturbation design that crosses image presence with text integrity, probing the reliance of the model on visual modality input versus language priors. Finally, a top K KL divergence regularizer is used to protect the answering ability of the model during finetuning. Across three Medical VQA benchmarks and two architectures (MedGemma 4B IT and Qwen2 VL 7B Instruct), our method reduces calibration error by 60% or more, and improves discrimination by 26% or more, while preserving predictive accuracy. On average across benchmarks, the technique outperforms prompting based, sampling based, and training based approaches, and ablation experiments confirm that each component of the loss function is indeed necessary for improving the calibration. All code for the experiments is publicly available.

Abstract

Uncertainty Calibration

Explore similar work

Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA

Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA

The Consistency Illusion: How Multi-Agent Debate Hides Reasoning Misalignment