VeriDx: Earning the Right to Diagnose with Disease-Centric Verification

Zhong Cao; Shuying Chen

cs.AISep 12, 2026

VeriDx: Earning the Right to Diagnose with Disease-Centric Verification

Authors: Zhong Cao, Shuying Chen

Organizations: Heidelberg University · University of International Business and Economics

Abstract

A correct diagnosis can still be reached for the wrong reasons. In clinical reasoning, every disease hypothesis creates obligations: key evidence must be checked, alternatives must be ruled out, contradictions must be resolved, useful tests must be considered, and closure must be justified. Current evaluations of medical LLMs mostly focus on final answers, local steps, or isolated facts, and therefore miss these hypothesis-induced commitments. We introduce \textbf{VeriDx}, a disease-centric verification framework that links free-form diagnostic reasoning to structured disease profiles. VeriDx tracks whether each hypothesis is satisfied, unresolved, or violated its clinical obligations, exposing failures such as missing critical tests, unresolved differentials, ignored contradictions, unsupported claims, and premature closure. We instantiate VeriDx for complex respiratory diagnosis using guideline-derived disease profiles and expert-annotated longitudinal cases. Our results show that many diagnostic errors are not isolated mistakes, but broken commitments made earlier in the reasoning process.

Figures & tables

Figure 1: Disease-centric DDx profiles for standardized diagnostic verification. Conventional case-centric LLM reasoning can produce fluent and locally plausible clinical narratives while inconsistently covering disease-specific diagnostic obligations. VeriDx introduces disease-centric DDx profiles that encode required evidence, differential diagnoses, exclusion constraints, and treatment prerequisites for each disease. These profiles can be automatically updated from clinical guidelines and evidence, and are used to guide and audit LLM-generated reasoning. By mapping free-text diagnostic narratives onto profile-defined obligations, VeriDx enables consistent, auditable, and safety-aware verification across cases.

Figure 2: Overview of VeriDx. A longitudinal diagnostic trajectory is first generated under progressive evidence disclosure, including stage-wise differential diagnoses, examination plans, final diagnosis, diagnostic basis, and treatment plan. Each hypothesized disease is matched to a disease profile that encodes its clinical obligations, including phenotype consistency, supporting and opposing evidence, important mimics, critical examinations, missing information, and closure conditions. VeriDx checks these obligations for each hypothesis, labels them as satisfied, unresolved, or violated, and produces verification signals for detecting reasoning failures, guiding bounded revision, and evaluating trajectory-level clinical reasoning across D1–D6.

	LLM-only Reasoning			Gold $\mathcal{D}_{1}$ / $\mathcal{E}_{1}$			Gold $\mathcal{D}_{1}$ / $\mathcal{E}_{1}$ / $\mathcal{D}_{2}$ / $\mathcal{E}_{2}$
Model	Acc.	Recall	F1	Acc.	Recall	F1	Acc.	Recall	F1
AntAngelMed	0.20	0.38	0.28	0.27	0.45	0.34	0.71	0.74	0.72
Baichuan-M3	0.30	0.37	0.35	0.35	0.44	0.40	0.75	0.77	0.76
Claude-Opus-4.6	0.29	0.47	0.33	0.35	0.55	0.42	0.79	0.82	0.80
Claude-Opus-4.6 (+Image)	0.31	0.47	0.37	0.37	0.56	0.45	0.81	0.84	0.82
DeepSeek-v4-Pro	0.27	0.40	0.32	0.33	0.48	0.39	0.78	0.80	0.79

Table 1: Performance under progressively increasing levels of expert-annotated intermediate reasoning. LLM-only Reasoning uses model-generated intermediate states throughout the stepwise pipeline. Gold

\mathcal{D}_{1}

\mathcal{E}_{1}

replaces the first-stage differential diagnosis

\mathcal{D}_{1}

and examination plan

\mathcal{E}_{1}

with specialist annotations while preserving model-generated reasoning in subsequent stages. Gold

\mathcal{D}_{1}

\mathcal{E}_{1}

\mathcal{D}_{2}

\mathcal{E}_{2}

replaces all intermediate reasoning states with specialist annotations, representing the oracle upper-bound setting.

Figure 3: Task-level clinical reasoning performance of GPT-5.5 in the the multimodal setting across six evaluation dimensions and different verifier ablation settings. Bars indicate mean scores and error bars denote standard deviations across evaluation cases. DDx-only verification substantially improves dynamic reasoning and diagnostic precision, indicating that explicit differential-diagnosis correction mitigates premature diagnostic closure and promotes broader hypothesis exploration. In contrast, Exam-only verification primarily improves investigation planning, suggesting that structured examination completion enhances procedural completeness and diagnostic workup adequacy. The full VeriDx framework achieves the strongest overall performance across nearly all dimensions, demonstrating that differential-diagnosis verification and examination gap-filling provide complementary mechanisms for improving clinical reasoning safety, coherence, and diagnostic reliability.

Setting	D1	D2	D3	D4	D5	D6
w/o profiles	0.80	0.36	0.33	0.62	0.79	0.48
w/o phenotype	0.85	0.61	0.59	0.64	0.82	0.61
w/o DDx check	0.86	0.65	0.60	0.69	0.74	0.56
w/o closure	0.87	0.50	0.46	0.65	0.82	0.60
VeriDx (LLM)	0.88	0.68	0.64	0.71	0.84	0.66
VeriDx (Gold)	0.88	0.71	0.66	0.72	0.87	0.68

Table 2: Component ablation and profile-robustness results across six clinical reasoning dimensions for GPT-5.5 under the multimodal setting. The component-ablation settings use LLM-extracted DDx profiles. In the profile-robustness comparison, VeriDx (LLM) uses LLM-extracted DDx profiles, whereas VeriDx (Gold) uses specialist-constructed DDx profiles. All other experimental conditions are held fixed.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Condition	D1	D2	D3	D4	D5	D6
GPT-5.5 (+Image)	Baseline	$0.778\pm 0.053$	$0.297\pm 0.120$	$0.241\pm 0.082$	$0.601\pm 0.131$	$0.784\pm 0.045$	$0.613\pm 0.077$
	DDx-only	$0.879\pm 0.009$	$0.658\pm 0.114$	$0.341\pm 0.085$	$0.642\pm 0.131$	$0.822\pm 0.051$	$0.632\pm 0.073$
	Exam-only	$0.878\pm 0.049$	$0.423\pm 0.123$	$\mathbf{0.639\pm 0.087}$	$0.632\pm 0.136$	$0.830\pm 0.061$	$0.652\pm 0.086$
	Full VeriDx	$\mathbf{0.882\pm 0.018}$	$\mathbf{0.682\pm 0.116}$	$0.638\pm 0.089$	$\mathbf{0.711\pm 0.136}$	$\mathbf{0.837\pm 0.050}$	$\mathbf{0.661\pm 0.076}$
Gemini-3.1-Pro (+Image)	Baseline	$0.756\pm 0.059$	$0.354\pm 0.120$	$0.318\pm 0.085$	$0.594\pm 0.134$	$0.782\pm 0.049$	$0.618\pm 0.080$
	DDx-only	$0.876\pm 0.014$	$0.661\pm 0.115$	$0.395\pm 0.087$	$0.641\pm 0.131$	$0.818\pm 0.053$	$0.631\pm 0.074$

Appendix

Table 3: Cross-model verification and rewriting results across six clinical reasoning dimensions. All models are evaluated under the same multimodal setting using the identical VeriDx pipeline. Values denote mean scores and standard deviations across evaluation cases. Bold indicates the best result within each model family.

Explore similar work

VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs
Apr 25, 2026Yurui Xiang, Xingyi Mao, Rui Sheng +7Healthcare LLM Auditing
SymDiag: Explainable Diagnosis for LLM Reasoning via Neuro-Symbolic Verification
Aug 9, 2026Wenyao Cui, Huaping Zhang, Yongyi Huang +6CoT Faithfulness LLM Answer Verification
Beijing Institute of Technology Beijing, China · Zhongguancun Academy Beijing, China · Xinjiang Future Enterprise Incubator Co., Ltd. Xinjiang, China +1
A safety-oriented hypothetico-deductive framework for AI-assisted differential diagnosis
Jul 9, 2026Fan Ma, Mauro Giuffrè, Donald Wright +12Scientific Reasoning Medical Diagnosis
Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, USA · Department of Medical, Surgical and Health Sciences, Università degli Studi di Trieste, Trieste, Italy · Department of Emergency Medicine, Yale School of Medicine, Yale University, New Haven, USA +3

Apr 25, 2026, cs.CLJ/K move · Enter open · S save

VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs

Yurui Xiang, Xingyi Mao, Rui Sheng, Zixin Chen+6

Healthcare LLM Auditing Visual Analytics Medical LLMs LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) show promise in medical diagnosis, but real-world deployment remains challenging due to high-stakes clinical decisions and imperfect reasoning reliability. As a result, careful inspection of model behavior is essential for assessing whether diagnostic reasoning is reliable and clinically grounded. However, debugging medical LLMs remains difficult. First, developers often lack sufficient medical domain expertise to interpret model errors in clinically meaningful terms. Second, models can fail across a large and diverse set of instances involving different input types, tasks, and reasoning steps, making it challenging for developers to prioritize which errors deserve focused inspection. Third, developers struggle to identify recurring error patterns across cases, as existing debugging practices are largely instance-centric and rely on manual inspection of isolated failures. To address these challenges, we present VeriLLMed, a visual analytics system that integrates external biomedical knowledge to audit and debug medical LLM diagnostic reasoning. VeriLLMed transforms model outputs into comparable reasoning paths, constructs knowledge graph-grounded reference paths, and identifies three recurring classes of diagnosis errors: relation errors, branch errors, and missing errors. Case studies and expert evaluation demonstrate that VeriLLMed helps developers identify clinically implausible reasoning and generate actionable insights that can inform the improvement of medical LLMs.

Abstract

Explore similar work

VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs

SymDiag: Explainable Diagnosis for LLM Reasoning via Neuro-Symbolic Verification

A safety-oriented hypothetico-deductive framework for AI-assisted differential diagnosis