MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis

Junda Wang; Zonghai Yao; Yujan Ting; Eric Z. Chen; Hieu Tran; Hong Yu; Weijing Huang; Terrence Chen

cs.CLOct 19, 2024

MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis

Authors: Junda Wang, Zonghai Yao, Yujan Ting, Eric Z. Chen, Hieu Tran, Hong Yu, Weijing Huang, Terrence Chen

Organizations: United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA · Miner School of Computer and Information Sciences, University of Massachusetts Lowell, MA, USA

Abstract

Medical multimodal large language models (MLLMs) can perform well on existing medical visual question answering (MedVQA) benchmarks, but their training data often does not match clinical diagnosis. Most supervision is organized around isolated images or short QA pairs, leaving two structures weakly specified: how evidence leads to a decision, and how views, series, modalities, and patient context from the same case are linked. We introduce MultiViewDx, a partly physician-validated multimodal instruction dataset for evidence-linked multi-view medical imaging diagnosis. MultiViewDx uses the clinical case as the supervision unit. It links imaging studies with patient context, normalizes heterogeneous reports into an evidence-linked workflow (evidence -> findings -> differential discussion -> diagnosis), and uses a unified image-text retriever to constrain instruction synthesis to source-supported evidence. It covers X-ray, CT, MRI, ultrasound, histopathology, and other clinical visual sources. We fine-tune MultiViewDx-8B-AN and evaluate it on both existing MedVQA benchmarks and real-world case-based diagnostic reasoning. Across four MedVQA benchmarks, it achieves the best average accuracy among compared systems (79.0%), outperforming HuatuoGPT-Vision-34B (66.7%) and Claude3-Opus (55.7%). Beyond MedVQA, on JAMA Clinical Challenge cases, it receives the strongest overall rating under a physician-designed rubric for key clinical points, diagnostic inference, and evidence grounding. Controlled ablations and clinician evaluation show that both case-level multi-view organization and evidence-linked reasoning targets contribute to the gain.

Figures & tables

Figure 1: From image-level QA to case-level diagnosis. Left: common medical VQA or caption data use one displayed image and a short target answer. Right: MultiViewDx uses the clinical case as the training unit, linking patient context, multiple views or series, localized evidence, intermediate findings, differential discussion, and the final diagnosis or next step. The key change is the supervision unit: from an isolated image-answer pair to an evidence-linked diagnostic case.

Figure 2: Modality coverage of MultiViewDx. MultiViewDx covers a broad range of medical imaging modalities and clinical visual sources, including X-ray, CT, MRI, histopathology, ultrasound, endoscopy, clinical photography, fundus/optic imaging, and electrophysiology. The central distribution summarizes the major modality composition of the dataset, with CT, MRI, histopathology, and X-ray forming the largest proportions. This diverse modality coverage supports training on multi-view and multi-modal clinical evidence rather than a single image type.

Dataset	N	Mod	ROI	HA	Slice	Cap	Lic (short)
DeepLesion	24,821	CT	$\times$	$\times$	$\times$	Y	CC-BY4
PadChest	150,730	XR	$\times$	$\checkmark$	–	Y	AGR-PadChest
Eurorad	691,370	Multi	$\checkmark$	$\checkmark$	$\checkmark$	Y	CC-BY4
MIMIC-CXR-JPG	620,113	XR	$\times$	$\checkmark$	–	N	PN-CHDL1.5
LLD	30,390	MR	$\checkmark$	$\times$	$\checkmark$	Y	AGR-LLD
MAMA-MIA	76,381	MR	$\checkmark$	$\times$	$\checkmark$	Y	CC-BY-NC-SA4

Table 1: Unified data sources with annotation, caption availability, and license. N denotes processed sample records. For fragmented sources, the retained metadata support sample-level counts only; unique patient-, study-, and case-level counts are unavailable. Mod: CT=CT, XR=X-ray, MR=MRI, Path=pathology, Histo=histopathology, Multi=multi-modality, Mix=multi + text. ROI: ROI annotation. HA: human annotation. Slice: per-slice identifier. Cap: caption availability (Y/N). Lic (short): license type (CC-BY4, CC-BY-SA, CC-BY-NC4, CC-BY-NC-SA4, PN-CHDL1.5, OpenRAIL, AGR-* = dataset-specific agreement). “–” means not applicable or not provided. ∗ PMC-VQA and the Stage 1 corpus share PMC/PMC-OA source provenance and may contain overlapping articles, images, or captions.

Figure 3: MultiViewDx data construction and training pipeline. The pipeline starts from clinical cases that contain multi-view evidence, image findings, diagnosis, and treatment-related information. These cases are organized into a multimodal medical library that combines large-scale image-caption pretraining data with a clinical case library. For incomplete sources, retrieval and GPT-4o synthesis are used to complete missing clinical history, image findings, diagnosis, and instruction QA. The model is then trained in three stages: pretraining on PMC image-text data, fine-tuning on GPT-4o-synthesized instruction data, and sequential fine-tuning on human-annotated clinical cases. The resulting model is evaluated on both real-world medical tasks and traditional medical VQA benchmarks.

Proprietary Models
Model	VQA-RAD	SLAKE	PathVQA	PMC-VQA ∗	Avg.
GPT-4.1	65.0	72.2	55.5	55.2	62.0
GPT-4o	61.0	71.2	55.5	49.7	59.3
Claude Sonnet 4	67.6	70.6	54.2	54.4	61.7
Gemini-2.5-Flash	68.5	75.8	55.4	55.4	63.8
GPT-4o-mini	45.9	59.0	37.9	33.3	44.0

Table 2: Main comparison on official held-out medical VQA test splits. MultiViewDx-8B-AN uses the four official training partitions. Additional MultiViewDx variants appear in Table 5 , and Appendix Table 12 reports exposure for every compared model. ∗ PMC-VQA and Stage 1 share PMC/PMC-OA source provenance.

Model	VQA-RAD	SLAKE	PathVQA	PMC-VQA ∗	Avg.
LLAVA-v1.5-LLAMA3-8B	63.3	68.9	85.2	50.3	66.9
LLAVA_Med-8B	66.3	69.5	90.7	52.7	69.8
HuatuoGPT-Vision-8B	68.9	84.1	93.0	57.3	75.8
MultiViewDx	88.3	91.1	92.7	88.6	90.2

Table 3: Benchmark-specific supervised adaptation results. Every row is fine-tuned on the corresponding official training partition and evaluated on the held-out test split. Appendix Table 13 summarizes the setting. ∗ PMC-VQA shares the source provenance noted in Table 2 .

Model	Acc.	UMLS	KP	Inf.	Evi.	Overall
C3-Opus	58.4	0.18	1.27	1.56	0.67	1.17 $\pm$ 0.04
G4o-mini	46.2	0.16	0.99	1.13	0.60	0.91 $\pm$ 0.06
H-7B	34.5	0.13	1.11	1.06	1.08	1.08 $\pm$ 0.03
H-34B	44.7	0.16	1.01	1.06	1.31	1.13 $\pm$ 0.05
MG-4B	45.8	0.19	1.06	1.10	1.21	1.14 $\pm$ 0.03
LS-7B	53.6	0.19	1.10	1.19	1.27	1.18 $\pm$ 0.04

Table 4: Results on the JAMA Clinical Challenge. Acc.=close-ended accuracy; UMLS=UMLS-F1 factuality; KP=Key Points; Inf.=Inference; Evi.=Evidence. Model abbreviations: C3-Opus=Claude3-Opus, G4o-mini=GPT-4o-mini, H-7B/H-34B=Huatuo-7B/34B, MG-4B=Med-Gemma-4B-IT, LS-7B=Lingshu-7B, HM-7B=Hulu-Med-7B, and MVDX-AN=MultiViewDx-AN. We additionally rescore a subset with DeepSeek in Appendix Table 7 ; the relative ranking is consistent.

Variant	VQA-RAD	SLAKE	PathVQA	PMC-VQA ∗	Avg.
MultiViewDx-8B-20M	67.8	76.1	57.8	53.6	63.8
MultiViewDx-8B-Synthesis	69.2	77.2	63.6	58.4	67.1
MultiViewDx-8B-Mix	74.2	81.3	76.3	59.1	72.2
MultiViewDx-8B-AN	86.1	87.7	80.4	61.9	79.0

Table 5: Ablation of MultiViewDx training variants on standard medical VQA benchmarks. MultiViewDx-8B-20M uses all available 3D slices during training; Synthesis uses GPT-4o-synthesized instruction data with slice-capped construction; Mix combines synthesized and human-annotated cases in one SFT stage; AN first trains on synthesized data and then sequentially fine-tunes on human-annotated cases. ∗ PMC-VQA shares the source provenance noted in Table 2 .

Model	Syn.	MV	Reason.	HA	Doc.
C3-Opus	-	-	-	-	0.48
H-7B	✓	$\times$	$\times$	$\times$	0.21
H-34B	✓	$\times$	$\times$	$\times$	0.35
MG-4B	✓	$\times$	$\times$	$\times$	0.40
LS-7B	✓	$\times$	✓	$\times$	0.44
HM-7B	✓	$\times$	✓	$\times$	0.50

Table 6: Component-level clinician evaluation on 100 held-out JAMA cases. Doctor Score (Doc.) is the averaged and normalized holistic rating from three clinicians, who consider Key Points, Inference, and Evidence with reference to the case and gold rationale. Syn.=synthetic or teacher-distilled instruction data; MV=case-level multi-view grouping; Reason.=evidence-linked diagnostic reasoning targets; HA=human-annotated clinical cases. Baseline checkmarks denote analogous training signals reported in the original papers.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Acc.	UMLS-F	GPT4-O	DS-O
Claude3-Opus	58.4	0.18	1.17	2.31
GPT-4o-mini	46.2	0.16	0.91	1.95
Huatuo-7B	34.5	0.13	1.08	2.06
Huatuo-34B	44.7	0.16	1.13	2.24
MultiViewDx	41.2	0.11	0.78	1.86
MultiViewDx-AN	58.5	0.23	1.29	2.55

Appendix

Table 7: Auxiliary automatic-evaluator sensitivity analysis. Acc.=Accuracy; UMLS-F=UMLS-based factuality; GPT4-O=GPT-4o Overall; DS-O=DeepSeek Overall. MultiViewDx-AN ranks highest under both automatic evaluators among the displayed rows; DS-O is interpreted on the DeepSeek scale.

	Count
Oto-HNS	513
Ophth	466
Derm	368
JAMA	328
JN Learn	299
Surg	133

Appendix

Table 8: Distribution across JAMA specialty journals. Abbr.: Oto-HNS=JAMA Otolaryngology–Head & Neck Surgery, Ophth=JAMA Ophthalmology, Derm=JAMA Dermatology, Surg=JAMA Surgery, Oncol=JAMA Oncology, Cardiol=JAMA Cardiology, Neurol=JAMA Neurology, Pediatr=JAMA Pediatrics, Psych=JAMA Psychiatry, JN Learn=JN Learning.

Figure 4: This figure illustrates the proportion of questions assessing knowledge and inference in the Slake, VQA-RAD, Path-VQA, and JAMA Clinical Challenge datasets.

Stage	Data	Samples	LR	Updated Params.	GPU Setting	Approx. Compute
Stage 1: Medical pretraining	Filtered PMC image-text data	14M	$1\mathrm{e}{-4}$	Adapter only	4 H100 GPUs	$\sim$ 1,600 H100 hours
Stage 2: Instruction tuning	GPT-4o-synthesized MultiViewDx data	4M	$2\mathrm{e}{-5}$	Full model	8 H100 GPUs	$\sim$ 640 H100 hours
Stage 3: Sequential fine-tuning	Human-annotated clinical cases	1M	$1\mathrm{e}{-5}$	Full model	8 H100 GPUs	$\sim$ 160 H100 hours
Total	–	–	–	–	–	$\sim$ 2,400 H100 hours

Appendix

Table 9: Training configuration and approximate compute cost for MultiViewDx-8B. Stage 1 provides broad medical image-text grounding; Stage 2 adds instruction following and evidence-linked QA from GPT-4o-synthesized data; Stage 3 sequentially fine-tunes on human-annotated clinical cases. The overall estimated compute is on the order of

0.81

–

1.0\times 10^{21}

FLOPs.

Model	Simulated generated output on the same input	Observable result
MultiViewDx-AN (full multi-view)	Answer: C. Measure heart rate, respiratory rate, and blood pressure. The right- and left-eye fundus photographs show bilateral optic-disc edema, more pronounced in the left eye with associated nerve-fiber-layer hemorrhage. OCT additionally shows center-involving macular edema in both eyes. In a 15-week pregnant patient with hypertension and chronic kidney disease, this bilateral fundus-plus-OCT pattern is concerning for hypertensive end-organ injury rather than isolated progression of diabetic retinopathy. The immediate next step is to check vital signs and blood pressure to evaluate for malignant hypertension with papillopathy.	Correct. Explicitly links both fundus views, bilateral OCT findings, and systemic risk factors before selecting the urgent next step.
MultiViewDx-NoMV (non-multi-view)	Answer: B. Determine blood glucose level and perform glycated hemoglobin testing. The patient has type 1 diabetes and proliferative diabetic retinopathy. The retinal hemorrhages and bilateral macular edema are compatible with worsening diabetic retinopathy during pregnancy. Blood glucose and HbA1c should therefore be assessed to determine whether poor glycemic control is causing the visual decline and to guide diabetes management.	Incorrect. Recognizes plausible local abnormalities but anchors on diabetes and fails to connect the bilateral optic-disc pattern and OCT findings with hypertension, kidney disease, and pregnancy.

Appendix

Table 10: Illustrative simulated outputs from the full and Non-Multi-View settings on the JAMA case in Table 14 .

Inference condition	Cases	Accuracy
Evidence-bearing view removed; vignette and remaining inputs retained	100	31%

Appendix

Table 11: Inference-time removal of the evidence-bearing view on 100 multi-view cases. Accuracy decreases to 31%.

Model	VQA-RAD	SLAKE	PathVQA	PMC-VQA ∗	Documented training stage / exposure basis
Medical multimodal VLMs
BiomedGPT	NAN	NAN	NAN	NAN	The paper reports task-specific fine-tuning on VQA-RAD, SLAKE, and PathVQA; the Table 2 checkpoint is unspecified.
Med-R1-2B	$\times$	$\times$	$\times$	$\times$	GRPO training uses an open-access OmniMedVQA split; the four official training partitions are not listed.
MedVLM-R1-2B	NAN	NAN	NAN	NAN	GRPO uses a pooled set containing all four benchmarks; per-source split provenance is undisclosed.
HealthGPT-M3	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	Stage 3 visual instruction fine-tuning includes VQA-RAD, SLAKE, and PathVQA; PMC-VQA is not listed.
BioMedix2-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	Multimodal instruction alignment uses restructured VQA-RAD, SLAKE, and PathVQA training data.

Appendix

Table 12: Benchmark-training exposure for every model reported in Table 2 .

Model	VQA-RAD	SLAKE	PathVQA	PMC-VQA ∗	Training stage in the Table 3 setting
LLAVA-v1.5-LLAMA3-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	Benchmark-specific supervised fine-tuning on each official training split.
LLAVA_Med-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	Benchmark-specific supervised fine-tuning on each official training split.
HuatuoGPT-Vision-8B	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	Benchmark-specific supervised fine-tuning on each official training split.
MultiViewDx	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	Benchmark-specific supervised adaptation on the corresponding official training split.

Appendix

Table 13: Benchmark-training exposure for every model in Table 3 .

Clinical Prioritization Error Question. A woman in her mid-20s presented with subacute bilateral vision loss that was worse in the left eye. Her medical history was remarkable for type 1 diabetes diagnosed at 16 years of age and proliferative diabetic retinopathy in both eyes that had been treated with panretinal photocoagulation 7 years earlier. She had undergone pars plana vitrectomy with endolaser to treat a tractional retinal detachment in her right eye 2 years before this presentation. She also had a history of hypertension and chronic kidney disease, and she was

15 weeks into pregnancy. Visual acuity was 20/50 OD and 20/100 OS. Intraocular pressure was normal bilaterally, and no relative afferent pupillary defect was detected. Findings of an anterior segment examination were normal. The patient was in no apparent distress and denied any headache, chest pain, or focal weakness. Ophthalmoscopic examination (Figure) revealed mild optic nerve head edema that was greater in the left eye than the right eye with associated nerve fiber layer hemorrhage in the left eye. Nerve fiber layer infarctions, dot and blot hemorrhages, and lesions caused by panretinal photocoagulation also were seen bilaterally. Optical coherence tomography showed macular edema that involved the center of the macula in both eyes (Figure, inset).

Choices:

(A) Obtain a fluorescein angiogram;

(B) Determine blood glucose level and perform glycated hemoglobin test;

(D) Perform immediate computed tomography of the head.

Appendix

Table 14: JAMA case study illustrating a clinical prioritization error.

Human Annotated Sample Case

Image Caption: 1. Sagittal T2-weighted image shows a large high signal intensity cystic mass (red arrow) with a nodular, low signal intensity component (yellow arrow). Normal left ovary with follicles (white arrow) is seen posteriorly – negative beak sign. 2. Sagittal T2-weighted image shows a large high signal intensity cystic mass (red arrow) arising from right ovary (white arrow) – positive beak sign. There is another small, nodular, low signal intensity component (yellow arrow). 3. Axial T1-weighted image shows a large cystic mass (red arrow). The lesion has parts of low signal intensity (yellow arrow) and its content is slightly hyperintense (asterisk). 4. Axial fat-suppressed post-contrast T1-weighted image shows wall enhancement (red arrow) and solid component enhancement (yellow arrow) and its content is hypointense (asterisk). 5. Diffusion-weighted MR image (b1000) shows hyperintensity of the solid component (yellow arrow). 6. Axial ADC map from diffusion-weighted MR image (Fig. 1e) demonstrates marked hypointensity of the solid component (yellow arrow), in keeping with dense cellularity of the lesion.

Clinical History: A 21-year-old G0P0 woman with no medical history was referred to our institution for a sonographically detected cystic right adnexal mass. She has a history of pelvic discomfort without other complaints. Physical examination was normal. Laboratory findings were also normal except for an elevated

CA 125 65.2 U/mL (normal <35.0).

Image Findings: MRI examination revealed a cystic tumour arising from the right ovary with 7.5 cm. On T2-weighted images, the signal intensity of the cyst content was high and two small nodular peripheral solid components were detected, adhering to its internal wall, with low signal (Fig. 1a, b). The normal left ovary was present with follicles (Fig. 1a). On pre-contrast T1-weighted images, the mass exhibited slightly high signal intensity (Fig. 1c). On contrast-enhanced fat-suppressed T1-weighted images, wall enhancement and solid component enhancement were detected (Fig. 1d). Finally, the ADC map (Fig. 1f) from diffusion-weighted image (Fig. 1e) demonstrates marked hypointensity of the solid component, in keeping with its dense cellularity. Surgical excision was proposed and accepted by the patient. The histopathological investigation revealed a typical ovarian serous borderline tumour.

Discussion: Borderline ovarian tumours are uncommon ovarian neoplasms, intermediate between benign and malignant types, corresponding to 5% of all epithelial ovarian tumours. [1, 2] Serous borderline tumour represents the most common type of borderline tumours arising in the ovary, and typically, it is confined to the adnexa and presents an indolent course. [3] However, up to 6.8% of these tumours can progress to low grade serous carcinoma. [3] Serous borderline tumours are divided into typical (90%) and borderline tumours with micro-papillary patterns (5%–10%). [4] These neoplasms usually present as bilateral adnexal masses with more

proliferation of papillary projections than do benign cystadenomas, they are often seen in younger patients, and laboratory findings show the serum CA-125 level mildly elevated. [2, 3, 5, 6] The peak age of presentation is 45 years. [5] Small tumours usually do not cause symptoms and are often detected as an incidental finding on sonography. [7] Larger or more advanced neoplasms might cause pain or pelvic discomfort. The diagnosis of this type of tumour is based on histopathological examination. As they are staged using the same ovarian cancer staging of malignant ovarian neoplasms [5], MRI plays a crucial rule in this evaluation. There are no pathognomonic imaging features of borderline ovarian tumours and they frequently resemble a benign lesion. Nevertheless, some pathologic and MR imaging studies suggested that large papillary projections are highly suggestive of borderline or malignant neoplasms. [1, 8] The preferred treatment is bilateral oophorectomy. Fertility sparing procedures such as unilateral oophorectomy or cystectomy are related with a higher incidence of recurrence when compared with bilateral oophorectomy (10-20% vs. approximately 5% for radical surgery). [3, 4] Young age (<30 years) has also been found to be a risk factor for recurrent disease in patients treated conservatively. As in this case, patients should be instructed to complete child-bearing within a short time frame following initial diagnosis, due to the high risk of recurrence. [3] The most important feature in predicting an adverse outcome in serous borderline tumours is the presence of serous lesions involving the peritoneum (so-called invasive implants). [3, 4] Tumours without invasive implants, as in this case, have been considered to be nonaggressive tumours, associated with a good prognosis and high survival rate. Follow-up using ultrasound imaging is mandatory, with particular attention paid to the remaining ovary in conservatively treated young patients. [4] Written informed patient consent for publication has been obtained.

Appendix

Table 15: Human-annotated clinical case normalized into the MultiViewDx evidence-linked schema.

Retriever-Guided Example

Prompt: Generate the Image description based on the image. GPT-4o: The image is a sagittal CT scan, likely of the thoracic or abdominal region. The scan shows various soft tissue structures, including major organs. The blue arrow points toward a specific structure, possibly a blood vessel or part of an organ. This could indicate an abnormality such as a mass, an aortic aneurysm, or an area of vessel narrowing or dissection. In the background, major anatomical landmarks like the heart, aorta, and vertebral column are visible, as well as some portions of the liver or spleen. The areas of high contrast indicate dense structures, such as bones, while softer tissues are seen in varying shades of gray, highlighting different densities. This scan could be used to evaluate conditions like vascular abnormalities, tumors, or other pathologies in the chest or upper abdominal region. Further clinical information would be necessary to confirm the exact diagnosis.

Prompt(with retriever)

Prompt: Generate the Image description based on the image. Here are some medical guidelines and Clinical cases for you. Case Discussion: A case of asymmetric hypertrophic cardiomyopathy with mild subaortic stenosis and partial systolic anterior motion of the mitral valve. Since the pressure gradient obtained by phase contrast imaging was rather low, echocardiographic reevaluation and/or follow-up was recommended. The case also demonstrates a typical pattern of myocardial

fibrosis in the basal and midventricular septal segments at the anterior and posterior right ventricular insertion points.

GPT-4o: The image is a coronal oblique CT scan of the thoracic region, focused on the heart and its surrounding structures. The blue arrow indicates a subaortic ventricular septal defect (VSD), measuring approximately 2.7 cm. This congenital heart defect refers to an abnormal opening in the interventricular septum (the wall separating the left and right ventricles of the heart), located just below the aortic valve. The surrounding anatomy includes parts of the heart chambers, likely showing the left and right ventricles, as well as nearby vascular structures such as the aorta. The defect appears to allow abnormal communication between the ventricles, which could lead to altered blood flow dynamics, potentially causing symptoms such as shortness of breath, fatigue, or heart failure if left untreated.

Appendix

Table 16: Retriever-guided synthesis example comparing GPT-4o output with and without retrieved clinical context.

Explore similar work

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
Jun 10, 2026Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6Medical Vision-Language Models Recent Vision-Language Models
York University · Vector Institute · University of British Columbia +5
DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs
May 22, 2026Jiazhen Pan, Weixiang Shen, Jun Li +7Patient Trajectories Traces
Technical University of Munich (TUM) · TUM University Hospital · Munich Center for Machine Learning (MCML) +3
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
Sep 3, 2026Benlu Wang, Yifan Zhang, Jiaqing Yu +7Medical Visual Question Answering Multimodal Clinical Data
University of Massachusetts Amherst · 4Yale University · University of Massachusetts Lowell +6

Jun 10, 2026, cs.CVJ/K move · Enter open · S save

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci, Abeer Badawi+5

York University · Vector Institute · University of British Columbia+5

Medical Vision-Language Models Recent Vision-Language Models Text Corpora Scientific Discovery

Open paper AskPDF arXiv

High-stakes clinical use of large vision-language models (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers. We introduce OpenMedReason, a large-scale, open multimodal medical reasoning corpus comprising approximately 450K image-question-answer instances whose reasoning traces are primarily derived from curated biomedical, human-authored scientific articles. OpenMedReason provides high-fidelity supervision beyond synthetic chains of thought, covering diverse medical domain vision modalities such as radiological scans, microscopic images, visible light photographs, charts, and others. We complement it with OpenMedReason-Bench, a held-out benchmark that allows fine-grained evaluation of LVLMs along three complementary axes of capability, including perception, medical knowledge, and rationale, enabling diagnostic evaluation beyond final-answer accuracy. OpenMedReason is a rich training resource that exhibits its effectiveness in both supervised fine-tuning (SFT) and reinforcement-based alignment. Training with OpenMedReason yields a 20% average improvement in VQA accuracy over the base model and achieves performance within 4.2% of the strongest comparable-scale medical LVLMs. Fine-grained performance analysis confirms that the gains are not concentrated in any single axis: OpenMedReason improves perception, medical knowledge, and rationale jointly, and its reasoning traces are preferred over those of the base model in 86.1% of pairwise comparisons. We release the code and dataset at huggingface.co/datasets/neginb/OpenMedReason.

Abstract

Explore similar work

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs

MedQA-MM: Shortcuts Behind Medical Visual Reasoning