MedCORE: Criteria-Grounded Clinical Reasoning for Interpretable Medical Image Diagnosis

Asim Khan; Samee Ullah Khan; Dwarikanath Mahapatra

cs.CVOct 6, 2026

MedCORE: Criteria-Grounded Clinical Reasoning for Interpretable Medical Image Diagnosis

Authors: Asim Khan, Samee Ullah Khan, Dwarikanath Mahapatra

Organizations: Deparment of Computer Science, Khalifa University, Abu Dhabi, UAE · Department of Aerosapce Engineering, Khalifa University, Abu Dhabi, UAE

Abstract

Clinical diagnosis is inherently a structured reasoning process, yet existing deep learning models often bypass this structure by mapping image features directly to disease labels without explicitly interrogating the morphological and textural criteria that clinicians systematically evaluate. This limits diagnostic transparency and may compromise safe clinical deployment. We present MedCORE (Medical Criteria-Oriented Reasoning and Evidence), a structured diagnostic framework that operationalizes clinical reasoning within a vision-language architecture. For each input image, MedCORE decomposes the diagnostic process into clinically defined criteria, spatially localizes each criterion to diagnostically relevant image regions, encodes evidence through multi-scale representations that capture macro-structural and micro-textural pathological characteristics, and refines criterion representations using a Graph Attention Network that explicitly models inter-criteria dependencies. Criterion representations are further aligned with clinical text descriptors, reinforced through class-wise visual prototypes, and aggregated using uncertainty-calibrated weighting that proportionally discounts low-confidence diagnostic evidence. MedCORE is validated across three clinically heterogeneous imaging modalities, including dermoscopic lesion classification on ISIC 2018, breast ultrasound lesion characterization on BUSI, and diabetic retinopathy grading on IDRiD. Quantitatively, MedCORE achieves 89.2% accuracy, 85.7% macro-F1, and 96.4% AUC on ISIC 2018; 96.1% accuracy, 95.2% macro-F1, and 98.4% AUC on BUSI; and 84.3% accuracy, 80.2% macro-F1, and 92.8% AUC on IDRiD. These results demonstrate consistent improvements over strong CNN, transformer, biomedical vision-language, concept-based, and prototype-based baselines.

Figures & tables

Figure 1: Clinically guided interpretable medical image classification framework. Input images are encoded using a frozen BioMedCLIP ViT-B/16 backbone, while clinical criterion queries extract localized evidence. The extracted evidence is refined through multi-scale encoding and graph attention reasoning with clinical priors. The model outputs diagnosis probabilities, criterion reliability scores, uncertainty estimates, and interpretable heatmaps.

Dataset	Task	Classes	Clinical criteria used by MedCORE
ISIC 2018	Dermoscopic lesion classification	7 classes	Asymmetry; border irregularity; colour variation; dermoscopic structures; vascular patterns
BUSI	Breast ultrasound classification	3 classes	Lesion shape; margin characteristics; internal echo pattern; posterior acoustic features; lesion orientation
IDRiD	Diabetic retinopathy grading	5 grades	Microaneurysm density; haemorrhage severity; hard exudate extent; soft exudate presence; neovascularisation

Table 1: Experimental setup and dataset-specific clinical criteria used by MedCORE.

Method	ISIC 2018			BUSI			IDRiD			Avg. F1
Method	Acc.	F1	AUC	Acc.	F1	AUC	Acc.	F1	AUC	Avg. F1
ResNet-50 [ 15 ]	81.2	75.8	90.4	87.4	85.9	93.5	73.5	68.4	84.7	76.7
DenseNet-121 [ 11 ]	82.7	77.3	91.2	89.2	87.8	94.6	75.4	70.1	86.3	78.4
EfficientNet-B0 [ 24 ]	84.1	78.8	92.0	90.6	89.1	95.5	76.9	72.0	87.5	80.0
ViT-B/16 [ 8 ]	84.8	79.6	92.5	91.2	89.8	96.1	78.0	73.1	88.4	80.8
Swin-T [ 16 ]	85.6	80.8	93.1	92.1	90.7	96.6	79.2	74.6	89.2	82.0

Table 2: Quantitative comparison with baseline methods. Accuracy, Macro-F1, and AUC are reported as percentages. The last column reports the average Macro-F1 across the three datasets.

Figure 2: Ablation study showing the average Macro-F1 drop from the full MedCORE model. Larger values indicate a more important component.

Variant	F1 drop	Avg. F1
Full MedCORE	0.0	87.0
w/o criterion localization	2.9	84.1
w/o multi-scale encoder	1.7	85.3
w/o GAT reasoning	2.0	85.0
w/o VL alignment	1.4	85.6
w/o prototypes	1.1	85.9

Table 3: Ablation study based on the average Macro-F1 over ISIC 2018, BUSI, and IDRiD. The full MedCORE model obtains an average Macro-F1 of 87.0%.

Figure 3: Sensitivity analysis with respect to the number of clinical criteria $K$ . The best performance is obtained with $K=5$ , which corresponds to the clinically defined criterion set used in MedCORE.

Number of criteria $K$	Avg. Macro-F1
1	83.6
3	85.5
5	87.0
7	86.6
9	85.9

Table 4: Average Macro-F1 under different numbers of clinical criteria.

Figure 4: Reliability diagram comparing MedCORE with BiomedCLIP fine-tuning. MedCORE remains closer to the perfect calibration line, indicating better agreement between predicted confidence and empirical accuracy.

Conf.	Perfect calibration	BioMedCLIP-FT	MedCORE
0.55	0.55	0.50	0.56
0.65	0.65	0.58	0.66
0.75	0.75	0.67	0.74
0.85	0.85	0.76	0.84
0.95	0.95	0.83	0.93

Table 5: Reliability diagram values from Fig. 4 . Values report empirical accuracy at each confidence bin.

Explore similar work

Jun 10, 2026cs.CV

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

High-stakes clinical use of large vision-language models (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers. We introduce OpenMedReason, a large-scale, open multimodal medical reasoning corpus comprising approximately 450K image-question-answer instances whose reasoning traces are primarily derived from curated biomedical, human-authored scientific articles. OpenMedReason provides high-fidelity supervision beyond synthetic chains of thought, covering diverse medical domain vision modalities such as radiological scans, microscopic images, visible light photographs, charts, and others. We complement it with OpenMedReason-Bench, a held-out benchmark that allows fine-grained evaluation of LVLMs along three complementary axes of capability, including perception, medical knowledge, and rationale, enabling diagnostic evaluation beyond final-answer accuracy. OpenMedReason is a rich training resource that exhibits its effectiveness in both supervised fine-tuning (SFT) and reinforcement-based alignment. Training with OpenMedReason yields a 20% average improvement in VQA accuracy over the base model and achieves performance within 4.2% of the strongest comparable-scale medical LVLMs. Fine-grained performance analysis confirms that the gains are not concentrated in any single axis: OpenMedReason improves perception, medical knowledge, and rationale jointly, and its reasoning traces are preferred over those of the base model in 86.1% of pairwise comparisons. We release the code and dataset at huggingface.co/datasets/neginb/OpenMedReason.

Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6

York University · Vector Institute · University of British Columbia +5

Medical Vision-Language Models Recent Vision-Language Models Text Corpora Scientific Discovery

Ask PDF

Oct 19, 2024cs.CL

MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis

Medical multimodal large language models (MLLMs) can perform well on existing medical visual question answering (MedVQA) benchmarks, but their training data often does not match clinical diagnosis. Most supervision is organized around isolated images or short QA pairs, leaving two structures weakly specified: how evidence leads to a decision, and how views, series, modalities, and patient context from the same case are linked. We introduce MultiViewDx, a partly physician-validated multimodal instruction dataset for evidence-linked multi-view medical imaging diagnosis. MultiViewDx uses the clinical case as the supervision unit. It links imaging studies with patient context, normalizes heterogeneous reports into an evidence-linked workflow (evidence -> findings -> differential discussion -> diagnosis), and uses a unified image-text retriever to constrain instruction synthesis to source-supported evidence. It covers X-ray, CT, MRI, ultrasound, histopathology, and other clinical visual sources. We fine-tune MultiViewDx-8B-AN and evaluate it on both existing MedVQA benchmarks and real-world case-based diagnostic reasoning. Across four MedVQA benchmarks, it achieves the best average accuracy among compared systems (79.0%), outperforming HuatuoGPT-Vision-34B (66.7%) and Claude3-Opus (55.7%). Beyond MedVQA, on JAMA Clinical Challenge cases, it receives the strongest overall rating under a physician-designed rubric for key clinical points, diagnostic inference, and evidence grounding. Controlled ablations and clinician evaluation show that both case-level multi-view organization and evidence-linked reasoning targets contribute to the gain.

Junda Wang, Zonghai Yao, Yujan Ting +5

United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1

Multimodal Clinical Data Medical Visual Question Answering Diagnosis Multimodal Dataset Multi-View

Ask PDF

Feb 26, 2026cs.CV

PRIMA: Pre-training with Risk-integrated Image-Metadata Alignment for Medical Diagnosis via LLM

Medical diagnosis requires the effective synthesis of visual manifestations and clinical metadata. However, existing methods often treat metadata as isolated tags, failing to exploit the rich semantic knowledge embedded in clinical descriptions. We propose PRIMA (Pre-training with Risk-integrated Image-Metadata Alignment), a framework that integrates domain-specific knowledge into multi-modal representation learning. We first curate an expert corpus of risk--disease correlations via Retrieval-Augmented Generation (RAG) to refine Clinical ModernBERT, embedding diagnostic priors into the text encoder. To bridge the modality gap, we introduce a dual-encoder pre-training strategy utilizing DINOv3 and our refined Clinical ModernBERT, optimized by a suite of four complementary loss functions. These losses are designed to capture multi-granular semantic alignment and handle the ambiguity of clinical correlations through soft labels. Finally, we leverage Qwen3 to fuse these aligned features for precise disease classification. Extensive experiments demonstrate that PRIMA effectively harmonizes pixel-level features with abstract clinical expertise, consistently outperforming other state-of-the-art methods. Notably, our framework achieves strong performance without requiring massive data collection or exhaustive computational resources. Our code will is available at https://github.com/yqwang01/PRIMA.

Yiqing Wang, Chunming He, Ming-Chen Lu +4

Department of Biomedical Engineering, Duke University, Durham, NC, USA · Department of Ophthalmology and Visual Sciences, University of Michigan, Ann Arbor, MI, USA

Multimodal Clinical Data Multimodal Pretraining Multimodal Alignment Prior Knowledge Integration Text Corpora Audio-Visual Consistency

Ask PDF