Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models
Authors: Sumra Khan, Sagar Chhabriya, Aizan Zafar, Sheeraz Arif, Amgad Muneer, Anas Zafar, Shaina Raza, Rizwan Qureshi
Organizations: Department of Computer Science, Salim Habib University, Karachi, Pakistan · Computer Science, Institute of Business Administration Sukkur, Pakistan · Center for Research in Computer Vision, University of Central Florida, USA · The University of Texas MD Anderson Cancer Center, USA · Toronto Metropolitan University, Vector Institute, Canada
Medical vision-language models (VLMs) show strong performance on radiology tasks but often produce fluent yet weakly grounded conclusions due to over-reliance on a dominant modality. We introduce a context-aligned reasoning framework that enforces agreement across heterogeneous clinical evidence before generating diagnostic conclusions. The proposed approach augments a frozen VLM with structured contextual signals derived from radiomic statistics, explainability activations, and vocabulary-grounded semantic cues. Instead of producing free-form responses, the model generates structured outputs containing supporting evidence, uncertainty estimates, limitations, and safety notes. We observe that auxiliary signals alone provide limited benefit; performance gains emerge only when these signals are integrated through contextual verification. Experiments on chest X-ray datasets demonstrate that context alignment improves discriminative performance (AUC 0.918 to 0.925) while maintaining calibrated uncertainty. The framework also substantially reduces hallucinated keywords (1.14 to 0.25) and produces more concise reasoning explanations (19.4 to 15.3 words) without increasing model confidence (0.70 to 0.68). Cross-dataset evaluation on CheXpert further reveals that modality informativeness significantly influences reasoning behavior. These results suggest that enforcing multi-evidence agreement improves both reliability and trustworthiness in medical multimodal reasoning, while preserving the underlying model architecture.
Figures & tables
Figure 1: Context-aligned reasoning for responsible medical VLMs. Conventional medical vision–language models often rely on a dominant modality, which can produce confident but weakly grounded conclusions. Our framework augments the VLM with heterogeneous clinical evidence sources, including radiomic statistics, explainability activations, and vocabulary-grounded semantic cues. These signals are unified through a context agreement constraint before inference, forcing the model to justify predictions across independent evidence channels. Instead of generating free-form text, the model produces a structured clinical response containing impression, supporting evidence, calibrated uncertainty, limitations, and safety notes. This transforms prediction from generative output into an auditable decision process, improving reliability while preserving performance.
Feature Configuration
Qwen
BLIP
Radiomics only
0.555
0.564
XAI only
0.519
0.514
Text only
0.918
0.925
Radiomics + Text
0.921
0.926
XAI + Text
0.918
0.927
Radiomics + XAI + Text
0.925
0.930
Table 1: Multimodal ablation study on the OpenI dataset (AUC).
Configuration
Hallucination Rate (HR) ↓
Image only
1.14
Image + Radiomics
1.11
Image + XAI
1.01
Image + Radiomics + XAI
0.60
Image + Text
0.25
Image + Text + Radiomics + XAI
0.28
Table 2: Image-centric ablation study evaluating hallucination metrics. HR denotes the Hallucination Rate (average hallucinated keywords per study).
Evaluation Metric
Base Conf (Step 0)
Tool Augmented (Step 1)
Absolute Change ( Δ )
Uncertainty Calibration (mean ± std)
0.70±0.00
0.68±0.14
↑0.014
Evidence Length (total words)
19.4±14.0
15.3±18.0
↓4.1
Presence of Limitations
1.00
0.80
↓0.20
Presence of Safety Notes
1.00
0.80
↓0.20
Table 3: Quantitative analysis of agentic reasoning performance across fifty clinical studies. Values represent mean and standard deviation where applicable.
Model
Variant
N
R-1 ↑
R-2 ↑
R-L ↑
BERT ↑
PHI % ↓
Unsafe % ↓
Unc. rate
LLaVA-1.5-7B
A0_image_only
217
0.0808
0.0107
0.0494
0.7539
0.00
0.0
0.1344
A1_radiomics
217
0.0829
0.0079
0.0463
0.7522
0.00
0.0
0.0661
A2_xai
217
0.0832
0.0100
0.0490
0.7527
0.00
0.0
0.0929
A3_rad_xai
217
0.0773
0.0062
0.0429
0.7498
0.00
0.0
0.0269
A4_image_text
217
0.4910
0.3190
0.4711
0.7640
0.92
0.0
0.0453
A5_image_text_rad_xai
217
0.4384
0.2825
0.4168
0.7646
0.92
0.0
0.0553
Table 4: Responsible-AI evaluation and text quality metrics on the test split ( N=217 per variant). We report ROUGE (R-1/R-2/R-L) and BERTScore-F1 for generation quality, along with heuristic Responsible-AI indicators: PHI leakage rate, unsafe content rate, and uncertainty marker rate. Best values per model (higher is better for ROUGE/BERT; lower is better for PHI/Unsafe) are highlighted in bold.
Figure 2: AUC comparison across modality combinations for Qwen and BLIP embeddings on the OpenI dataset. Multimodal fusion consistently improves performance compared to single-modality features.
Question: Based on the chest X-ray, is there evidence of active cardiopulmonary abnormality?
Ground Truth (Radiology Report): No active cardiopulmonary disease.
Input Image
Agentic Reasoning and Structured Output
Step 1 (Visual Observation): Clear bilateral lung fields and normal cardiac silhouette. Step 2 (Feature Assessment): No focal consolidation, pleural effusion, or pneumothorax identified. Step 3 (Structured Reasoning): Absence of acute radiographic findings suggests no active cardiopulmonary abnormality. Step 4 (Uncertainty Calibration): Numeric uncertainty reflects limited clinical context and single-view imaging. Final Structured Output: Impression: No obvious radiographic evidence of active cardiopulmonary abnormality. Evidence: Lung fields clear. Cardiac silhouette within normal limits. Uncertainty 0 to 1: 0.35 Limitations: Single frontal radiograph without prior comparison. Safety Note: For research use only; not a substitute for expert interpretation.
Table 5: Qualitative visual reasoning example of Qwen. Green text indicates visual grounding; blue text indicates the structured reasoning chain.
Question: Based on the chest X-ray and patient history, is there evidence of active cardiopulmonary abnormality?
Ground Truth (Radiology Report): No active disease.
Input Image
Agentic Reasoning and Structured Output of BLIP
Step 1 (Visual Observation): Chest radiograph demonstrates clear bilateral lung fields and a normal cardiac silhouette. Step 2 (Feature Assessment): No focal consolidation, pleural effusion, or pneumothorax is identified via radiomic feature mapping. The heart and mediastinum appear normal. Step 3 (Structured Reasoning): The absence of acute radiographic findings suggests no active cardiopulmonary abnormality. Step 4 (Uncertainty Calibration): Uncertainty is assigned at a moderate-to-high level due to the patient’s history of renal cell carcinoma, requiring cautious clinical interpretation despite clear imaging findings. Final Structured Output: Impression: No obvious radiographic evidence of active cardiopulmonary abnormality (Uncertain). Evidence: The chest X-ray shows clear and expanded lung fields with no active disease. The heart and mediastinum appear normal, with no signs of pulmonary pathology. Uncertainty 0 to 1: 0.7 Limitations: Assessment is based on a two-view chest radiograph, and radiomics analysis reflects only a single temporal observation. Safety Note: The patient has a history of renal cell carcinoma; this output should be interpreted by a qualified radiologist.
Table 6: Qualitative visual reasoning example of BLIP. Green text indicates visual grounding; blue text indicates the structured reasoning chain.
1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1
United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1