cs.AIMar 24, 2026
SaveWhen Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models
Organizations: The Ohio State University · Hunan University · Amazon
Abstract
Vision-Language Models (VLMs) have enabled interpretable medical diagnosis by integrating visual perception with linguistic reasoning. Yet, existing medical chain-of-thought (CoT) models lack explicit mechanisms to represent and enforce causal reasoning, leaving them vulnerable to spurious correlations and limiting their clinical reliability. We pinpoint three core challenges in medical CoT reasoning: how to adaptively trigger causal correction, construct high-quality causal-spurious contrastive samples, and maintain causal consistency across reasoning trajectories. To address these challenges, we propose MedCausalX, an end-to-end framework explicitly models causal reasoning chains in medical VLMs. We first introduce the CRMed dataset providing fine-grained anatomical annotations, structured causal reasoning chains, and counterfactual variants that guide the learning of causal relationships beyond superficial correlations. Building upon CRMed, MedCausalX employs a two-stage adaptive reflection architecture equipped with causal and verify tokens, enabling the model to autonomously determine when and how to perform causal analysis and verification. Finally, a trajectory-level causal correction objective optimized through error-attributed reinforcement learning refines the reasoning chain, allowing the model to distinguish genuine causal dependencies from shortcut associations. Extensive experiments on multiple benchmarks show that MedCausalX consistently outperforms state-of-the-art methods, improving diagnostic consistency by +5.4 points, reducing hallucination by over 10 points, and attaining top spatial grounding IoU, thereby setting a new standard for causally grounded medical reasoning. The code and dataset are available at https://github.com/zhcz328/MedCausalX.
Figures & tables
Figure 1: Existing medical VLMs often lack causality-awareness, which manifests as (1) spurious CoT reasoning attends to mislocalized anatomy or (2) overlooking causal evidence for pathology diagnosis. MedCausalX can achieve trustworthy diagnosis by adaptive causal verification and correction during inference.
Figure 2: Overview of MedCausalX framework. We first construct CRMed dataset via fine-grained annotations, then factorize reasoning via a structural causal model (SCM) with reflective tokens ( <causal> , <verify> ) enabling two-stage adaptive reasoning. After causal Supervised Fine-Tuning (SFT), MedCausalX is subsequently optimized via dual-policy that couples off-policy preference alignment (DPO) with on-policy causal reinforcement (GRPO), bridging stability and adaptability to enhance the model’s causal reasoning capability.
| Method | Region-to-Text Identification | Text-to-Region Detection | ||||||||||||||||
| Structure Identification | Lesion Identification | Single-Region Detection | Multi-Region Detection | |||||||||||||||
| BLEU-1 | F1 | Recall | BertScore | Acc | BLEU-1 | F1 | Recall | BertScore | Acc | Obj-F1 | Reg-F1 | Align-F1 | IoU | Obj-F1 | Reg-F1 | Align-F1 | IoU | |
| Qwen2.5-VL [ 2 ] | 0.15 | 0.25 | 0.58 | 52.34 | 0.12 | 0.14 | 0.23 | 0.55 | 48.56 | 0.11 | 57.82 | 8.15 | 6.23 | 17.45 | 49.35 | 5.87 | 4.65 | 14.82 |
| InternVL-2 [ 10 ] | 0.13 | 0.21 | 0.52 | 51.78 | 0.11 | 0.13 | 0.20 | 0.51 | 47.92 | 0.10 | 56.60 | 7.24 | 5.45 | 16.61 | 48.23 | 5.41 | 4.18 | 13.99 |
| Med-Flamingo [ 64 ] | 0.13 | 0.72 | 4.90 | 24.87 | 0.01 | 0.12 | 0.68 | 4.72 | 23.45 | 0.01 | 18.47 | 2.35 | 1.82 | 5.23 | 12.68 | 1.54 | 1.18 | 3.47 |
| LLaVA-Med [ 41 ] | 0.43 | 1.15 | 10.69 | 34.53 | 1.74 | 0.38 | 1.05 | 9.85 | 32.18 | 1.52 | 24.83 | 3.76 | 2.94 | 7.85 | 18.95 | 2.42 | 1.87 | 5.36 |
Table 1: Comprehensive quantitative comparison on diverse region-centric tasks, including region-to-text identification and text-to-region detection. Abbreviations in this Table: Obj-F1 denotes Object F1, Reg-F1 denotes Region F1, Align-F1 denotes Alignment F1, and Acc denotes Accuracy. The best results are highlighted in bold , while the second-best are underlined .
| Method | SLAKE [ 51 ] | VQA-RAD [ 37 ] | PathVQA [ 23 ] | PMC-VQA [ 103 ] | Average | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | Diag-C | Hall | Acc | Diag-C | Hall | Acc | Diag-C | Hall | Acc | Diag-C | Hall | Acc | Diag-C | Hall | |
| General Vision-Language Models | |||||||||||||||
| Qwen2.5-VL [ 2 ] | 68.5 1.8 | 58.3 2.1 | 62.5 1.9 | 52.4 2.3 | 54.2 2.0 | 65.8 1.7 | 48.3 2.1 | 52.8 1.9 | 68.2 2.2 | 58.2 1.6 | 56.7 1.8 | 63.5 2.0 | 56.9 1.9 | 55.5 1.9 | 65.0 1.9 |
| InternVL-2 [ 10 ] | 70.8 1.6 | 61.2 1.8 | 59.8 2.1 | 55.1 2.0 | 57.5 1.9 | 62.5 1.8 | 52.7 1.9 | 55.3 2.0 | 65.3 2.1 | 61.5 1.5 | 59.2 1.7 | 60.8 1.9 | 60.0 1.8 | 58.3 1.9 | 62.1 2.0 |
| Medical-Specific Vision-Language Models | |||||||||||||||
| Med-Flamingo [ 64 ] | 33.2 2.7 | 38.5 2.5 | 75.2 2.3 | 44.6 2.6 | 42.8 2.4 | 71.8 2.5 | 47.9 2.4 | 45.6 2.3 | 68.5 2.6 | 35.9 2.8 | 40.5 2.5 | 73.8 2.4 | 40.4 2.6 | 41.9 2.4 | 72.3 2.5 |
Table 2: Performance comparison on medical VQA benchmarks with causal reasoning evaluation. Metrics include answer Accuracy (Acc), Diagnostic Consistency (Diag-C), and Hallucination Rate (Hall, lower is better). Results are reported as mean over 5-fold cross-validation. Best results in bold , second-best underlined . ’*’ denotes fine-tuned models.
Figure 3: Comparison of causal reasoning in medical VQA and region-grounded diagnosis. (a) MedRegA and Med-R1 exhibit spurious correlations and mistakes, while MedCausalX captures correct predictions via structured causal decomposition. (b) Two-stage adaptive causal reasoning and verification, producing refined bounding boxes and logically consistent diagnostic descriptions. Attention maps are used to visualize the model’s focus. GT: ground truth; A1–A3: predictions from MedCausalX, Med-R1, and MedRegA, respectively.
| Method | BLEU-1 | Region Acc | Align Acc | IoU |
|---|---|---|---|---|
| Qwen2.5-VL [ 2 ] | 18.43 | 22.30 | 8.15 | 17.45 |
| InternVL [ 11 ] | 19.46 | 57.00 | 12.00 | 38.00 |
| Med-Flamingo [ 64 ] | 21.67 | - | - | - |
| MedDr [ 22 ] | 31.27 | - | - | - |
| MedRegA [ 82 ] | 33.18 | 76.59 | 62.29 | 52.07 |
| MedCausalX | 37.18 | 81.12 | 67.38 | 55.71 |
Table 3: Performance comparison on chest X-ray report generation benchmark. Our method employs two-stage causal reasoning with adaptive verification based on MedCausalX-32B model. Best results are highlighted in bold , second-best are underlined .
| Configuration | BLEU-1 | Region Acc | Align Acc | IoU |
| MedCausalX (Full) | 37.18 | 81.12 | 67.38 | 55.71 |
| Main Component Ablation | ||||
| ✗ CRMed Dataset | 26.47 | 41.20 | 18.35 | 26.55 |
| ✗ Reflective Tokens | 29.93 | 51.20 | 32.50 | 38.46 |
| ✗ Causal SFT | 26.02 | 43.80 | 21.75 | 28.55 |
| ✗ RL Training | 26.96 | 63.50 | 47.30 | 38.09 |
Table 4: Systematic ablation of core components and training stage variants on MIMIC-CXR benchmark evaluates each module’s contribution to causal reasoning and verification capabilities.
Figure 4: Radiologists’ evaluation and comparative results. Left: 3-point clinical criteria for spatial localization (Q1), diagnostic description (Q2). Right: radar chart comparing MedRegA, Med-R1, and MedCausalX on localization (blue) and description (orange); lower scores indicate better performance.
| Param. | BLEU-1 | Region Acc | Align Acc | IoU |
| Error Localization Threshold ( ) | ||||
| 33.47 | 77.85 | 63.91 | 52.74 | |
| 35.12 | 79.67 | 66.24 | 54.83 | |
| (default) | 37.18 | 81.12 | 67.38 | 55.71 |
| 33.23 | 80.17 | 65.82 | 54.96 | |
| 34.91 | 80.34 | 61.15 | 52.68 | |
Table 5: Hyperparameter sensitivity analysis. Selected configuration (shaded) achieves optimal balance across metrics.
Figure 5: Modality distribution and factual-counterfactual composition of CRMed dataset. The dataset spans multiple imaging modalities with constructed counterfactual variants through controlled interventions, enabling robust causal reasoning evaluation.
| Hyperparameter | Value |
|---|---|
| Base Model | Qwen2.5-VL-32B |
| LoRA Rank ( ) | 64 |
| LoRA Alpha ( ) | 128 |
| LoRA Dropout | 0.05 |
| Target Modules | q_proj, k_proj, v_proj |
| Reflective Tokens | , |
Table 6: Complete hyperparameter configuration for MedCausalX training and inference.
| Model Scale | Region-to-Text Identification | Text-to-Region Detection | ||||||||||||||||
| Structure Identification | Lesion Identification | Single-Region Detection | Multi-Region Detection | |||||||||||||||
| BLEU-1 | F1 | Recall | BertScore | Acc | BLEU-1 | F1 | Recall | BertScore | Acc | Obj-F1 | Reg-F1 | Align-F1 | IoU | Obj-F1 | Reg-F1 | Align-F1 | IoU | |
| MedCausalX-3B | 53.42 | 54.68 | 55.20 | 52.15 | 48.20 | 38.29 | 39.12 | 40.35 | 48.65 | 35.12 | 68.45 | 21.47 | 18.23 | 24.49 | 55.18 | 14.23 | 12.47 | 19.82 |
| MedCausalX-14B | 73.49 | 74.23 | 75.80 | 54.20 | 55.30 | 56.43 | 57.80 | 59.25 | 70.83 | 52.80 | 79.82 | 34.62 | 28.15 | 33.75 | 65.75 | 19.56 | 17.85 | 26.18 |
| MedCausalX-32B | 79.82 | 80.15 | 81.25 | 88.45 | 70.15 | 63.27 | 64.35 | 65.80 | 81.90 | 61.35 | 82.45 | 46.82 | 44.92 | 44.35 | 72.38 | 29.51 | 26.82 | 35.76 |
Table 7: Model scaling analysis across region-centric medical reasoning tasks. We evaluate MedCausalX variants with 3B, 14B, and 32B parameters to assess the impact of model capacity on causal reasoning performance. Best results are highlighted in bold , second-best are underlined . All models trained with identical data and optimization strategies.
| Model | VQA-RAD | SLAKE | PathVQA | PMC-VQA | Average |
|---|---|---|---|---|---|
| Vision-Language Models (Zero-shot) | |||||
| LLaVA-1.5-13B [ 55 ] | 53.2 | 58.5 | 52.8 | 56.3 | 55.2 |
| InternVL-2-40B [ 10 ] | 59.8 | 65.2 | 61.5 | 67.1 | 63.4 |
| MedRegA-40B [ 82 ] | 76.9 | 84.1 | 68.5 | 79.5 | 77.3 |
| Medical CoT Methods (Qwen2.5-VL-32B Backbone) | |||||
| Med-R1 [ 36 ] | 76.9 | 85.4 | 69.5 | 81.9 | 78.4 |
Table 8: Model scale and architecture comparison across medical VQA benchmarks. Light blue background indicates selected configuration.
Figure 6: Comparison of model efficiency and performance across medical VLMs. MedCausalX achieves the highest composite score at 32B parameters, demonstrating superior causal reasoning capabilities while maintaining computational efficiency.
| Config. | Value | VQA-RAD | SLAKE | PathVQA | PMC-VQA | Average |
|---|---|---|---|---|---|---|
| (a) Training Stage Order | ||||||
| Seq. | GRPO→DPO | 73.5 | 81.4 | 68.2 | 78.9 | 75.5 |
| DPO→GRPO | 79.8 | 87.2 | 73.2 | 84.6 | 81.2 | |
| (b) GRPO Group Size | ||||||
| 4 | 76.8 | 84.3 | 70.5 | 81.7 | 78.3 | |
| 6 | 78.5 | 86.1 | 72.1 | 83.5 | 80.1 | |
Table 9: Ablation study on training strategy configuration. Light blue background indicates selected configuration, bold values denote best performance in each row.
| Config. | Value | VQA-RAD | SLAKE | PathVQA | PMC-VQA | Average |
|---|---|---|---|---|---|---|
| (a) Reward Weight Configuration | ||||||
| 1.0/0.0 | 71.5 | 78.9 | 65.3 | 76.2 | 73.0 | |
| 0.7/0.3 | 76.8 | 84.1 | 70.5 | 81.7 | 78.3 | |
| 0.5/0.5 | 79.8 | 87.2 | 73.2 | 84.6 | 81.2 | |
| 0.3/0.7 | 78.2 | 85.7 | 71.8 | 83.1 | 79.7 | |
| 0.0/1.0 | 67.3 | 74.5 | 61.2 | 71.8 | 68.7 | |
Table 10: Ablation of reward components and error localization threshold. Light blue background indicates selected configuration.
| Method | OVQA | EndoVis-VQA | Skin-VQA | Average |
|---|---|---|---|---|
| GPT-4o [ 28 ] | 43.5 | 39.8 | 46.1 | 43.1 |
| LLaVA-Med [ 41 ] | 47.2 | 44.3 | 49.5 | 47.0 |
| InternVL-2 [ 10 ] | 49.8 | 46.7 | 51.2 | 49.2 |
| Med-R1 [ 36 ] | 44.8 | 40.5 | 46.3 | 43.9 |
| MedVLM-R1 [ 67 ] | 50.3 | 47.1 | 49.8 | 49.1 |
| MedCausalX (Ours) | 58.4 | 54.2 | 59.7 | 57.4 |
Table 11: Out-of-domain evaluation on specialized medical VQA datasets. Models trained on core benchmarks and evaluated zero-shot on unseen domains.
Figure 7: Training dynamics of on-policy GRPO optimization. MedCausalX (Full) with DPO pretraining achieves higher rewards and maintains stable, compact reasoning compared to direct GRPO training from preliminary SFT baseline.
Figure 8: Structured prompt template for MedCausalX showing system instructions for pathological analysis approach and output format specifications enabling causal chain generation.
Figure 9: Chest radiograph demonstrating bilateral lung detection where MedCausalX accurately quantifies two distinct pulmonary structures through structured anatomical enumeration and spatial boundary verification.
Figure 10: Abdominal examination section validating gastric cardia identification through causal anatomical landmark detection, confirming presence via systematic spatial relationship analysis with esophagogastric junction.
Figure 11: Echocardiographic image demonstrating successful causal localization of endocardial lesion positioned adjacent to lower intraventricular septum through structured anatomical-pathological reasoning chain.
Figure 12: Pelvic CT scan validating binary anatomical detection where MedCausalX correctly confirms uterine presence through causal verification eliminating confounding structures.