cs.CVSep 30, 2026

From Image Interpretation to Clinical Reasoning: Upstream Physician-Context-Aware Multimodal Learning with Causal Reinforcement Learning

Authors: Jialu Pi, Yanan Ma, Weijie Chen, Owen Crystal, Shubham Trivedi, Stephen Xie, Anna Silverman, Matthew Stib, +3 more

Organizations: Dept. of Radiology, Mayo Clinic, Phoenix, AZ, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, USA · Dept. of Radiology, Mayo Clinic, Phoenix, USA · Mayo Clinic, Phoenix, USA · Dept. of Cardiology, Mayo Clinic, Phoenix, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Major adverse cardiovascular events (MACE) remain the leading cause of mortality worldwide. Opportunistic screening using routinely acquired clinical data offers a scalable approach for identifying high-risk individuals before acute events occur. Although chest X-rays (CXRs) capture latent cardiovascular biomarkers and clinical histories provide complementary patient context, existing medical vision-language models are primarily optimized for radiology interpretation rather than prognostic reasoning. We propose a causal reinforcement learning framework for multimodal clinical reasoning that integrates CXRs and physician-authored clinical histories for opportunistic MACE prediction. The framework introduces (1) a role-decoupled dual-LLM architecture that separates reasoning from risk prediction, (2) a dual-action causal reinforcement learning policy for evidence selection and reasoning optimization, and (3) causal token pruning to learn compact multimodal representations. Evaluated on an internal cohort, an emergency department cohort, and the external MIMIC dataset, the proposed framework consistently outperformed unimodal baselines and state-of-the-art medical vision-language models, achieving AUROCs of 0.720, 0.760, and 0.845, respectively. It also substantially improved reasoning quality, achieving higher GREEN scores and higher expert preference while maintaining robust predictive performance across diverse patient populations.

Figures & tables

Fig. 1 : Proposed causal reinforcement learning framework with a multimodal LLM. (a) End-to-end training pipeline integrating multimodal inputs, multimodal token generation, causal reasoning, and reinforcement learning. Red arrows indicate gradient backpropagation and parameter updates. (b) Causal policy optimization, where reinforcement learning refines the reasoning policy using causal feedback to improve reasoning consistency and align generated reasoning with targeted clinical prediction objectives. LLM denotes large language model, and LM denotes language model.
CharacteristicTotalMayo InternalMIMIC ExternalMayo ED ExternalP-value
Number of patients22,57315,8024,6162,155p<0.001p<0.001
Demographics     
Age (years)63.14 ± 17.3464.16 ± 18.0161.84 ± 17.3358.38 ± 9.65–
Female sex, n (%)9,611 (42.6%)6,454 (40.6%)2,343 (50.8%)832 (38.6%)p<0.001p<0.001
Male sex, n (%)13,001 (57.6%)9,431 (59.3%)2,272 (49.2%)1,323 (61.4%)p<0.001p<0.001
Race, n (%)     
TABLE I : Baseline demographic and clinical characteristics of patients across the internal and external cohorts. Continuous variables are presented as mean ±\pm standard deviation (or median [IQR], as appropriate), and categorical variables are presented as number (%). P-values compare cohorts using ANOVA or Kruskal–Wallis tests for continuous variables and χ2\chi^{2} test or Fisher’s exact test for categorical variables, as appropriate.
 ModelsAUCNPVPPV
  Mayo-InternalMIMIC-ExternalMayo ED-ExternalMayo-InternalMIMIC-ExternalMayo ED-ExternalMayo-InternalMIMIC-ExternalMayo ED-External
Single modality without reasoning
 Image only (CheXpert-DenseNet)0.500 ±\pm 0.0040.526 ±\pm 0.0030.499 ±\pm 0.0070.684 ±\pm 0.0050.798 ±\pm 0.0030.883 ±\pm 0.0030.322 ±\pm 0.0050.234 ±\pm 0.0030.100 ±\pm 0.002
 Image only (MedCLIP-ViT)0.539 ±\pm 0.0030.520 ±\pm 0.0030.489 ±\pm 0.0130.712 ±\pm 0.0040.789 ±\pm 0.0040.932 ±\pm 0.0030.347 ±\pm 0.0050.225 ±\pm 0.0030.051 ±\pm 0.002
 Text only (BioGPT)0.740 ±\pm 0.0030.772 ±\pm 0.0040.808 ±\pm 0.0070.797 ±\pm 0.0030.893 ±\pm 0.0030.961 ±\pm 0.0020.518 ±\pm 0.0040.416 ±\pm 0.0050.315 ±\pm 0.006
 Text only (MedCLIP-ClinicalBERT)0.743 ±\pm 0.0030.798 ±\pm 0.0040.804 ±\pm 0.0050.803 ±\pm 0.0040.903 ±\pm 0.0020.960 ±\pm 0.0020.515 ±\pm 0.0040.425 ±\pm 0.0060.294 ±\pm 0.006
TABLE II : Performance comparison of single modality and multimodal models across internal and external datasets. Values are reported as point estimate ±\pm half-width of the 95% confidence interval.
Fig. 2 : Subgroup analysis on External MIMIC datasets comparing Three models across AUC (above) and average green score (below).
Evaluation MetricMayo InternalMIMIC ExternalMayo ED External
 BaselineProposedP-valueBaselineProposedP-valueBaselineProposedP-value
Reasoning Quality
Factual correctness0.405 ±\pm 0.4530.547 ±\pm 0.425p<0.001p<0.0010.692 ±\pm 0.3410.670 ±\pm 0.385p=0.404p=0.4040.269 ±\pm 0.4060.660 ±\pm 0.392p<0.001p<0.001
Reasoning completeness0.194 ±\pm 0.2970.321 ±\pm 0.312p<0.001p<0.0010.368 ±\pm 0.2600.285 ±\pm 0.227p<0.001p<0.0010.091 ±\pm 0.2020.364 ±\pm 0.285p<0.001p<0.001
Hallucination ↓\downarrow0.587 ±\pm 0.4560.447 ±\pm 0.424p<0.001p<0.0010.302 ±\pm 0.3410.326 ±\pm 0.383p=0.329p=0.3290.724 ±\pm 0.4110.337 ±\pm 0.390p<0.001p<0.001
Clinical Utility
TABLE III : Evaluation of multimodal clinical reasoning quality and utility. Reasoning quality metrics are reported as mean ±\pm standard deviation, while clinical utility metrics are reported as point estimate ±\pm half-width of the 95% confidence interval. ‘–’ means not applicable.
Fig. 3 : Reasoning quality evaluation. (a) Example true-negative case comparing baseline and proposed model reasoning. The baseline model incorrectly predicted high risk without considering relevant CXR findings, whereas the proposed RL-based model correctly predicted low risk with clinically grounded reasoning. (b) Expert preference comparison between baseline and proposed model reasoning across user groups.
AblationMayo-InternalMIMIC-ExternalMayo ED-External
Lingshu-7BAUCNPVPPVGreen ScoreAUCNPVPPVGreen ScoreAUCNPVPPVGreen Score
Modality contribution
Image-only0.682 ±\pm 0.0060.784 ±\pm 0.0040.471 ±\pm 0.006–0.638 ±\pm 0.0040.852 ±\pm 0.0030.305 ±\pm 0.004–0.750 ±\pm 0.0070.949 ±\pm 0.0030.206 ±\pm 0.006–
Text-only0.693 ±\pm 0.0030.788 ±\pm 0.0040.466 ±\pm 0.0030.577 ±\pm 0.4160.762 ±\pm 0.0030.897 ±\pm 0.0010.387 ±\pm 0.0040.832 ±\pm 0.2820.752 ±\pm 0.0090.946 ±\pm 0.0020.238 ±\pm 0.0090.759 ±\pm 0.310
Representation learning
No Text Adapter0.639 ±\pm 0.0020.746 ±\pm 0.0030.437 ±\pm 0.0030.429 ±\pm 0.3640.719 ±\pm 0.0070.874 ±\pm 0.0040.367 ±\pm 0.0070.728 ±\pm 0.2960.733 ±\pm 0.0060.944 ±\pm 0.0020.208 ±\pm 0.0070.626 ±\pm 0.309
TABLE IV : Ablation study of Lingshu-7B across internal and external datasets. ‘–’ means not applicable

Explore similar work

CardsList

  1. Mammography Foundation Models for Opportunistic Prediction of Major Adverse Cardiovascular Events

    Sep 16, 2026Paula Feldman, Nusrat Binta Nizam, Sunwoo Kwak +3Cardiovascular Risk PredictionMammography

    Department of Radiology, Weill Cornell Medicine, New York, NY, USA · Cornell Tech, New York, NY, USA · Cornell University, New York, NY, USA

  2. MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays

    May 15, 2026Sunghwan Steve Cho, Yunseok Han, Jaeyoung DoChest X-RayClinical Reasoning Training

    AIDAS Laboratory, 1ECE & 2IPAI, Seoul National University

  3. Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

    May 19, 2026Guangzhi Xiong, Qiao Jin, Sanchit Sinha +2Medical Vision-Language ModelsChest X-Ray

    University of Virginia · National Institutes of Health