Imaging-anchored multiomics in cardiovascular disease: integrating cardiac imaging, bulk, single-cell, and spatial transcriptomics
Authors: Minh H. N. Le, Thanh-Huy Nguyen, Tao Li, Bao Quang Gia Le, Han H. Huynh, Monika Raj, Carl Yang, Min Xu, Tuan Vinh, Nguyen Quoc Khanh Le
Organizations: International Ph.D. Program in Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan · AIBioMed Research Group, Taipei Medical University, Taipei, Taiwan · Computational Biology Department, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA · Department of Computer Science, Emory University, Atlanta, GA, USA · Department of Chemistry, Emory University, Atlanta, GA, USA · International Master Program for Translational Science, College of Medical Science and Technology, Taipei Medical University, Taipei 110, Taiwan · Medical Sciences Division, University of Oxford, Oxford, United Kingdom · In-Service Master Program in Artificial Intelligence in Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan · Translational Imaging Research Center, Taipei Medical University Hospital, Taipei, Taiwan
Cardiovascular disease arises from interactions between inherited risk, molecular programmes, and tissue-scale remodelling that are observed clinically through imaging. Cardiac MRI (CMR), computed tomography (CT), and echocardiography are integral to routine cardiovascular care, while bulk RNA sequencing, single-cell RNA sequencing, and spatial transcriptomics are providing increasingly detailed molecular characterization of cardiac tissue. Yet, these imaging and molecular data are still analysed in largely separate pipelines. This review examines joint representations that link cardiac imaging phenotypes to transcriptomic and spatially resolved molecular states. An imaging-anchored perspective is adopted in which echocardiography, CMR, and CT define a spatial phenotype of the heart, and bulk, single-cell and spatial transcriptomics provide cell-type- and location-specific molecular context. We define the representation requirements of each modality, compare multimodal fusion strategies, and synthesize integrative pipelines for radiogenomics, spatial alignment, and image-based gene-expression prediction, together with their validation requirements, limitations, and failure modes. Spatial multiomic maps of human myocardium and atherosclerotic plaque, together with single-cell, spatial, and multimodal medical foundation models, are advancing imaging-anchored multiomics; however, cost, scalability, and tissue availability remain substantial barriers to large-scale cardiovascular translation.
Figures & tables
Dataset/study
Type
Cardiovascular focus
Modalities included
Approximate scale and access
Link
UK Biobank imaging extension [ 17 ]
Population imaging cohort
Cardiac structure, function, and risk in mid-life adults
CMR, vascular ultrasound and other MRI sequences linked to genome-wide genotyping, exome sequencing, plasma proteomics, metabolomics, EHR, and questionnaires
∼ 100,000 participants with CMR and ¿500,000 with genomics; application-based access
UK Biobank
Multi-Ethnic Study of Atherosclerosis (MESA) [ 31 ]
Population cohort with multimodal imaging
Subclinical atherosclerosis and incident CVD in a diverse community sample
CMR, coronary artery calcium CT, carotid ultrasound, and other imaging linked to genotype, labs, and risk-factor profiling
∼ 6,800 participants; controlled access via NHLBI repositories
Risk stratification in hypertrophic cardiomyopathy
CMR, echocardiography and electrocardiography (ECG) combined with targeted gene sequencing and circulating biomarkers
∼ 2,700 patients; collaborative access via data-use agreements
HCMR
MIMIC-IV and related PhysioNet modules [ 33 ]
Critical-care waveform and EHR resource
Acute CVD and haemodynamic instability in ICU/ED settings
Twelve-lead ECG waveforms linked to high-resolution EHR; additional echo and note modules for subsets
∼ 800,000 ECGs in ∼ 160,000 subjects; credentialed access via PhysioNet
PhysioNet
EchoNet family (EchoNet-Dynamic, EchoNet-LVH) [ 5 , 16 ]
Curated echocardiography video datasets
LV systolic function, wall motion, and hypertrophy phenotypes
Apical four-chamber echo videos with contour tracings, volumetric measurements, and labels (EF, LVH)
On the order of 104 videos; de-identified datasets downloadable for research
EchoNet
Cardiac Atlas Project CMR cohorts [ 34 ]
Aggregated CMR repositories
Cardiac structure and function in health and early disease
3D cine CMR studies with segmentations, finite-element meshes and clinical covariates in selected cohorts
Thousands of CMR exams; access under data-use agreements
Cardiac Atlas
Table 1: Public multimodal cardiology datasets relevant to imaging-anchored multiomics, including resources that directly pair cardiovascular imaging with molecular data and adjacent multimodal cardiovascular resources useful for encoder pretraining and transfer learning.
Biomarker (modality)
Clinical interpretation
Typical computational extraction
Plausible or reported molecular correlates
LGE extent and pattern (CMR) [ 20 ]
Focal myocardial scar/fibrosis; arrhythmic and HF risk
Segmentation of LGE-positive regions; regional burden by AHA segments; texture/radiomics
Fibroblast activation, collagen deposition (ECM programmes), immune infiltration in border zones; stress and remodelling programmes in cardiomyocytes
Global longitudinal strain (echo) [ 21 ]
Subclinical systolic dysfunction; prognostic across many CVD states
Speckle-tracking strain curves; regional strain maps; video encoders for deformation fields
Inflammatory adipose and vascular wall transcripts; cytokine signalling and fibrosis-related programmes
Table 2: Representative cardiovascular imaging biomarkers that often serve as anchors in imaging-anchored multiomics, with typical computational extraction strategies and plausible molecular correlates.
Figure 1: The resolution scale illustrates the gap between in vivo cardiac imaging at centimetre-to-millimetre resolution and molecular spatial profiling, from Visium 55 µm spots and Visium HD/Stereo-seq 2-8 µm bins to single-cell platforms such as Xenium and MERSCOPE. Cardiac imaging modalities, including echocardiography, cardiac magnetic resonance imaging (CMR), and cardiac CT/CTA, define spatial phenotypes of the heart using measures such as ejection fraction, global longitudinal strain, wall motion, late gadolinium enhancement, T1 mapping, cine imaging, plaque features, fat attenuation index, and coronary artery calcium. These phenotypes can be indexed at the voxel, American Heart Association segment, or joint latent-space level and linked to molecular omics layers, including spatial transcriptomics, bulk or single-nucleus RNA sequencing, and proteomics/spatial omics. The resulting joint representations support downstream tasks including clinical prediction, virtual transcriptomics, spatial alignment, radiogenomics, and multimodal fusion. Foundation models spanning single-cell/spatial omics, vision–omics, imaging, and medical multimodal data provide computational support across the framework. Applications include disease subtyping, mechanism discovery, biomarker prioritisation, and risk prediction. Abbreviations: AHA, American Heart Association; CMR, cardiac magnetic resonance imaging; CT, computed tomography; CTA, computed tomography angiography; EF, ejection fraction; GLS, global longitudinal strain; LGE, late gadolinium enhancement; FAI, fat attenuation index; CAC, coronary artery calcium; scVI, single-cell variational inference; DCA, deep count autoencoder; DCCA, deep canonical correlation analysis; MOFA+, multi-omics factor analysis; VLM, vision–language model.
Figure 2: Multimodal fusion architectures for radiomics–omics integration. Panel (A): early fusion concatenates radiomics and omics feature vectors followed by a single predictive model. Panel (B): intermediate fusion uses modality-specific encoders and a shared latent space, often with reconstruction heads; the shared latent captures biology concordant across modalities while private components retain modality-specific information. Panel (C): late fusion combines predictions from separate radiomics and omics models via an ensemble or calibration layer; it is robust to missing modalities but cannot capture cross-modal feature interactions. Panel (D): graph-based and contrastive approaches construct cross-modal latent spaces using biological or spatial graphs and contrastive objectives.
Approach
Strengths
Common failure modes
When it is a good default
Classical regression/survival models with hand-crafted radiomics summaries
High interpretability; strong calibration; stable in small samples
Limited ability to capture spatial heterogeneity and cross-modal interactions; features may miss subtle biology
Early translational work; limited paired data; when interpretability and calibration dominate
Early fusion (feature concatenation)
Simple; uses standard ML toolkits; often competitive baselines
Sensitive to missing modalities; scaling/normalization issues; collinearity, and overfitting with high-dimensional omics
Moderate sample sizes with mostly complete modalities; when a strong baseline is needed quickly
Intermediate fusion (joint latent space: DCCA, multimodal VAE, and HEALNet)
Captures non-linear cross-modal interactions; supports “virtual omics” and mechanistic exploration; handles missing modalities via product-of-experts
Latent space collapse; modality imbalance; requires careful regularization and validation
When paired subsets exist but are not large; when cross-modal generation/interpretation is desired
Cannot learn feature-level synergy; may underperform when cross-modal interactions are important; forfeits opportunity to learn from costly paired data
When missingness is high; when unimodal models are already mature and well-validated; as deployment fallback
Graph/contrastive and transformer-based fusion
Naturally incorporates spatial and biological priors; can scale with pretraining; captures neighbourhood relationships lost by flat vectors
Data-hungry; computationally expensive; prone to spurious shortcuts without strong evaluation
When large weakly labelled data exist for pretraining, or strong spatial/biological graphs are available
Table 3: Practical comparison of traditional unimodal/low-dimensional modelling versus multimodal fusion strategies for imaging-anchored multi-omics.
Bulk and single-cell transcriptomics, epigenomics, proteomics; imaging-derived features can be included as an additional view
Probabilistic factor models (MOFA+) and hierarchical VAEs (scVI) that learn joint low-dimensional representations while explicitly modelling batch effects and missing data
Provide molecular latent spaces—shared factors or scVI latent variables—that can be aligned with cardiac imaging embeddings for radiogenomic and spatial integration
Established ecosystem with documented APIs; standard entry point for cardiovascular multiomics before fusion with imaging
Multimodal VAEs [ 80 , 81 ]
Heterogeneous combinations of imaging, omics, and clinical data
Product-of-experts or mixture-of-experts aggregation of modality-specific encoder posteriors into a shared latent space; joint posterior remains well-defined when modalities are missing
Prototype framework for learning shared imaging–omics latent spaces; supports “virtual omics” generation from imaging inputs alone
Handle missing modalities by design; probabilistic formulation enables uncertainty quantification and cross-modal simulation
PORPOISE [ 87 ]
Whole-slide H&E histopathology and multi-omic profiles (RNA-seq, CNV, mutations) across 14 TCGA cancer types
Attention-based multiple instance learning (MIL) for whole-slide images (WSIs) fused with a self-normalizing network for omics via Kronecker-product gated fusion; weakly supervised with survival labels
Architecturally transferable to cardiac histology–omics; interpretable morpho-molecular prognostic markers at pan-cancer scale
First large-scale interpretable histology–genomic fusion; open-source code and interactive atlas; highlighted batch-effect risks in site-unaware splits [ 88 ]
HEALNet [ 82 ]
Whole-slide histopathology and multi-omic data (four TCGA cancer datasets)
Iterative cross-attention between a shared latent bottleneck and modality-specific token streams; processes raw inputs rather than precomputed embeddings
Directly applicable to cardiac tissue sections with matched omics; handles missing modalities at both training and inference
State-of-the-art survival prediction among end-to-end fusion models; open-source; modular design facilitates addition of imaging modalities
Cardiovascular imaging–omics studies
Table 5: Representative frameworks and studies integrating cardiovascular imaging with molecular or other high-dimensional omic data, including general-purpose frameworks that have been applied or are directly applicable to imaging-anchored multiomics and cardiovascular studies that exemplify concrete imaging–omics applications in cardiology.
Figure 3: Archetypal pipelines for imaging–omics integration. Radiogenomics workflows link imaging-derived features to bulk or single-cell molecular profiles via feature extraction, joint latent modelling, and association or causal inference; spatial molecular alignment pipelines register in vivo imaging, ex vivo imaging, histology and spatial transcriptomics into a common coordinate system; virtual transcriptomics approaches use deep models to predict molecular modules or spatial expression maps directly from imaging inputs.
Figure 4: Translational roadmap from imaging-anchored multiomics to digital twins. Top lane: model stack comprising modality-specific encoders, multimodal fusion models and transformer or foundation model architectures. Middle lane: an agentic layer in which large language models orchestrate tool calls to encoders, fusion models and knowledge bases to generate structured reports and explanations. Bottom lane: translational path from data curation and model development through external validation and deployment to decision support and digital twin–enabled in silico trials.
Cardiovascular diseases are the leading cause of death. Cardiac phenotypes (CPs), e.g., ejection fraction, are the gold standard for assessing cardiac health, but they are derived from cine cardiac magnetic resonance imaging (CMR), which is costly and requires high spatio-temporal resolution. Every magnetic resonance (MR) examination begins with rapid and coarse localizers for scan planning, which are discarded thereafter. Despite non-diagnostic image quality and lack of temporal information, localizers can provide valuable structural information rapidly. In addition to imaging, patient-level information, including demographics and lifestyle, influence the cardiac health assessment. Electrocardiograms (ECGs) are inexpensive, routinely ordered in clinical practice, and capture the temporal activity of the heart. Here, we introduce C-TRIP (Cardiac Tri-modal Representations for Imaging Phenotypes), a multi-modal framework that aligns localizer MRI, ECG signals, and tabular metadata to learn a robust latent space and predict CPs using localizer images as an opportunistic alternative to CMR. By combining these three modalities, we leverage cheap spatial and temporal information from localizers, and ECG, respectively while benefiting from patient-specific context provided by tabular data. Our pipeline consists of three stages. First, encoders are trained independently to learn uni-modal representations. The second stage fuses the pre-trained encoders to unify the latent space. The final stage uses the enriched representation space for CP prediction, with inference performed exclusively on localizer MRI. Proposed C-TRIP yields accurate functional CPs, and high correlations for structural CPs. Since localizers are inherently rapid and low-cost, our C-TRIP framework could enable better accessibility for CP estimation.
Busra Nur Zeybek, Özgün Turgut, Yundi Zhang +5
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Germany · Munich Center for Machine Learning (MCML), Germany · Department of Diagnostic and Interventional Neuroradiology, School of Medicine, TUM University Hospital, Germany. +1
We introduce EchoXFlow, a clinical echocardiography dataset with ultrasound in its native acquisition geometry. Existing public datasets offer limited opportunities to study how cardiac anatomy, myocardial motion, and blood flow relate: Doppler velocities are typically absent or fused as RGB overlays on grayscale anatomical images, and acquisitions are released only after scan conversion and lossy vendor display processing. EchoXFlow comprises 35,377 recordings from 666 routine-care examinations of 622 patients. It preserves the timing, geometry, and modality relationships needed for physically grounded echo learning. Each recording is retained as separable modality-specific streams: temporally resolved 1D, 2D, and 3D data alongside multiple Doppler modalities, paired with a synchronized ECG. Clinical annotations range from guideline-defined measurements to dense 2D myocardial contours and 3D left-ventricular endocardial meshes. With its associated open-source tooling, EchoXFlow enables cross-modal, acquisition-aware learning tasks that cannot be formulated from conventional scan-converted videos alone, and serves as a testbed for 4D vision and physically grounded multi-modal learning more broadly. The dataset is available at https://huggingface.co/datasets/Ahus-AIM/EchoXFlow and the tooling at https://github.com/Ahus-AIM/EchoXFlow
Elias Stenhede, Joanna Sulkowska, Eivind Bjørkan Orstad +2
Medical Technology & E-Health, Akershus University Hospital, Norway · Department of Technology Systems, University of Oslo, Norway · Department of Cardiology, Akershus University Hospital, Norway
Cardiovascular disease remains the leading cause of global mortality, with progress hindered by human interpretation of complex cardiac tests. Current AI vision-language models are limited to single-modality inputs and are non-interactive. We present MARCUS (Multimodal Autonomous Reasoning and Chat for Ultrasound and Signals), an agentic vision-language system for end-to-end interpretation of electrocardiograms (ECGs), echocardiograms, and cardiac magnetic resonance imaging (CMR) independently and as multimodal input. MARCUS employs a hierarchical agentic architecture comprising modality-specific vision-language expert models, each integrating domain-trained visual encoders with multi-stage language model optimization, coordinated by a multimodal orchestrator. Trained on 13.5 million images (0.25M ECGs, 1.3M echocardiogram images, 12M CMR images) and our novel expert-curated dataset spanning 1.6 million questions, MARCUS achieves state-of-the-art performance surpassing frontier models (GPT-5 Thinking, Gemini 2.5 Pro Deep Think). Across internal (Stanford) and external (UCSF) test cohorts, MARCUS achieves accuracies of 87-91% for ECG, 67-86% for echocardiography, and 85-88% for CMR, outperforming frontier models by 34-45% (P<0.001). On multimodal cases, MARCUS achieved 70% accuracy, nearly triple that of frontier models (22-28%), with 1.7-3.0x higher free-text quality scores. Our agentic architecture also confers resistance to mirage reasoning, whereby vision-language models derive reasoning from unintended textual signals or hallucinated visual content. MARCUS demonstrates that domain-specific visual encoders with an agentic orchestrator enable multimodal cardiac interpretation. We release our models, code, and benchmark open-source.
Jack W O'Sullivan, Mohammad Asadi, Lennart Elbe +10
Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Electrical Engineering, Stanford University, CA, USA +7