cs.CVMay 8, 2026

Pan-FM: A Pan-Organ Foundation Model with Saliency-Guided Masking for Missing Robustness

Authors: Qiangqiang Wu, Grace McIlvain, Zhou Yu, Junhao Wen

Organizations: Laboratory of AI and Biomedical Science (LABS) · 2Columbia University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Foundation models (FMs) have shown great promise in medical imaging, but most FMs are trained on unimodal data within isolated domains, such as brain MRI alone. Human aging and disease arise through coordinated biological processes across organs, therefore motivating multimodal FMs that learn whole-body representations. A key challenge, however, is that real-world multimodal biomedical data are often missing not at random, which can reduce power, limit generalizability, and introduce bias. We propose Pan-FM, a pan-organ foundation model pre-trained on imaging from seven organs (Brain, Heart, Adipose, Liver, Kidney, Spleen, and Pancreas) under realistic missing-organ scenarios. Pan-FM uses a unified backbone that handles organ missingness during both training and inference, and is pre-trained with masking-based self-distillation. We find that naive multimodal pre-training leads to dominant-organ shortcut learning bias, with the model over-relying on dominant organs such as adipose and heart. To address this, we introduce Saliency-Guided Masking (SGM), which uses the model attention distribution to adaptively mask dominant organs during pre-training, thus encouraging more balanced cross-organ, whole-body learning. Notably, SGM introduces negligible computational overhead and can be seamlessly integrated into existing self-supervised learning frameworks to improve multi-organ representation learning. On the UK Biobank, Pan-FM achieves stronger prediction across 13 disease categories and 14 single disease entities than single-organ and multi-organ baselines, with improved robustness under missing-organ settings. Pan-FM serves as a scalable solution to realistic modality-missingness in multimodal learning in system neuroscience and as a step toward more generalizable whole-body FMs.

Explore similar work

CardsList
Jul 15, 2026cs.CV

FM2^2: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM2^2, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM2^2 equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable O(1/T)O(1/\sqrt{T}) convergence and generalization guarantees. FM2^2 further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.
Shengchao Chen, Ting Shu
Multimodal Foundation ModelMedical ImagingWireless Foundation ModelsMedical Visual Question Answering
AskPDF
Jun 15, 2026cs.LG

Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to datasets under distribution shift remains underexplored. This work systematically evaluates FM-based representations on a suite of computational pathology tasks across two real-world commercial cohorts, IH-BC and IH-NSCLC, drawn from the licensed in-house (IH) oncology dataset. The analysis focuses on two modalities, whole-slide images and transcriptomic profiles, drawn from the IH multimodal data. We first benchmark unimodal probing performance across five FMs on eight downstream classification tasks, and find that image and omics representations carry complementary predictive signals. Then we investigate whether multimodal fusion can yield additional gains over unimodal baselines by comparing three image-omics fusion strategies built on paired representations. The trustworthiness of selected unimodal and multimodal pipelines is further assessed through conformal prediction. Our results show that FM representations achieve competitive performance on out-of-distribution data and that multimodal fusion helps mainly when no single modality dominates the signal. Conformal prediction reveals that in the majority of cases where a point prediction fails, the true diagnosis remains recoverable within the prediction set, reinforcing the value of uncertainty-aware inference for clinical support.
Jingyu Hu, Giuseppe Tripodi, Reed Naidoo +2
Pathology Foundation ModelsMultimodal Clinical DataMultimodal Foundation ModelMulti-Omic FrameworkFoundation ModelMultimodal Learning
AskPDF
Feb 24, 2026cs.CV

Federated Learning for Cross-Modality Medical Image Segmentation via Augmentation-Driven Generalization

Purpose: Developing generalizable medical image segmentation models is challenging because imaging data are distributed across institutions and differ in modality and acquisition protocol. Federated learning (FL) enables collaborative training without centralizing raw medical images, but cross-modality domain shifts between computed tomography (CT) and magnetic resonance imaging (MRI) can substantially reduce model performance. This study investigates augmentation-driven cross-modality FL for abdominal organ and whole-heart segmentation. Methods: We evaluate convolution-based spatial augmentation, frequency-domain argumentation, domain-specific normalization, and global intensity nonlinear (GIN) augmentation for multimodal segmentation. Abdominal organ segmentation and whole-heart segmentation are first evaluated using a 2D U-Net framework. For whole-heart segmentation, we additionally perform native 3D experiments using a self-configuring nnU-Net architecture on the CARE-WHS 2026 dataset, enabling evaluation of whether the observed cross-modality FL behavior persists when moving from slice-based 2D segmentation to volumetric 3D segmentation. Results: GIN provides the most consistent cross-modality performance among the evaluated approaches in the original 2D experiments. For pancreas segmentation, the Dice similarity coefficient (DSC) improved from 0.073 to 0.437 when CT data were incorporated through federated cross-modality training. In 3D whole-heart segmentation, FedGIN improved mean DSC over FedAvg from 0.8696 to 0.8901 on the unseen CT center and from 0.7160 to 0.7956 on the unseen MRI center. Relative to centralized GIN training, FedGIN retained 92.4% of performance on unseen CT data and achieved comparable performance on unseen MRI data (0.7956 versus 0.7937).
Sachin Dudda Nagaraju, Ashkan Moradi, Bendik Skarre Abrahamsen +1
Medical Image SegmentationWhole-Heart SegmentationCross-ModalityFederated LearningMagnetic Resonance ImagingAugmentation
AskPDF