Organizations: Department of Radiology & Faculty of Medical Imaging Technology, Ruijin Hospital, College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China · Clinical Neuroscience Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China · School of Data Science, Fudan University, Shanghai 200433, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai 200433, China · Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan 250031, China · Department of Clinical Psychology, School of Medical Psychology, Air Force Medical University, Xi’an 710032, Shaanxi, China
Accurately isolating disease-related features from confounding covariates (e.g., age, gender, site) and individual variations remains a fundamental challenge in medical image classification. Traditional regression-based approaches may ignore non-linear relations between image features and true covariates. To overcome this issue, we present a generalized Medical Imaging Disentanglement Learning (MedIDL) framework. MedIDL maps image features into three mutually orthogonal latent spaces through specialized disentanglement heads: a disease classification head guided by a supervised loss, a covariate-alignment head constrained by cross-subject similarity matching, and a Gaussian head absorbing individual variations. We evaluated our framework across 7 datasets encompassing diverse imaging modalities. MedIDL outperforms state-of-the-art supervised and self-supervised classification methods in accuracy across all datasets. Association analyses demonstrate that MedIDL successfully isolates target-specific latent representations. Gradient-based interpretability mappings localize pathognomonic patterns aligning with established clinical literature.
Figures & tables
Fig. 1: Overview of the datasets, covariate association, and advantage of the proposed framework. (a) illustrates the dataset description, showing the sample sizes across diverse modalities. (b) visualizes the most significantly associated biomarker(s) for a covariate (e.g. age or scanner site) within the HCs of a dataset. The image biomarkers are functional/structural connectivity for fMRI/DTI, intensity for sMRI and X-ray. The p -values were false discovery rate (FDR)-corrected in each dataset. (c) conceptually illustrates the latent representation spaces, highlighting that conventional methods typically enforce linear relationships between the extracted covariate features and true covariates, whereas our framework allows for non-linear manifold alignment.
Fig. 2: Overall architecture of the proposed disentangled representation learning framework. For any given input modality (e.g., 3D images, brain graphs, or 2D images), the data is processed by an encoder and subsequently projected into three separate latent spaces via dedicated heads. The disease head extracts disease-specific features ( ziD ) optimized for separating patients from HCs. The individual head captures normalized, individual-specific latent variations ( ziI ). The covariate head extracts demographic and site-related features ( ziC ), which are supervised by aligning the channel-wise subject similarity matrices with the corresponding ground truth similarity matrices from the covariates (e.g., age, gender, and site). Finally, the three disentangled components are concatenated and fed into a decoder to reconstruct the original input.
TABLE II: Classification performance on 2D/3D vision tasks.
Fig. 3: Quantitative validation of feature disentanglement using Kendall’s Tau rank correlation across seven datasets. Each row represents a ground-truth covariate (age, gender, or site) for evaluating the correlation. A dot within a subplot denotes the mean correlation over all the subjects in a test set, where the correlation for a subject measures the order consistency between similarities from this subject to all the others measured by a distinct feature (by color) and similarities measured by the true covariate.
Fig. 4: Comprehensive ablation and sensitivity analysis of the MedIDL framework across diverse imaging modalities. (Row 1) Impact of the unified latent embedding dimension ( d ) on classification accuracy. Shaded regions indicate standard deviations. The dashed line denotes the top competing method. (Row 2) Robustness validation across various modality-aware backbones. The dashed line denotes the top competing method. (Row 3) Component-wise ablation of the tri-branch disentanglement module. (Row 4) Joint hyperparameter sensitivity heatmaps for the covariate alignment weight ( λ1 ) and latent reconstruction weight ( λ2 ), suggesting a stable, broad high-accuracy zone (darker red regions). (Row 5) Superiority of our implicit latent disentanglement (solid bars) over traditional input-level de-confounding via linear regression. Dark hatched bars represent explicit de-confounding (regressing out covariates prior to classification), while light hatched bars denote classification without covariate removal.
Fig. 5: Quantitative validation of biomarker importance via counterfactual perturbation analysis across seven datasets. A plot depicts the degradation trajectories of the model’s mean prediction probability as input features are progressively perturbed/masked (replaced by means from HCs) from the patients in the test sets. Solid red lines represent the targeted perturbation of the “top important features” identified by our framework. Dashed blue lines with shades serve as the baseline representing the mean and standard deviation from 5 “random perturbations”. To accommodate the distinct spatial topologies of different modalities, the x -axis denotes the number of ROIs perturbed for fMRI and DTI, 3D voxels masked for sMRI, or 2D pixels masked for X-ray.
Fig. 6: Saliency maps identified by the proposed framework across seven datasets. (Top): For graph-based brain networks, the framework maps the most discriminative ROIs in the MNI space, where the marker size scales proportionally with the node’s importance score. (Bottom): For voxel/pixel-based datasets, gradient-based feature attribution maps localize structural and radiological anomalies. On ADNI (sMRI), the gradient is with respect to T1 intensity after affine registration. Hence, positive values in the cortical region indicate gray matter atrophy; negative values in the ventricle indicate enlarged ventricle. On Lung (X-ray), red rectangles are pulmonary lesions annotated by a clinical expert.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Dataset & Modality
Class
Sites
Subjects
Gender (F/M)
Age ( mean±std )
Scanner Hardware
Acquisition Parameters
ADHD-200 ( fMRI )
ADHD HC
4
275 205
79/196 105/100
11.5 ± 2.8 12.1 ± 3.2
Siemens/Philips 3T
TR = 2000/2500 ms, TE = 15–30 ms, FA = 75 ∘ /90 ∘ , voxel size ≈ 3 mm 3
Accurate medical image segmentation is fundamental to precision medicine, yet robust delineation remains challenging under heterogeneous appearances, ambiguous boundaries, and large anatomical variability. Similar intensity and texture patterns between targets and surrounding tissues often lead to blurred activations and unreliable separation. We attribute these failures to representation collapse during encoding and insufficient fine grained multi scale decoding. To address these issues, we propose Med DisSeg, a dispersion driven medical image segmentation framework that jointly improves representation learning and anatomical delineation. Med DisSeg combines a lightweight Dispersive Loss with adaptive attention for fine grained structure segmentation. The Dispersive Loss enlarges inter sample margins by treating in batch hidden representations as negative pairs, producing well dispersed and boundary aware embeddings with negligible overhead. Based on these enhanced representations, the encoder strengthens structure sensitive responses, while the decoder performs adaptive multi scale calibration to preserve complementary local texture and global shape information. Extensive experiments on five datasets spanning three imaging modalities demonstrate consistent state of the art performance. Moreover, Med DisSeg achieves competitive results on multi organ CT segmentation, supporting its robustness and cross task applicability.
Zhiquan Chen, Haitao Wang, Guowei Zou +1
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China
Although deep learning models in medical imaging often achieve excellent classification performance, they can rely on shortcut learning, exploiting spurious correlations or confounding factors that are not causally related to the target task. This poses risks in clinical settings, where models must generalize across institutions, populations, and acquisition conditions. Feature disentanglement is a promising approach to mitigate shortcut learning by separating task-relevant information from confounder-related features in latent representations. In this study, we systematically evaluated feature disentanglement methods for mitigating shortcuts in medical imaging, including adversarial learning and latent space splitting based on dependence minimization. We assessed classification performance and disentanglement quality using latent space analyses across one artificial and two medical datasets with natural and synthetic confounders. We also examined robustness under varying levels of confounding and compared computational efficiency across methods. We found that shortcut mitigation methods improved classification performance under strong spurious correlations during training. Latent space analyses revealed differences in representation quality not captured by classification metrics, highlighting the strengths and limitations of each method. Model reliance on shortcuts depended on the degree of confounding in the training data. The best-performing models combine data-centric rebalancing with model-centric disentanglement, achieving stronger and more robust shortcut mitigation than rebalancing alone while maintaining similar computational efficiency. The project code is publicly available at https://github.com/berenslab/medical-shortcut-mitigation.
Sarah Müller, Philipp Berens
Hertie Institute for AI in Brain Health, Faculty of Medicine, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany
Deep learning has shown strong potential in medical image analysis, but most existing methods rely on large-scale annotations and a closed-world assumption that rarely holds in clinical practice. Although Generalized Category Discovery (GCD) has advanced rapidly on natural images, it remains underexplored in medical imaging. To address this issue, we propose MedXplore, a unified framework for reliable and unbiased medical GCD, optimizing from both perceptual and decision levels. Specifically, at the perceptual level, taking a frequency domain perspective, Frequency-SNR Adaptive Attention and Consistency (FAAC) performs learnable full-spectrum filtering and global-local energy contrast activation to not only highlight local abnormal signals relative to the global context, but also provide reliable semantic anchors for patch consistency learning. At the decision level, Adaptive Cosine-Angular Margin (ACAM) adjusts angular margins using semantic difficulty and feature confidence to balance intra-class compactness and inter-class separability. Together, the two modules improve lesion-sensitive representation learning and mitigate old-class bias. Experiments on multiple benchmarks show an average 8.5% gain in All accuracy over the strongest competing methods. On Kvasir, MedXplore reduces false-old errors from 14.50% to 0.80%, demonstrating strong robustness under severe old-new ambiguity.
Jianwei He, Kailin Lyu, Junhao Dong +6
Institute of Automation, Chinese Academy of Sciences Beijing, China · Nanyang Technological University Singapore, Singapore