Despite the rapid progress of deep neural networks in visual recognition, their adoption in high-risk medical applications remains limited due to reliability and robustness concerns. Models may exploit spurious correlations, particularly in medical imaging, where devices or treatment artifacts often co-occur with pathology. In small or imbalanced datasets, such cues further reduce worst-group performance and undermine clinical trust. To solve these issues, two major challenges should be addressed: identifying dataset-specific spurious cues, which typically require domain knowledge, and mitigating reliance on them. To tackle both, we propose SpurCon, a lightweight framework based on a novel supervised contrastive loss formulation that leverages available metadata and predicted spurious labels to enhance robustness. We introduce a fast few-shot procedure, without network training, to estimate spurious labels using a small number of expert-annotated samples. We then propose a weighted supervised contrastive objective, WtSupCon, that reshapes the representation geometry by assigning sample-specific weights that depend on the [pathology, spurious, metadata] combination. For example, the highest weight is assigned to samples that differ only in their spurious label. This yields highly similar representations for images with the same metadata and pathology, differing only in the predicted spurious label. Our method operates on pretrained image encoders (such as BiomedCLIP) and trains only a lightweight projection head. We evaluate SpurCon on a synthetic setting and on Waterbirds, CheXpert, a chest X-ray classification dataset, and ISIC 2020, a skin cancer classification dataset. Our approach delivers the best spurious-mitigation performance, balancing well worst-group and overall accuracy on multiple datasets.
Figures & tables
Figure 1: SpurCon overview. Given an anchor, we define anchor-sample pairs using [class, spurious, ID]. Pairs with different classes ( e.g . , pathology) are pushed apart (negatives). Positive pairs are pulled together, with decreasing strength: (1) same ID & different spurious cue, (2) same ID & same spurious cue, (3) same class only.
Figure 2: Pick-and-Predict overview. Illustration of reliable and annotation-efficient few-shot spurious-label estimation, shown for binary class and spurious attributes. (a) Given known training class labels, an expert annotates N samples per spurious category, evenly distributed across classes. Their embeddings are averaged to form a prototype for each spurious label. (b) Each remaining training or validation sample is encoded and assigned the spurious label whose prototype has the highest cosine similarity.
Set
Same ID
Same spurious label
Same class
1
✓
✗
✓
2
✓
✓
✓
3
✗
-
✓
Table 1: Split into Sets. Set definitions ordered by importance in the final loss. The highest-weight set shares ID and class with the anchor but differs in spurious label.
Figure 3: Examples of spurious attributes across datasets. (a) Toy example, (b) Waterbirds, (c) ISIC 2020, (d) CheXpert. In (a) and (b), the background serves as the spurious attribute. In (c) and (d), the spurious attribute is circled.
Waterbirds
WG
Avg.
Adj. Avg.
AUC
Baseline
58.1
89.0
97.4
92.5
Baseline-WT
79.4
88.6
96.4
94.0
JTT [ 8 ]
84.5 ± 3.0
89.1 ± 2.6
89.7 ± 4.5
95.2 ± 1.2
DFR [ 6 ]
82.2 ± 1.1
87.2 ± 0.4
92.4 ± 0.7
97.8 ± 0.2
CA [ 21 ]
77.7 ± 4.8
86.1 ± 2.4
91.9 ± 3.1
93.1 ± 1.8
Table 2: Spurious mitigation results. Best results in bold . Across all datasets, SpurCon substantially outperforms prior methods in worst-group (WG) accuracy , demonstrating its effectiveness in mitigating spurious correlations.
#Epochs
epochTime (s) ↓
Total training time (s) ↓
JTT [ 8 ]
10(×2)
137.1±5.3
2742±107
DFR [ 6 ]
75
59.9±0.8
4494±61
CA [ 21 ]
20
138.3±3.5
2767±69
SpurCon (Ours)
35
7.9±0.6
497±36
Table 3: Training runtime comparison . SpurCon is considerably more computationally efficient than the evaluated prior methods. Measurements were taken on a single NVIDIA GeForce RTX 3090 GPU and 20 CPU cores. Best result in bold . Obtained on ISIC 2020.
Figure 4: Performance vs. # annotated samples. Metrics largely plateau with few samples. We select N/2=10 to balance annotation efficiency and performance. Obtained on Waterbirds.
baseline-WT
Original
Expert annotation mistakes
Estimated annotations mistakes
10%
20%
10%
20%
WG
58.6
65.0±3.8
64.0±4.7
58.9±4.6
60.8±3.8
54.7±4.2
Avg.
74.8
80.3±0.9
80.3±1.3
79.2±1.9
81.2±1.9
80.0±1.3
Adj. Avg.
74.4
79.9±0.9
79.8±1.3
78.7±1.9
80.7±2.0
79.2±1.3
AUC
83.5
84.8±0.4
84.8±1.0
84.9±1.1
84.8±1.0
85.8±0.5
Table 4: Robustness analysis - Pick-and-Predict. Under moderate corruption, Pick-and-Predict remains above the baseline. Higher WG accuracy under corrupt expert annotations than corrupt estimated labels indicates the robustness of the estimation process. In each case, p% of labels are corrupted within each (class,spurious) . Obtained on ISIC 2020.
SupCon
WtSupCon variants (Ours)
Ablation 1 α1=α2=α3
Ablation 2 α1=α2>α3
Ablation 3 α1>α2=α3
Ablation 4 α1>α2>α3
WG
61.2±1.5
60.3±2.3
58.1±2.4
78.5±3.4
80.7±2.0
Avg.
84.3±0.5
83.9±0.8
82.9±1.2
91.6±0.9
92.0±0.5
Adj. Avg.
98.0±0.1
97.9±0.2
98.1±0.1
97.0±0.4
96.6±0.3
AUC
93.6±0.5
93.3±0.6
92.7±0.8
97.5±0.5
97.8±0.1
Table 5: WtSupCon weights relations. Evaluation against standard SupCon and multiple WtSupCon weight settings on the toy dataset. Ablation 4 achieves the highest worst-group (WG) accuracy, validating our choice of this weighting scheme for SpurCon. Results obtained using true spurious labels. Best results in bold.
Table 6: (a): Top configurations results. (α1,α2,α3)=(4,2,1) achieves the highest rank according to μ−σ values. Obtained on ISIC 2020. (b): Evaluated hyperparameter values.
Figure 5: Variation across combinations. Performance across the 15 weight combinations. Each segment spans the worst-to-best result. Low variance across weight combinations indicates robustness to weight selection. Obtained on ISIC 2020.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Toy dataset example. Images (a) and (b) belong to the class y=onehole=0 and share the same ID, orange, while differing in their spurious background attribute. Images (c) and (d) belong to the class y=twoholes=1 and share the same ID, pink, while also differing in the background. The examples illustrate variations in template, including hole size and location, as well as scale, jitter, and random background objects.
Split
(0,0)
(0,1)
(1,0)
(1,1)
Total
Train
17527
3427
254
135
21343
Validation
4156
771
53
24
5004
Test
5605
625
88
27
6345
Total
27288
4823
395
186
32692
Appendix
Table 7: ISIC 2020 distribution. Each group is defined as (y,s) , where y is the class label and s is the spurious label. Predicted spurious labels are used for the training and validation sets, true spurious labels are used for the test set. Note the severe class imbalance, with substantially more benign than malignant samples.
Split
(0,0)
(0,1)
(1,0)
(1,1)
Total
Train
13733
277
246
13147
27403
Validation
3415
67
61
3254
6797
Test
416
416
803
844
2479
Total
17564
760
1110
17245
36679
Appendix
Table 8: CheXpert distribution. Each group is defined as (y,s) , where y is the class label and s is the spurious label. Predicted spurious labels are used for the training and validation sets, true spurious labels are used for the test set.
Figure 7: Comparison of images from the same patient. All four chest X-ray images were acquired from the same patient. (a) Pneumothorax with a chest tube (support device), marked by yellow arrows, (b-c) Pneumothorax without a chest tube, (d) neither Pneumothorax nor a chest tube. Pneumothorax regions are outlined in red. Visual characteristics unrelated to the pathology or medical device remain largely consistent across the images.
We propose SparseContrast, a new framework that merges dynamic sparse attention with contrastive learning for medical imaging, with a focus on chest X-ray disease detection in low-data settings. Traditional contrastive learning methods rely on dense attention mechanisms, which are computationally expensive and often process redundant regions in medical images. To resolve this, SparseContrast introduces a sparse attention mechanism that selectively concentrates on diagnostically pertinent areas, markedly decreasing computational burden without compromising accuracy. The framework adaptively trims attention maps in the training phase, directed by a compact saliency predictor which concurrently optimizes sparsity and feature quality. This method not only speeds up training and inference by as much as 40% relative to dense attention benchmarks but also boosts diagnostic accuracy by focusing on areas of clinical importance. Moreover, the approach remains indifferent to the selection of backbone architecture, which permits its application to both convolutional and transformer-based models. Experiments show SparseContrast attains comparable or better performance in disease identification tasks with greater efficiency relative to current approaches. The proposed framework delivers a practical approach for implementing contrastive learning in medical imaging settings with limited resources, where computational efficiency and diagnostic accuracy are paramount.
Paarth Prasad, Ruchika Malhotra
Department of Software Engineering, Delhi Technological University
Deep learning has shown strong potential in medical image analysis, but most existing methods rely on large-scale annotations and a closed-world assumption that rarely holds in clinical practice. Although Generalized Category Discovery (GCD) has advanced rapidly on natural images, it remains underexplored in medical imaging. To address this issue, we propose MedXplore, a unified framework for reliable and unbiased medical GCD, optimizing from both perceptual and decision levels. Specifically, at the perceptual level, taking a frequency domain perspective, Frequency-SNR Adaptive Attention and Consistency (FAAC) performs learnable full-spectrum filtering and global-local energy contrast activation to not only highlight local abnormal signals relative to the global context, but also provide reliable semantic anchors for patch consistency learning. At the decision level, Adaptive Cosine-Angular Margin (ACAM) adjusts angular margins using semantic difficulty and feature confidence to balance intra-class compactness and inter-class separability. Together, the two modules improve lesion-sensitive representation learning and mitigate old-class bias. Experiments on multiple benchmarks show an average 8.5% gain in All accuracy over the strongest competing methods. On Kvasir, MedXplore reduces false-old errors from 14.50% to 0.80%, demonstrating strong robustness under severe old-new ambiguity.
Jianwei He, Kailin Lyu, Junhao Dong +6
Institute of Automation, Chinese Academy of Sciences Beijing, China · Nanyang Technological University Singapore, Singapore
Although deep learning models in medical imaging often achieve excellent classification performance, they can rely on shortcut learning, exploiting spurious correlations or confounding factors that are not causally related to the target task. This poses risks in clinical settings, where models must generalize across institutions, populations, and acquisition conditions. Feature disentanglement is a promising approach to mitigate shortcut learning by separating task-relevant information from confounder-related features in latent representations. In this study, we systematically evaluated feature disentanglement methods for mitigating shortcuts in medical imaging, including adversarial learning and latent space splitting based on dependence minimization. We assessed classification performance and disentanglement quality using latent space analyses across one artificial and two medical datasets with natural and synthetic confounders. We also examined robustness under varying levels of confounding and compared computational efficiency across methods. We found that shortcut mitigation methods improved classification performance under strong spurious correlations during training. Latent space analyses revealed differences in representation quality not captured by classification metrics, highlighting the strengths and limitations of each method. Model reliance on shortcuts depended on the degree of confounding in the training data. The best-performing models combine data-centric rebalancing with model-centric disentanglement, achieving stronger and more robust shortcut mitigation than rebalancing alone while maintaining similar computational efficiency. The project code is publicly available at https://github.com/berenslab/medical-shortcut-mitigation.
Sarah Müller, Philipp Berens
Hertie Institute for AI in Brain Health, Faculty of Medicine, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany