SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging

Shenhav Nadir; Meir Yossef Levi; Eyal Gofer; Guy Gilboa

cs.CVAug 18, 2026

SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging

Authors: Shenhav Nadir, Meir Yossef Levi, Eyal Gofer, Guy Gilboa

Organizations: Viterbi Faculty of Electrical and Computer Engineering, Technion - Israel Institute of Technology

Abstract

Despite the rapid progress of deep neural networks in visual recognition, their adoption in high-risk medical applications remains limited due to reliability and robustness concerns. Models may exploit spurious correlations, particularly in medical imaging, where devices or treatment artifacts often co-occur with pathology. In small or imbalanced datasets, such cues further reduce worst-group performance and undermine clinical trust. To solve these issues, two major challenges should be addressed: identifying dataset-specific spurious cues, which typically require domain knowledge, and mitigating reliance on them. To tackle both, we propose SpurCon, a lightweight framework based on a novel supervised contrastive loss formulation that leverages available metadata and predicted spurious labels to enhance robustness. We introduce a fast few-shot procedure, without network training, to estimate spurious labels using a small number of expert-annotated samples. We then propose a weighted supervised contrastive objective, WtSupCon, that reshapes the representation geometry by assigning sample-specific weights that depend on the [pathology, spurious, metadata] combination. For example, the highest weight is assigned to samples that differ only in their spurious label. This yields highly similar representations for images with the same metadata and pathology, differing only in the predicted spurious label. Our method operates on pretrained image encoders (such as BiomedCLIP) and trains only a lightweight projection head. We evaluate SpurCon on a synthetic setting and on Waterbirds, CheXpert, a chest X-ray classification dataset, and ISIC 2020, a skin cancer classification dataset. Our approach delivers the best spurious-mitigation performance, balancing well worst-group and overall accuracy on multiple datasets.

Figures & tables

Figure 1: SpurCon overview. Given an anchor, we define anchor-sample pairs using [class, spurious, ID]. Pairs with different classes ( e.g . , pathology) are pushed apart (negatives). Positive pairs are pulled together, with decreasing strength: (1) same ID & different spurious cue, (2) same ID & same spurious cue, (3) same class only.

Figure 2: Pick-and-Predict overview. Illustration of reliable and annotation-efficient few-shot spurious-label estimation, shown for binary class and spurious attributes. (a) Given known training class labels, an expert annotates $N$ samples per spurious category, evenly distributed across classes. Their embeddings are averaged to form a prototype for each spurious label. (b) Each remaining training or validation sample is encoded and assigned the spurious label whose prototype has the highest cosine similarity.

Set	Same ID	Same spurious label	Same class
1	✓	✗	✓
2	✓	✓	✓
3	✗	-	✓

Table 1: Split into Sets. Set definitions ordered by importance in the final loss. The highest-weight set shares ID and class with the anchor but differs in spurious label.

Figure 3: Examples of spurious attributes across datasets. (a) Toy example, (b) Waterbirds, (c) ISIC 2020, (d) CheXpert. In (a) and (b), the background serves as the spurious attribute. In (c) and (d), the spurious attribute is circled.

	Waterbirds
	WG	Avg.	Adj. Avg.	AUC
Baseline	58.1	89.0	97.4	92.5
Baseline-WT	79.4	88.6	96.4	94.0
JTT [ 8 ]	84.5 $\pm$ 3.0	89.1 $\pm$ 2.6	89.7 $\pm$ 4.5	95.2 $\pm$ 1.2
DFR [ 6 ]	82.2 $\pm$ 1.1	87.2 $\pm$ 0.4	92.4 $\pm$ 0.7	97.8 $\pm$ 0.2
CA [ 21 ]	77.7 $\pm$ 4.8	86.1 $\pm$ 2.4	91.9 $\pm$ 3.1	93.1 $\pm$ 1.8

Table 2: Spurious mitigation results. Best results in bold . Across all datasets, SpurCon substantially outperforms prior methods in worst-group (WG) accuracy , demonstrating its effectiveness in mitigating spurious correlations.

	#Epochs	$\frac{\text{Time}}{\text{epoch}}$ (s) $\downarrow$	Total training time (s) $\downarrow$
JTT [ 8 ]	$10(\times 2)$	$137.1\!\pm\!5.3$	$2742\!\pm\!107$
DFR [ 6 ]	$75$	$59.9\!\pm\!0.8$	$4494\!\pm\!61$
CA [ 21 ]	$20$	$138.3\!\pm\!3.5$	$2767\!\pm\!69$
SpurCon (Ours)	$35$	$\bm{7.9\!\pm\!0.6}$	$\bm{497\!\pm\!36}$

Table 3: Training runtime comparison . SpurCon is considerably more computationally efficient than the evaluated prior methods. Measurements were taken on a single NVIDIA GeForce RTX 3090 GPU and 20 CPU cores. Best result in bold . Obtained on ISIC 2020.

Figure 4: Performance vs. $\#$ annotated samples. Metrics largely plateau with few samples. We select $\bf{N/2=10}$ to balance annotation efficiency and performance. Obtained on Waterbirds.

	baseline-WT	Original	Expert annotation mistakes		Estimated annotations mistakes
			$10\%$	$20\%$	$10\%$	$20\%$
WG	$58.6$	$65.0\pm 3.8$	$64.0\pm 4.7$	$58.9\pm 4.6$	$60.8\pm 3.8$	$54.7\pm 4.2$
Avg.	$74.8$	$80.3\pm 0.9$	$80.3\pm 1.3$	$79.2\pm 1.9$	$81.2\pm 1.9$	$80.0\pm 1.3$
Adj. Avg.	$74.4$	$79.9\pm 0.9$	$79.8\pm 1.3$	${78.7\pm 1.9}$	$80.7\pm 2.0$	${79.2\pm 1.3}$
AUC	$83.5$	$84.8\pm 0.4$	$84.8\pm 1.0$	$84.9\pm 1.1$	$84.8\pm 1.0$	$85.8\pm 0.5$

Table 4: Robustness analysis - Pick-and-Predict. Under moderate corruption, Pick-and-Predict remains above the baseline. Higher WG accuracy under corrupt expert annotations than corrupt estimated labels indicates the robustness of the estimation process. In each case,

p\%

of labels are corrupted within each

(\text{class},\text{spurious})

. Obtained on ISIC 2020.

	SupCon	WtSupCon variants (Ours)
		Ablation 1 $\alpha_{1}=\alpha_{2}=\alpha_{3}$	Ablation 2 $\alpha_{1}=\alpha_{2}>\alpha_{3}$	Ablation 3 $\alpha_{1}>\alpha_{2}=\alpha_{3}$	Ablation 4 $\alpha_{1}>\alpha_{2}>\alpha_{3}$
WG	$61.2\!\pm\!1.5$	$60.3\!\pm\!2.3$	$58.1\!\pm\!2.4$	$78.5\!\pm\!3.4$	$\bm{80.7\!\pm\!2.0}$
Avg.	$84.3\!\pm\!0.5$	$83.9\!\pm\!0.8$	$82.9\!\pm\!1.2$	$91.6\!\pm\!0.9$	$\bm{92.0\!\pm\!0.5}$
Adj. Avg.	$98.0\!\pm\!0.1$	$97.9\!\pm\!0.2$	$\bm{98.1\!\pm\!0.1}$	$97.0\!\pm\!0.4$	$96.6\!\pm\!0.3$
AUC	$93.6\!\pm\!0.5$	$93.3\!\pm\!0.6$	$92.7\!\pm\!0.8$	$97.5\!\pm\!0.5$	$\bm{97.8\!\pm\!0.1}$

Table 5: WtSupCon weights relations. Evaluation against standard SupCon and multiple WtSupCon weight settings on the toy dataset. Ablation 4 achieves the highest worst-group (WG) accuracy, validating our choice of this weighting scheme for SpurCon. Results obtained using true spurious labels. Best results in bold.

Table 6: (a): Top configurations results.

\bf{(\alpha_{1},\alpha_{2},\alpha_{3})=(4,2,1)}

achieves the highest rank according to

\mu-\sigma

values. Obtained on ISIC 2020. (b): Evaluated hyperparameter values.

Panel of Figure 5 — Figure 5: Variation across combinations. Performance across the 15 weight combinations. Each segment spans the worst-to-best result. Low variance across weight combinations indicates robustness to weight selection. Obtained on ISIC 2020.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Toy dataset example. Images (a) and (b) belong to the class $y=one\>hole=0$ and share the same ID, orange, while differing in their spurious background attribute. Images (c) and (d) belong to the class $y=two\>holes=1$ and share the same ID, pink, while also differing in the background. The examples illustrate variations in template, including hole size and location, as well as scale, jitter, and random background objects.

Split	$\bm{(0,0)}$	$\bm{(0,1)}$	$\bm{(1,0)}$	$\bm{(1,1)}$	Total
Train	17527	3427	254	135	21343
Validation	4156	771	53	24	5004
Test	5605	625	88	27	6345
Total	27288	4823	395	186	32692

Appendix

Table 7: ISIC 2020 distribution. Each group is defined as

(y,s)

, where

y

is the class label and

s

is the spurious label. Predicted spurious labels are used for the training and validation sets, true spurious labels are used for the test set. Note the severe class imbalance, with substantially more benign than malignant samples.

Split	$\bm{(0,0)}$	$\bm{(0,1)}$	$\bm{(1,0)}$	$\bm{(1,1)}$	Total
Train	13733	277	246	13147	27403
Validation	3415	67	61	3254	6797
Test	416	416	803	844	2479
Total	17564	760	1110	17245	36679

Appendix

Table 8: CheXpert distribution. Each group is defined as

(y,s)

, where

y

is the class label and

s

is the spurious label. Predicted spurious labels are used for the training and validation sets, true spurious labels are used for the test set.

Figure 7: Comparison of images from the same patient. All four chest X-ray images were acquired from the same patient. (a) Pneumothorax with a chest tube (support device), marked by yellow arrows, (b-c) Pneumothorax without a chest tube, (d) neither Pneumothorax nor a chest tube. Pneumothorax regions are outlined in red. Visual characteristics unrelated to the pathology or medical device remain largely consistent across the images.

Explore similar work

SparseContrast: Dynamic Sparse Attention for Efficient and Accurate Contrastive Learning in Medical Imaging
Apr 27, 2026Paarth Prasad, Ruchika MalhotraContrastive Learning Chest X-Ray Classification
Department of Software Engineering, Delhi Technological University
MedXplore: Towards Reliable and Unbiased Generalized Category Discovery in Medical Imaging
Jul 30, 2026Jianwei He, Kailin Lyu, Junhao Dong +6Medical Image Representation Learning Generalized Category Discovery
Institute of Automation, Chinese Academy of Sciences Beijing, China · Nanyang Technological University Singapore, Singapore
Mitigating Shortcut Learning via Feature Disentanglement in Medical Imaging: A Benchmark Study
Feb 17, 2026Sarah Müller, Philipp BerensShortcut Learning Disentangled Representation Learning
Hertie Institute for AI in Brain Health, Faculty of Medicine, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany

Apr 27, 2026J/K move · Enter open · S save

SparseContrast: Dynamic Sparse Attention for Efficient and Accurate Contrastive Learning in Medical Imaging

Paarth Prasad, Ruchika Malhotra

Department of Software Engineering, Delhi Technological University

Contrastive Learning Chest X-Ray Classification Sparse Attention Medical Image Representation Learning

Open paper AskPDF arXiv

We propose SparseContrast, a new framework that merges dynamic sparse attention with contrastive learning for medical imaging, with a focus on chest X-ray disease detection in low-data settings. Traditional contrastive learning methods rely on dense attention mechanisms, which are computationally expensive and often process redundant regions in medical images. To resolve this, SparseContrast introduces a sparse attention mechanism that selectively concentrates on diagnostically pertinent areas, markedly decreasing computational burden without compromising accuracy. The framework adaptively trims attention maps in the training phase, directed by a compact saliency predictor which concurrently optimizes sparsity and feature quality. This method not only speeds up training and inference by as much as 40% relative to dense attention benchmarks but also boosts diagnostic accuracy by focusing on areas of clinical importance. Moreover, the approach remains indifferent to the selection of backbone architecture, which permits its application to both convolutional and transformer-based models. Experiments show SparseContrast attains comparable or better performance in disease identification tasks with greater efficiency relative to current approaches. The proposed framework delivers a practical approach for implementing contrastive learning in medical imaging settings with limited resources, where computational efficiency and diagnostic accuracy are paramount.

Abstract

Explore similar work

SparseContrast: Dynamic Sparse Attention for Efficient and Accurate Contrastive Learning in Medical Imaging

MedXplore: Towards Reliable and Unbiased Generalized Category Discovery in Medical Imaging

Mitigating Shortcut Learning via Feature Disentanglement in Medical Imaging: A Benchmark Study