HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations

Aryan Yazdan Parast; Khawar Islam; Soyoun Won; Basim Azam; Naveed Akhtar

cs.CVMar 31, 2026

HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations

Authors: Aryan Yazdan Parast, Khawar Islam, Soyoun Won, Basim Azam, Naveed Akhtar

Organizations: The University of Melbourne, Melbourne, Australia

Abstract

Deep neural networks often rely on spurious features to make predictions, which makes them brittle under distribution shift and on samples where the spurious correlation does not hold (e.g., minority-group examples). Recent studies have shown that, even in such settings, the feature extractor of an Empirical Risk Minimization (ERM)-trained model can learn rich and informative representations, and that much of the failure may be attributed to the classifier head. In particular, retraining a lightweight head while keeping the backbone frozen can substantially improve performance on shifted distributions and minority groups. Motivated by this observation, we propose a bilevel meta-learning method that performs augmentation directly in feature space to improve spurious correlation handling in the classifier head. Our method learns support-side feature edits such that, after a small number of inner-loop updates on the edited features, the classifier achieves lower loss on hard examples and improved worst-group performance. By operating at the backbone output rather than in pixel space or through end-to-end optimization, the method is highly efficient and stable, requiring only a few minutes of training on a single GPU. We further validate our method with CLIP-based visualizations, showing that the learned feature-space updates induce semantically meaningful shifts aligned with spurious attributes.

Figures & tables

Figure 1: Overview of HSFM. (a) Initial training samples are passed through a frozen ERM feature extractor to obtain the support embeddings. The linear head is first adapted on the support set through the inner loss, and the outer loss is then computed on the hard set to improve performance on difficult examples. (b) Flowchart of the training procedure

	Group Info	Waterbirds		CelebA		MetaShift		Dominoes
Method	train/val	Worst	Average	Worst	Average	Worst	Average	Worst	Average
DFR	✗/✓✓	$92.3_{\pm 0.2}$	$93.3_{\pm 0.5}$	$88.3_{\pm 1.1}$	$91.3_{\pm 0.3}$	$72.8_{\pm 0.6}$	$77.5_{\pm 0.6}$	$90.0_{\pm 0.4}$	$92.3_{\pm 0.2}$
Group DRO	✓/✓	$91.4_{\pm 1.1}$	$93.5_{\pm 0.3}$	$88.9_{\pm 2.3}$	$92.9_{\pm 0.2}$	$66.0_{\pm 3.8}$	$73.6_{\pm 2.1}$	-	-
LISA	✓/✓	$89.2_{\pm 0.6}$	$91.8_{\pm 0.3}$	$\boldsymbol{89.3_{\pm 1.1}}$	$92.4_{\pm 0.4}$	$59.8_{\pm 2.3}$	$70.0_{\pm 0.7}$	-	-
JTT	✗/✓	$86.7$	$93.3$	$81.1$	$88.0$	$64.6_{\pm 2.3}$	$74.4_{\pm 0.6}$	-	-
DaC	✗/✓	$92.3_{\pm 0.4}$	$\underline{\boldsymbol{95.3_{\pm 0.4}}}$	$81.9_{\pm 0.7}$	$91.4_{\pm 1.1}$	$78.3_{\pm 1.6}$	$79.3_{\pm 0.1}$	$89.2_{\pm 0.1}$	$92.2_{\pm 0.3}$

Table 1: Comparison of worst-group accuracy and average accuracy between our method and the baselines. We report the mean and standard deviation over three runs with different random seeds. The best result overall is shown in bold, while the best result among methods that do not use group labels during training is underlined.

	Waterbirds		CelebA		MetaShift		Dominoes
Method	Worst	Average	Worst	Average	Worst	Average	Worst	Average
Base (ERM)	$53.9$	$75.37$	$53.3$	$\mathbf{95.70}$	$69.7$	$69.78$	$46.1$	$70.20$
HSFM (ours)	$\mathbf{81.5}$	$\mathbf{83.02}$	$\mathbf{88.3}$	$90.44$	$\mathbf{69.9}$	$\mathbf{70.22}$	$\mathbf{73.3}$	$\mathbf{75.46}$

Table 2: Comparison of worst-group accuracy and average between HSFM and ERM on four benchmarks with Vit-b-16 as backbone.

Method	Stanford Cars	CUB-Birds	Oxford Flowers
ERM	83.98	75.01	91.07
DFR-ERM	82.74	72.90	95.98
DFR-Pretrained	32.75	48.33	87.71
HSFM-ERM	83.44	72.83	97.27
HSFM-Pretrained	85.10	72.94	97.10

Table 3: Comparison of top-1 classification accuracy on fine-grained classification benchmarks. The best result in each column is shown in bold.

Figure 2: Visualization of three variants for each sample: original image, the image generated by SD unCLIP from the initial embedding, and the image generated from the optimized embedding. In CelebA, the first two rows show non-blond males shifted toward non-blond females. The last two rows show the shift from blond females to blond males. In Waterbirds, the first two rows show landbirds shifted from land backgrounds to water backgrounds, while the last two rows show waterbirds shifted from water backgrounds to land backgrounds.

Table 6

Split	Train	Validation	Test
(landbird, land background)	3,498	467	2,255
(landbird, water background)	184	466	2,255
(waterbird, land background)	56	133	642
(waterbird, water background)	1,018	133	642

Table 5: The number of samples in the training, validation, and test sets for the Waterbirds dataset.

Split	Train	Validation	Test
(NonBlond, female)	12,426	8,535	9,767
(NonBlond, male)	11,841	8,276	7,535
(Blond, female)	22,880	2,874	2,480
(Blond, male)	1,387	182	180

Table 6: The number of samples in the training, validation, and test sets for the CelebA dataset.

Split	Train	Validation	Test
(Blond, female)	5,000	100	100
(Non-blond, male)	5,000	100	100
(Blond, male)	100	100	100
(Non-blond, female)	100	100	100

Table 7: The number of samples in the training, validation, and test sets for the CelebA* dataset.

Figure 5: DDB generated samples on Dominoes dataset

Split	Train	Validation	Test
(car, 0)	3,592	500	490
(car, 1)	398	499	490
(truck, 0)	398	499	490
(truck, 1)	3,592	500	490

Table 8: The number of samples in the training, validation, and test sets for the Dominoes dataset.

Split	Train	Validation	Test
(Cat, sofa)	231	0	0
(Cat, bed)	380	0	0
(Dog, bench)	145	0	0
(Dog, bike)	367	0	0
(Cat, shelf)	0	34	201
(Dog, shelf)	0	47	259

Table 9: The number of samples in the training, validation, and test sets for the MetaShift dataset.

Split	Train	Validation	Test
(Cat, sofa)	231	0	0
(Cat, bed)	380	0	0
(Dog, bench)	145	0	0
(Dog, bike)	367	0	0
(Cat, shelf)	0	34	201
(Dog, shelf)	0	47	259

Table 10: The number of samples in the training, validation, and test sets for the MetaShift dataset.

Hyperparameter	CelebA	Waterbirds	MetaShift	Dominoes
Support set size per class	1024	16	32	16
$T$	10	15	15	10
Inner learning rate	$5\times 10^{-5}$	$5\times 10^{-5}$	$5\times 10^{-5}$	$1\times 10^{-4}$
Outer learning rate	$1\times 10^{-1}$	$1$	$1\times 10^{-2}$	$1$
$K_{H}$	5	15	10	15
$K_{hard}$	256	64	64	256

Table 11: Hyperparameters used for HSFM with a ResNet backbone on each dataset.

Hyperparameter	CelebA	Waterbirds	MetaShift	Dominoes
Support set size per class	1024	128	64	128
$T$	10	10	10	10
Inner learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-5}$	$1\times 10^{-4}$
Outer learning rate	$1$	$1$	$1\times 10^{-2}$	$1$
$K_{H}$	15	15	10	15
$K_{Hard}$	256	32	8	256

Table 12: Hyperparameters used for HSFM with a ViT backbone on each dataset.

Hyperparameter	CelebA	Waterbirds
Support set size per class	32	128
$T$	10	10
Inner learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$
Outer learning rate	$1$	$1$
$K_{H}$	15	15
$K_{Hard}$	8	16

Table 13: Hyperparameters used for HSFM with a ConvNeXt backbone on each dataset.

Method	DFR	CnC	JTT	MaskTune	DaC	Ours
Time (min)	4	85	58	6.5	18.9	1.43

Table 14: Training time comparison on the Waterbirds dataset, excluding ERM training, measured on an NVIDIA A100 GPU. Runtimes for DFR, CnC, JTT, MaskTune, and DaC are reported from DaC [ 22 ] , and our runtime is measured in our implementation.

Dataset	Runtime (minutes)
CelebA	2.98
Waterbirds	1.43
Dominoes	0.26
MetaShift	0.27

Table 15: Total program runtime of HSFM on each dataset after ERM training, measured on a single NVIDIA A100 GPU.

Spurious Correlation Robustness

Spurious correlation robustness studies how to prevent models from relying on incidental features that fail under distribution shifts.

See all 61 papers

Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning
Jul 7, 2026Suraj Yadav, Anjaneya Sharma, Siddharth YadavContrastive Learning Group Distributionally Robust Optimization
Indraprastha Institute of Information Technology Delhi
Cumulative Meta-Learning from Active Learning Queries for Robustness to Spurious Correlations
May 20, 2026Kin Whye Chew, Jingxian WangMeta-Learning Active Learning
Department of Computer Science National University of Singapore Singapore, S119077
Shortcut Mitigation via Spurious-Positive Samples
May 13, 2026Phuong Quynh Le, Jörg Schlötterer, Sari Sadiya +2Neural Network Robustness Spurious Correlation Robustness
University of Marburg · Goethe University Frankfurt

Jul 7, 2026J/K move · Enter open · S save

Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

Suraj Yadav, Anjaneya Sharma, Siddharth Yadav

Indraprastha Institute of Information Technology Delhi

Contrastive Learning Group Distributionally Robust Optimization OOD Generalization Spurious Correlation Robustness Counterfactual Data Augmentation Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Deep neural networks trained with Empirical Risk Minimization (ERM) often fail under distribution shifts because they exploit spurious correlations between object labels and background context. Recent generative approaches address this issue by creating counterfactual images with altered contexts, but typically use these samples as standard data augmentation, leaving the model free to retain background-sensitive representations. We propose a two-stage framework that uses generative intervention to explicitly learn background-invariant visual representations. First, we isolate the foreground object using zero-shot segmentation and generate context-shifted variants with a structure-preserving diffusion model, preserving object identity while varying the surrounding environment. We then introduce Cross-Variant Self-Supervised Learning, where variants of the same object under different backgrounds form positive pairs in a contrastive objective. This encourages the encoder to align object-centric representations while suppressing background-specific cues. Then, we fine-tune the pretrained encoder using an ERM warm-up followed by GroupDRO with layer-wise learning rates. Experiments on distribution-shift benchmarks demonstrate best worst-group performance, achieving 92.5% on Waterbirds, 81.7% on MetaShift, and 87.4% on NICO++. Code: https://github.com/surajyadav-research/GRSSL

Abstract

Spurious Correlation Robustness

Explore similar work

Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

Cumulative Meta-Learning from Active Learning Queries for Robustness to Spurious Correlations

Shortcut Mitigation via Spurious-Positive Samples