HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations

Aryan Yazdan Parast; Khawar Islam; Soyoun Won; Basim Azam; Naveed Akhtar

cs.CVMar 31, 2026

HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations

Authors: Aryan Yazdan Parast, Khawar Islam, Soyoun Won, Basim Azam, Naveed Akhtar

Organizations: The University of Melbourne, Melbourne, Australia

Abstract

Deep neural networks often rely on spurious features to make predictions, which makes them brittle under distribution shift and on samples where the spurious correlation does not hold (e.g., minority-group examples). Recent studies have shown that, even in such settings, the feature extractor of an Empirical Risk Minimization (ERM)-trained model can learn rich and informative representations, and that much of the failure may be attributed to the classifier head. In particular, retraining a lightweight head while keeping the backbone frozen can substantially improve performance on shifted distributions and minority groups. Motivated by this observation, we propose a bilevel meta-learning method that performs augmentation directly in feature space to improve spurious correlation handling in the classifier head. Our method learns support-side feature edits such that, after a small number of inner-loop updates on the edited features, the classifier achieves lower loss on hard examples and improved worst-group performance. By operating at the backbone output rather than in pixel space or through end-to-end optimization, the method is highly efficient and stable, requiring only a few minutes of training on a single GPU. We further validate our method with CLIP-based visualizations, showing that the learned feature-space updates induce semantically meaningful shifts aligned with spurious attributes.

Figures & tables

Figure 1: Overview of HSFM. (a) Initial training samples are passed through a frozen ERM feature extractor to obtain the support embeddings. The linear head is first adapted on the support set through the inner loss, and the outer loss is then computed on the hard set to improve performance on difficult examples. (b) Flowchart of the training procedure

	Group Info	Waterbirds		CelebA		MetaShift		Dominoes
Method	train/val	Worst	Average	Worst	Average	Worst	Average	Worst	Average
DFR	✗/✓✓	$92.3_{\pm 0.2}$	$93.3_{\pm 0.5}$	$88.3_{\pm 1.1}$	$91.3_{\pm 0.3}$	$72.8_{\pm 0.6}$	$77.5_{\pm 0.6}$	$90.0_{\pm 0.4}$	$92.3_{\pm 0.2}$
Group DRO	✓/✓	$91.4_{\pm 1.1}$	$93.5_{\pm 0.3}$	$88.9_{\pm 2.3}$	$92.9_{\pm 0.2}$	$66.0_{\pm 3.8}$	$73.6_{\pm 2.1}$	-	-
LISA	✓/✓	$89.2_{\pm 0.6}$	$91.8_{\pm 0.3}$	$\boldsymbol{89.3_{\pm 1.1}}$	$92.4_{\pm 0.4}$	$59.8_{\pm 2.3}$	$70.0_{\pm 0.7}$	-	-
JTT	✗/✓	$86.7$	$93.3$	$81.1$	$88.0$	$64.6_{\pm 2.3}$	$74.4_{\pm 0.6}$	-	-
DaC	✗/✓	$92.3_{\pm 0.4}$	$\underline{\boldsymbol{95.3_{\pm 0.4}}}$	$81.9_{\pm 0.7}$	$91.4_{\pm 1.1}$	$78.3_{\pm 1.6}$	$79.3_{\pm 0.1}$	$89.2_{\pm 0.1}$	$92.2_{\pm 0.3}$

Table 1: Comparison of worst-group accuracy and average accuracy between our method and the baselines. We report the mean and standard deviation over three runs with different random seeds. The best result overall is shown in bold, while the best result among methods that do not use group labels during training is underlined.

	Waterbirds		CelebA		MetaShift		Dominoes
Method	Worst	Average	Worst	Average	Worst	Average	Worst	Average
Base (ERM)	$53.9$	$75.37$	$53.3$	$\mathbf{95.70}$	$69.7$	$69.78$	$46.1$	$70.20$
HSFM (ours)	$\mathbf{81.5}$	$\mathbf{83.02}$	$\mathbf{88.3}$	$90.44$	$\mathbf{69.9}$	$\mathbf{70.22}$	$\mathbf{73.3}$	$\mathbf{75.46}$

Table 2: Comparison of worst-group accuracy and average between HSFM and ERM on four benchmarks with Vit-b-16 as backbone.

Method	Stanford Cars	CUB-Birds	Oxford Flowers
ERM	83.98	75.01	91.07
DFR-ERM	82.74	72.90	95.98
DFR-Pretrained	32.75	48.33	87.71
HSFM-ERM	83.44	72.83	97.27
HSFM-Pretrained	85.10	72.94	97.10

Table 3: Comparison of top-1 classification accuracy on fine-grained classification benchmarks. The best result in each column is shown in bold.

Figure 2: Visualization of three variants for each sample: original image, the image generated by SD unCLIP from the initial embedding, and the image generated from the optimized embedding. In CelebA, the first two rows show non-blond males shifted toward non-blond females. The last two rows show the shift from blond females to blond males. In Waterbirds, the first two rows show landbirds shifted from land backgrounds to water backgrounds, while the last two rows show waterbirds shifted from water backgrounds to land backgrounds.

Table 6

Split	Train	Validation	Test
(landbird, land background)	3,498	467	2,255
(landbird, water background)	184	466	2,255
(waterbird, land background)	56	133	642
(waterbird, water background)	1,018	133	642

Table 5: The number of samples in the training, validation, and test sets for the Waterbirds dataset.

Split	Train	Validation	Test
(NonBlond, female)	12,426	8,535	9,767
(NonBlond, male)	11,841	8,276	7,535
(Blond, female)	22,880	2,874	2,480
(Blond, male)	1,387	182	180

Table 6: The number of samples in the training, validation, and test sets for the CelebA dataset.

Split	Train	Validation	Test
(Blond, female)	5,000	100	100
(Non-blond, male)	5,000	100	100
(Blond, male)	100	100	100
(Non-blond, female)	100	100	100

Table 7: The number of samples in the training, validation, and test sets for the CelebA* dataset.

Figure 5: DDB generated samples on Dominoes dataset

Split	Train	Validation	Test
(car, 0)	3,592	500	490
(car, 1)	398	499	490
(truck, 0)	398	499	490
(truck, 1)	3,592	500	490

Table 8: The number of samples in the training, validation, and test sets for the Dominoes dataset.

Split	Train	Validation	Test
(Cat, sofa)	231	0	0
(Cat, bed)	380	0	0
(Dog, bench)	145	0	0
(Dog, bike)	367	0	0
(Cat, shelf)	0	34	201
(Dog, shelf)	0	47	259

Table 9: The number of samples in the training, validation, and test sets for the MetaShift dataset.

Split	Train	Validation	Test
(Cat, sofa)	231	0	0
(Cat, bed)	380	0	0
(Dog, bench)	145	0	0
(Dog, bike)	367	0	0
(Cat, shelf)	0	34	201
(Dog, shelf)	0	47	259

Table 10: The number of samples in the training, validation, and test sets for the MetaShift dataset.

Hyperparameter	CelebA	Waterbirds	MetaShift	Dominoes
Support set size per class	1024	16	32	16
$T$	10	15	15	10
Inner learning rate	$5\times 10^{-5}$	$5\times 10^{-5}$	$5\times 10^{-5}$	$1\times 10^{-4}$
Outer learning rate	$1\times 10^{-1}$	$1$	$1\times 10^{-2}$	$1$
$K_{H}$	5	15	10	15
$K_{hard}$	256	64	64	256

Table 11: Hyperparameters used for HSFM with a ResNet backbone on each dataset.

Hyperparameter	CelebA	Waterbirds	MetaShift	Dominoes
Support set size per class	1024	128	64	128
$T$	10	10	10	10
Inner learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-5}$	$1\times 10^{-4}$
Outer learning rate	$1$	$1$	$1\times 10^{-2}$	$1$
$K_{H}$	15	15	10	15
$K_{Hard}$	256	32	8	256

Table 12: Hyperparameters used for HSFM with a ViT backbone on each dataset.

Hyperparameter	CelebA	Waterbirds
Support set size per class	32	128
$T$	10	10
Inner learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$
Outer learning rate	$1$	$1$
$K_{H}$	15	15
$K_{Hard}$	8	16

Table 13: Hyperparameters used for HSFM with a ConvNeXt backbone on each dataset.

Method	DFR	CnC	JTT	MaskTune	DaC	Ours
Time (min)	4	85	58	6.5	18.9	1.43

Table 14: Training time comparison on the Waterbirds dataset, excluding ERM training, measured on an NVIDIA A100 GPU. Runtimes for DFR, CnC, JTT, MaskTune, and DaC are reported from DaC [ 22 ] , and our runtime is measured in our implementation.

Dataset	Runtime (minutes)
CelebA	2.98
Waterbirds	1.43
Dominoes	0.26
MetaShift	0.27

Table 15: Total program runtime of HSFM on each dataset after ERM training, measured on a single NVIDIA A100 GPU.

Spurious Correlation Robustness

Spurious correlation robustness studies how to prevent models from relying on incidental features that fail under distribution shifts.

See all 61 papers

Jul 7, 2026

Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

Deep neural networks trained with Empirical Risk Minimization (ERM) often fail under distribution shifts because they exploit spurious correlations between object labels and background context. Recent generative approaches address this issue by creating counterfactual images with altered contexts, but typically use these samples as standard data augmentation, leaving the model free to retain background-sensitive representations. We propose a two-stage framework that uses generative intervention to explicitly learn background-invariant visual representations. First, we isolate the foreground object using zero-shot segmentation and generate context-shifted variants with a structure-preserving diffusion model, preserving object identity while varying the surrounding environment. We then introduce Cross-Variant Self-Supervised Learning, where variants of the same object under different backgrounds form positive pairs in a contrastive objective. This encourages the encoder to align object-centric representations while suppressing background-specific cues. Then, we fine-tune the pretrained encoder using an ERM warm-up followed by GroupDRO with layer-wise learning rates. Experiments on distribution-shift benchmarks demonstrate best worst-group performance, achieving 92.5% on Waterbirds, 81.7% on MetaShift, and 87.4% on NICO++. Code: https://github.com/surajyadav-research/GRSSL

Suraj Yadav, Anjaneya Sharma, Siddharth Yadav

Indraprastha Institute of Information Technology Delhi

Contrastive Learning Group Distributionally Robust Optimization OOD Generalization Spurious Correlation Robustness Counterfactual Data Augmentation Self-Supervised Visual Representation Learning

Ask PDF

May 20, 2026

Cumulative Meta-Learning from Active Learning Queries for Robustness to Spurious Correlations

Spurious correlations in real-world datasets cause machine learning models to rely on irrelevant patterns, undermining reliability, generalization, and fairness. Active learning offers a promising way to address this failure mode by querying informative samples that distinguish core features from spurious ones. However, standard active-learning methods simply append queried examples to the labeled set, effectively updating only the likelihood term. In deep learning regimes, the influence of these informative samples can be diluted by the larger labeled set and memorized by overparameterized models. We propose Cumulative Active Meta-Learning (CAML), an active-learning framework that uses queried examples to meta-learn the prior, or inductive bias, governing how the model adapts. CAML casts each active-learning round as a meta-learning task: the current labeled set serves as meta-train data for adaptation, while the newly queried batch serves as meta-test data for evaluating generalization. Unlike conventional meta-learning, which treats tasks as independent and identically distributed, CAML exploits the sequential dependence between active-learning rounds by maintaining a cumulative inductive bias that is progressively refined. Theoretically, we show that this cumulative formulation introduces interaction terms that couple earlier meta-learned inductive biases with later query-induced objectives, capturing dependencies absent from standard meta-learning. Empirically, CAML improves minority-group accuracy across spurious-correlation benchmarks and acquisition strategies, with gains of up to 27.8% on Dominoes, 29.9% on Waterbirds, 14.3% on SpuCo, and 24.0% on CivilComments.

Kin Whye Chew, Jingxian Wang

Department of Computer Science National University of Singapore Singapore, S119077

Meta-Learning Active Learning Spurious Correlation Robustness Spurious Correlation

Ask PDF

May 13, 2026

Shortcut Mitigation via Spurious-Positive Samples

Shortcut mitigation strategies commonly rely on training data annotations, group-balanced held-out data or the presence of all groups, i.e., all combinations of (spurious) attributes and classes, in the training data. However, these requirements are rarely met in practice. We instead propose a method for targeted model analysis to identify a small set of instances in which the model relies on spurious attributes. Using that set and following ``this feature should not be used for prediction'' reasoning, we identify highly relevant neurons in an intermediate layer and regularize their impact. This ensures that models learn to depend on informative features rather than being right for the wrong reasons, thereby improving robustness without requiring additional balanced held-out data or annotations.

Phuong Quynh Le, Jörg Schlötterer, Sari Sadiya +2

University of Marburg · Goethe University Frankfurt

Neural Network Robustness Spurious Correlation Robustness Shortcut Learning Spurious Correlation Robust Classification

Ask PDF

Abstract

Spurious Correlation Robustness

Explore similar work

Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

Cumulative Meta-Learning from Active Learning Queries for Robustness to Spurious Correlations

Shortcut Mitigation via Spurious-Positive Samples