Deep models often rely on spurious correlations that fail on underrepresented groups. Existing methods either correct human-specified bias attributes or infer unknown biases from the learned representation space. However, annotated attributes may differ from the shortcuts actually used by a model, while a single representation provides incomplete and potentially noisy evidence. Our key insight is that a bias obscured in one feature space may be clearer in another. In this work, we propose CHORD (Cross-view Harmonic Outlier Reversal for Debiasing), a multi-view framework that allows samples to "sing in chorus" across diverse representation spaces to achieve robust cross-view consensus. CHORD aggregates aggregates within-class partitions in each view, which is a proxy for specifying candidate biases, to debias the target representation and classifier without bias supervision. Experiments on seven benchmarks across vision, language, speech, and medical imaging show consistent improvements in worst-group accuracy. Code and checkpoints are available at https://github.com/davelee-uestc/chord_debiasing_public .
Figures & tables
Figure 1: Overview of CHORD. CHORD separates proposing biases from testing them for reliable bias discovery in the target model. First, each view proposes a candidate bias. The intuition is that a noisy candidate bias pattern in one representation may be denoised in another. Each candidate partitions samples into majority and outlier subsets. We then test the candidate in the target space by removing the feature difference between the two subsets. Removing a real bias improves within-class compactness by moving outliers closer to their class centroid. Outliers whose nearest-centroid assignments return to their labeled class are used as minority candidates. Finally, we aggregate this evidence across views to retrain the classifier head.
Labels
Waterbirds
CelebA
MultiNLI
CivilComments
Mean
Tr.
Val
mAcc
WGA
mAcc
WGA
mAcc
WGA
mAcc
WGA
WGA
ERM
✗
✗
97.30
72.60
95.60
47.20
82.09
68.12
92.34
57.06
61.25
JTT [ 31 ]
✗
✓
93.30
86.70
88.00
81.10
78.60
72.60
91.10
69.30
77.43
MT [ 4 ]
✗
✗
93.00
86.40
91.30
78.00
———— Not Applicable ————
CNC [ 54 ]
✗
✗
90.90
88.50
89.90
88.80
-
-
81.70
68.90
-
AFR [ 38 ]
✗
✓
94.20
90.40
91.30
82.00
81.40
73.40
89.80
68.70
78.63
Table 1: Mean accuracy (mAcc) and worst-group accuracy (WGA) on four widely used benchmarks. For CHORD, we report means over 10 classifier seeds with fixed ERM features, with the standard deviations shown for WGA.
CREMA-D
MEAD
Mean
Method
mAcc
WGA
mAcc
WGA
mAcc
WGA
ERM
77.08
59.63
60.51
48.35
68.80
53.99
NSF
76.99
60.63
61.67
50.77
69.33
55.70
CHORD
77.11
67.02
61.19
52.84
69.15
59.93
GDRO ∗ [ 39 ]
76.02
59.20
61.30
47.29
68.67
53.25
Table 2: Macro-average of mAcc and WGA on CREMA-D and MEAD. GDRO ∗ averages attribute-matched experts, requiring oracle model switching.
Bias Label
Chexpert
Train
Val
mAcc
WGA
Gain
ERM
✗
✗
89.78
31.21
-
NSF
✗
✗
81.94
70.21
+39.00
CHORD
✗
✗
81.62
71.43
+40.22
GDRO [ 39 ]
✓
✓
78.90
74.50
+43.29
Table 3: Results on CheXpert. Both mAcc and WGA are reported; group-supervised GDRO is included as a reference.
Method
MV
SW
MD
WGA
ERM
47.20
Single View
✓
84.27
MV-Hard
✓
✓
86.78
CHORD
✓
✓
✓
89.06
Table 4: Component ablation on CelebA. MV adds multiple reversal-sample sets, SW aggregates their support as soft sampling weights, and MD uses the target transformation t0 for the final classifier update. We report mean WGA over ten random seeds.
Target Only: 84.27
Conf.
WGA
Conf.
WGA
BT
86.32
DINO
87.39
BLIP
87.49
SimCLR
87.78
CLIP
86.84
SwAV
86.71
Table 5: Effect of individual pretrained views on CelebA. We report WGA after adding one view to Target Only in (a) or removing one from CHORD in (b). Every addition improves WGA, while most removals lower it. BT denotes Barlow Twins.
Method
Target View
Rep. Views
Geo. View
WGA
ERM
68.12
Target Only
✓
73.12
Target + Pre. Views
✓
✓
74.19
Target + Geo. View
✓
✓
73.36
CHORD
✓
✓
✓
75.25
Table 6: View-level ablation on MultiNLI.
ResNet-50
ViT-B/16
Dataset
Method
mAcc
WGA
mAcc
WGA
Waterbirds
ERM
97.30
72.60
92.89
78.97
CHORD
95.48
93.11
96.64
89.42
CelebA
ERM
95.60
47.20
96.21
39.72
CHORD
90.78
89.06
93.26
87.36
Table 7: Cross-architecture comparison on Waterbirds and CelebA. We report mean accuracy (mAcc) and worst-group accuracy (WGA) for ResNet-50 and ViT-B/16. CHORD improves WGA in all four dataset–architecture settings.
Method
Train attr.
mAcc
WGA by evaluation attribute
Sex
Race
Age-B
Age-Q
SexRace
Mean
ERM
–
77.08
62.00
61.54
62.50
52.63
59.46
59.63
CHORD
–
77.11
66.70
69.62
69.41
64.52
64.86
67.02
GDRO
Sex
75.92
59.41
62.32
61.32
55.10
59.38
59.51
Race
74.04
55.56
57.89
60.49
53.49
48.44
55.17
Age-B
77.08
63.37
63.08
64.15
55.56
56.67
60.57
Table 8: Multi-attribute evaluation on CREMA-D (%). Each row evaluates one model across five post-hoc group definitions. Train attr. specifies GDRO’s group annotation. Underlined entries are the attribute-matched GDRO results. CHORD results are averaged over 10 classifier seeds on fixed ERM features; GDRO uses one model seed per attribute. Age-B/Q denote binary/quartile age groups; SexRace denotes sex × race. CHORD achieves highest WGA under all five definitions without attribute-specific training.
ERM
CHORD
Target
Eval. Attribute
mAcc
WGA
mAcc
WGA
Blond Hair
Gender
95.60
47.20
90.78
89.06±0.50
Gender
Blond Hair
98.64
87.22
97.73
96.03±0.51
Table 9: Attribute-role exchange on CelebA to assess transfer across prediction tasks. Blond Hair and Gender exchange their roles as the prediction target and evaluation attribute, using the standard ResNet-50 CHORD configuration without task-specific retuning. We report means over 10 random seeds, with WGA standard deviations. CHORD improves WGA under both assignments.
Foundation models trained on biased datasets often rely on spurious correlations between target labels and non-causal attributes, resulting in poor generalization on minority groups. Bias mitigation remains challenging due to two fundamental issues. First, when group labels are unavailable, existing group-unsupervised methods typically infer spurious attributes implicitly from model behavior, making it difficult to identify spurious factors that are semantically aligned with real-world biases. Second, even with pseudo spurious supervision, most existing debiasing methods follow a single-branch design that operates within a single shared feature space, where target and spurious attributes are intrinsically entangled. To address the first challenge, we introduce Confidence-guided Bias Concept Mining (CBCM), which leverages diffusion-disentangled, semantically grounded concept representations to identify reliable spurious attributes without attribute annotations. To address the second challenge, we propose Dual-branch Cross-projection Debiasing (DCD), a prompt-tuning framework that separates target and spurious representations into two branches and explicitly removes spurious information through cross null-space projection while preserving target-relevant semantics. Extensive experiments on four benchmark datasets show that our method achieves state-of-the-art worst group accuracy among group-unsupervised approaches, while tuning at most 0.22% of the model parameters. The source code is available in the supplementary materials.
Xiangqian Zhao, Xinyang Jiang, Zhipeng Xu +5
Xidian University · Shaanxi, China · Microsoft Research Asia +1
Real-world datasets often contain spurious correlations that are not causally related to the target label. When such correlations dominate the majority of training samples, models tend to rely on them, leading to misclassification of minority samples that do not exhibit the same spurious patterns. While a potential approach is to select subsets of data to better represent the minority samples, this may require access to group labels, which are typically unknown. Furthermore, as we demonstrate, widely used sample scoring functions in the invariant subset or coreset selection literature largely depend on spurious features and therefore fail to accurately capture the importance or difficulty of core, causally relevant features. Accordingly, we propose to mitigate spurious correlations by developing a two-stage sample scoring function that disentangles the learning dynamics of core and spurious features and evaluates their difficulty separately. Based on our proposed metric, we introduce a new algorithm to find and prioritize informative samples both with and without spurious correlations. Extensive experiments demonstrate that a standard ERM model trained on our selected samples achieves superior performance compared to state-of-the-art debiasing techniques, while requiring as little as 10% of the original training data.
Arda Fazla, Abolfazl Hashemi
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.
Vision classifiers can exploit spurious correlations, achieving high in-distribution accuracy yet failing under distribution shift. Existing approaches to bias mitigation and analysis often depend on curated datasets, spurious-attribute or group labels, or retraining, which may be infeasible once a model is deployed or the relevant bias is unknown. We present a bias-label-free, post-hoc method for identifying spurious concepts in frozen vision models, relying only on standard class labels from a held-out audit dataset. For each target class, we collect patches from inputs predicted as that class and apply non-negative matrix factorization to intermediate activations to obtain a bank of interpretable concept vectors. Candidate concepts are then ranked with a bias estimator derived from their interaction with backpropagated gradients on misclassified examples: bias concepts tend to get activated when correcting false negatives and suppressed when correcting false positives. On Colored MNIST and Waterbirds the method recovers concepts aligned with the known spurious cue, and on CelebA it surfaces decision-relevant directions that only partially coincide with the annotated gender attribute; suppressing the top-ranked concepts at inference time improves worst-group accuracy by up to 17.9 percentage points on Waterbirds and 10.4 on CelebA without any retraining or parameter updates. Our method identifies decision-relevant spurious directions that need not coincide with annotated ones, providing both an interpretable auditing tool and an actionable debiasing handle for frozen vision models. Code is available at https://github.com/vitryt/label-free-bias-identification.
Thomas Vitry, Kieran Edgeworth, Stefan Wermter +1
University of Hamburg, Hamburg, Germany · Ecole Normale Superieure de Rennes, Rennes, France