Reproducibility study on how to find Spurious Correlations, Shortcut Learning, Clever Hans or Group-Distributional non-robustness and how to fix them

Ole Delzer; Sidney Bender

cs.LGApr 6, 2026

Reproducibility study on how to find Spurious Correlations, Shortcut Learning, Clever Hans or Group-Distributional non-robustness and how to fix them

Authors: Ole Delzer, Sidney Bender

Organizations: Machine Learning Group Technische Universität Berlin

Abstract

Deep Neural Networks (DNNs) are increasingly utilized in high-stakes domains like medical diagnostics and autonomous driving where model reliability is critical. However, the research landscape for ensuring this reliability is terminologically fractured across communities that pursue the same goal of ensuring models rely on causally relevant features rather than confounding signals. While frameworks such as distributionally robust optimization (DRO), invariant risk minimization (IRM), shortcut learning, simplicity bias, and the Clever Hans effect all address model failure due to spurious correlations, researchers typically only reference work within their own domains. This reproducibility study unifies these perspectives through a comparative analysis of correction methods under challenging constraints like limited data availability and severe subgroup imbalance. We evaluate recently proposed correction methods based on explainable artificial intelligence (XAI) techniques alongside popular non-XAI baselines using both synthetic and real-world datasets. Findings show that XAI-based methods generally outperform non-XAI approaches, with Counterfactual Knowledge Distillation (CFKD) proving most consistently effective at improving generalization. Our experiments also reveal that the practical application of many methods is hindered by a dependency on group labels, as manual annotation is often infeasible and automated tools like Spectral Relevance Analysis (SpRAy) struggle with complex features and severe imbalance. Furthermore, the scarcity of minority group samples in validation sets renders model selection and hyperparameter tuning unreliable, posing a significant obstacle to the deployment of robust and trustworthy models in safety-critical areas.

Figures & tables

Figure 1: Research scope overview. A : Sketch illustrating how a confounder that is spuriously correlated with the class labels due to group imbalance can lead to Clever Hans behavior. Majority groups (green area) generalize well, while unseen minority group data (red area) is systematically misclassified. The few training samples in the minority group can easily be overfitted and hence do not prevent the classifier from learning a Clever Hans strategy. B : Conceptual visualization of the selected correction methods. C : Overview of the datasets used in our experiments, highlighting the distinction between the causal (x-axis) and the confounding (y-axis) feature. D : Identification of valid and Clever Hans decision strategies with the help of LRP and SpRAy; corresponding annotation of whole data point clusters with group labels. E : Application of the correction methods, followed by an evaluation and comparison of the correction quality. This specific example shows the decision boundary and confidence regions of the classifier trained on biased Squares before and after applying RR-ClArC. The ideal decision boundary would be a vertical line in the center, perfectly separating the samples by the value of the causal feature.

Figure 2: Selected samples from the CelebA dataset paired with their respective heatmaps as an overlay over the original image. The classifier was trained to distinguish between images of smiling and non-smiling persons. A watermark was added in the bottom right of each image, but the transparency of the watermark is correlated with the class labels. For images with the label Smiling , the watermark is bolder in most cases, and for images with the label Not Smiling , it is usually more transparent. Thus, the model learns to exploit the watermark as a Clever Hans feature, which can clearly be seen in the heatmap overlays: For the image on the very right in the Not Smiling row, we can see that the model correctly considered the mouth of the person to contribute to the Not Smiling class. However, the bold watermark actually had a negative contribution, possibly leading to a false classification. Conversely, for the Smiling class, a bold watermark acts as a strong positive indicator, sometimes leading the model to ignore the shape of the mouth completely, as e.g. in the first and second images in the bottom row.

Figure 3: Result of a Spectral Relevance Analysis on Class ’0’ samples of the Colored MNIST dataset, where 50% of samples from Class ’0’ are colored red, while all other samples are colored white. The spectral embedding is visualized with the help of t-SNE. It is clearly visible that the samples are arranged in two distinct clusters, representing the two decision strategies learned by the model: classification by coloring (left side), which constitutes a Clever Hans effect, and classification by the shape of the depicted digit (right side), which is the correct, causal concept that we actually want our model to base its decision on.

Panel of Figure 4 — Figure 4: Sketch illustrating how A-ClArC and P-ClArC use a CAV to (a) add the confounder to or (b) remove the confounder from all samples. During training, there are no examples from class B containing the confounder available, so the model strongly associates the confounder with class A, resulting in a Clever Hans decision boundary. During inference, if there are new data points from class B that do contain the confounder, they will get misclassified. To fix this Clever Hans behavior, both A-ClArC and P-ClArC project all data points onto a hyperplane that is orthogonal to the direction associated with the confounder (captured by the CAV), so that it can no longer be used by the model as a discriminative feature between the two classes. However, in this setting, P-ClArC comes with the advantage that no additional fine-tuning is necessary, because for data points on the hyperplane, the true decision boundary and the confounded decision boundary already result in the same classification. A-ClArC requires fine-tuning, because otherwise, all projected data points from class B would get misclassified, since the true decision boundary and the Clever Hans decision boundary deviate for class B samples containing the confounder.

Panel of Figure 5 — Figure 5: Counterfactual Explanations generated for source class Not Smiling and target class Smiling . From left to right, each panel shows the original image, its counterfactual, and the per-pixel difference between the original and the counterfactual. (a) shows a true counterfactual explanation, i.e. the Counterfactual Explainer changed the causal feature that distinguishes the source and target class by altering the mouth area in such a way that the depicted person is now actually smiling. (b) shows a false counterfactual, as the causal feature remains unchanged and instead the opacity of the watermark is increased, hinting at the model possibly being affected by Clever Hans.

Figure 6: Conceptual illustration of dataset rebalancing using CFKD. (Left) An imbalanced training set leads to a biased Clever Hans decision boundary, where the model incorrectly relies on a spurious correlation between the confounder and class labels. (Right) CFKD generates ’false’ counterfactuals (indicated by red arrows), for which only the value of the confounder is modified and the causal feature remains unchanged. By augmenting the dataset with these new samples, CFKD populates the sparse minority group, consequently weakening the spurious correlation. Fine-tuning on the augmented dataset hence allows the model to learn a more robust decision boundary.

Panel of Figure 7 — Figure 7: Generalized depictions of (a) the symmetric and (b) the asymmetric dataset distributions in the training and validation splits. The perfect decision boundary is a vertical line in the center, separating the two classes based on the value of the true, causal feature. Because of the simplicity bias, models trained on these datasets will prefer exploiting the confounder as a shortcut feature instead of basing their decision on the causal feature. Clever Hans "pushes" the decision boundary horizontally into the area of the minority group samples, thereby sacrificing accuracy on the minority groups in exchange for easily classifying majority group samples correctly. Due to their small number, minority group samples can be overfitted in order to achieve perfect training accuracy, with poor generalization for the minority groups as a consequence.

Figure 8: Randomly selected samples of the Squares dataset grouped by their respective foreground and background intensities. Class A contains samples with low foreground intensity ( $[0.0,0.5]$ ) and Class B contains samples with high foreground intensity ( $[0.5,1.0]$ ). In the poisoned versions of this dataset, the background intensity acts as a Clever Hans feature.

Figure 9: Randomly selected samples of the CelebA dataset, where we added a watermark of varying opacity to each sample in order to artificially introduce a confounding feature. The samples are grouped by the two target classes, Smiling and Not Smiling , and by the opacity of their watermark ( $<0.5$ as transparent and $\geq 0.5$ as opaque)

Figure 10: Randomly selected samples of the CelebA dataset, grouped by the two target classes Not Blond (Class A) and Blond (Class B), and by the confounding feature, which is the gender of the depicted person.

Figure 11: Randomly selected samples of the CAMELYON17 dataset grouped by presence of cancer metastases (classification target) and hospital of origin (confounder). Images from hospital A depict a pink color cast, while images from hospital B tend to be more purple.

Figure 12: Randomly selected samples from the Follicles dataset, grouped by the two target classes Primordial (Class A) and Growing (Class B), and by the confounding feature, which is the size of the depicted follicle. Surrounding tissue is covered by a mask in order to help the model focus on the follicle.

Dataset	Train accuracy			Validation accuracy			Test accuracy
	empirical	AGA	WGA	empirical	AGA	WGA	empirical	AGA	WGA
Squares
symmetric	100.0	$\mathrm{1}\mathrm{0}\mathrm{0}\,\mathrm{0}$	100.0	97.5	49.7	$0.0$	$51.1$	$51.1$	$1.8$
Squares
asymmetric	100.0	$\mathrm{1}\mathrm{0}\mathrm{0}\,\mathrm{0}$	100.0	92.0	68.7	$0.0$	$68.1$	$68.1$	$12.0$
Smiling

Table 1: Accuracies achieved by the uncorrected student models on training, validation, and test data. The test splits for Squares, CelebA Smiling, and Camelyon17 are balanced, i.e. all data groups have the same size, so here the empirical accuracy and the AGA are equal. In contrast, the data distributions in the test splits for CelebA Blond and Follicles are more similar to the distribution on the validation and training splits.

Panel of Figure 13 — Figure 13: Decision boundaries and confidence regions of the uncorrected student models trained on (a) symmetric and (b) asymmetric Squares. Training samples from Class A with low foreground intensity are depicted as red triangles, training samples from Class B with high foreground intensity as blue triangles. Likewise, new data in the red area is classified as Class A, and in the blue area as Class B. Color saturation reflects the model’s confidence. A perfect decision boundary would be a vertical line at 0.5 foreground intensity, as that would separate the two classes by the true causal feature, rendering the model completely insensible towards the confounder (the background intensity). Due to Clever Hans, the actual decision boundaries deviate heavily from the perfect solution in both cases.

average group accuracy (AGA)
Dataset	Uncorrected	DFR	Group DRO	P-ClArC	RR-ClArC	CFKD
Squares symmetric	$51.1$	$52.1$	$61.3$	$78.6$	79.6	94.5
Squares asymmetric	$68.1$	$73.9$	$77.1$	$85.2$	92.4	91.5
Smiling symmetric	$51.3$	$57.5$	$56.3$	$65.9$	68.7	79.6
Smiling asymmetric	$59.4$	$58.2$	$64.0$	$65.6$	80.5	86.6
Blond symmetric	$72.7$	$73.1$	$73.0$	$74.4$	74.5	79.1

Table 2: Average group accuracies (AGAs) and worst group accuracies (WGAs) on unseen test data before and after applying various correction methods, using ground-truth group labels for DFR, Group DRO, P-ClArC, and RR-ClArC. Model selection was based on validation AGA.

average group accuracy (AGA)
Dataset	Uncorrected	DFR	Group DRO	P-ClArC	RR-ClArC	CFKD
Squares symmetric	$51.1$	$52.1$	$64.2$	$76.1$	86.8	94.5
Squares asymmetric	$68.1$	$68.8$	$76.8$	$76.2$	90.4	91.5
Smiling symmetric	$51.3$	$54.0$	$59.6$	64.6	$57.0$	79.6
Camelyon17 asymmetric	$75.4$	77.5	77.0	$76.5$	$75.8$	$75.4$
Follicles	$72.5$	$76.1$	$73.5$	78.3	$68.8$	83.9

Table 3: Average group accuracies (AGA) and worst group accuracies (WGA) on unseen test data before and after applying various correction methods, using group labels provided by SpRAy for DFR, Group DRO, P-ClArC, and RR-ClArC. Datasets for which SpRAy did not yield labels of sufficient quality to conduct experiments are omitted. Test metrics were computed using ground-truth labels. Model selection was based on validation AGA.

Dataset	True Group Sizes	SpRAy Group Sizes	SpRAy Group Label Accuracy
Squares symmetric	490/10/10/490	490/10/11/489	100.0/100.0/81.8/99.8
Squares asymmetric	250/250/490/10	248/252/491/9	99.2/98.4/99.8/100.0
Smiling symmetric	490/10/10/490	470/30/12/488	99.1/20.0/58.3/99.4
Smiling asymmetric	250/250/490/10	—	—
Blond symmetric	490/10/10/490	—	—
Blond asymmetric	250/250/490/10	—	—

Table 4: Quality of the confounder labels generated by SpRAy. For each dataset, the table lists the true group sizes, the corresponding group sizes identified by SpRAy, and the alignment between SpRAy-derived and ground-truth labels. The reported SpRAy Label Accuracy denotes the proportion of samples within each group (as identified by SpRAy) whose SpRAy-assigned confounder label matches the true confounder label. Within each cell, values from left to right refer to the data groups

A^{-}

A^{+}

B^{-}

B^{+}

Panel of Figure 14 — Figure 14: Examples of SpRAy clustering samples by unintended concepts. (a) In Camelyon17, SpRAy grouped images by the presence of an image artifact rather than by color cast. (b) In CelebA Blond, SpRAy clustered samples according to background color instead of gender.

Panel of Figure 15 — Figure 15: T-SNE visualizations of the Spectral Embedding for Class B of symmetric Squares. The circled clusters correspond to those manually identified with ViRelAy and used for the results reported in Table 4 . Data points are colored according to cluster assignments obtained through (a) Agglomerative Clustering, (b) K-Means, (c) DBSCAN, and (d) HDBSCAN. Except for DBSCAN, the automatically derived clusters align poorly with the true confounder labels, whereas the manually identified clusters match them almost perfectly.

Panel of Figure 16 — Figure 16: Comparison of (a) t-SNE and (b) UMAP visualizations of the Spectral Embedding for Class B of symmetric Squares. With UMAP, the two groups tend to appear more distinctly separated, but the majority group often is more fractured. In contrast, t-SNE typically yields smoother, more continuous alignments of samples by confounder value, but the two groups are less clearly delineated. Both techniques provide complementary perspectives, and exploring multiple visualizations proved beneficial for reliable manual clustering with Virelay.

Panel of Figure 17 — Figure 17: t-SNE visualizations of the spectral embeddings obtained by applying SpRAy to Class B samples of the symmetric Squares dataset using attribution maps from (a) an intermediate layer (layer 6) and (b) the penultimate layer (layer 12) of the biased model. Using layer 6 attribution maps leads to a better alignment of the Spectral Embedding with the confounder values.

Panel of Figure 18 — Figure 18: Decision boundaries and confidence regions for Squares after applying DFR to the student. The optimal decision boundary corresponds to a vertical line at $0.5$ foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy-derived labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.

Panel of Figure 19 — Figure 19: Decision boundaries and confidence regions for Squares after applying Group DRO to the student. The optimal decision boundary corresponds to a vertical line at $0.5$ foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy-derived labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.

Panel of Figure 20 — Figure 20: Impact of projection location $l$ on validation and test performance when correcting with P-ClArC. Each data point shows the validation and test AGA for the optimal model at a given layer $l$ , selected based on the highest validation AGA. The layer $l$ indicates where the projection was inserted into the architecture, with $l=0$ signifying a projection directly on the input image. The charts highlight a significant divergence between validation and test AGA, demonstrating that validation performance is an unreliable estimator for true generalization in our setting.

Panel of Figure 21 — Figure 21: Decision boundaries and confidence regions for Squares after applying P-ClArC to the students. The optimal decision boundary corresponds to a vertical line at 0.5 foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.

Panel of Figure 22 — Figure 22: Comparison of the validation and test AGAs on symmetric Squares for the top 20 models, which were selected from a grid search based on their validation performance. The results are shown for corrections using (a) true confounder labels and (b) confounder labels provided by SpRAy.

Panel of Figure 23 — Figure 23: Comparison of the validation and test AGAs on symmetric CelebA Smiling for the top 20 models, which were selected from a grid search based on their validation performance. The results are shown for corrections using (a) true confounder labels and (b) confounder labels provided by SpRAy.

Panel of Figure 24 — Figure 24: Comparison of the validation and test AGAs on Follicles for the top 20 models, which were selected from a grid search based on their validation performance. The results are shown for corrections using (a) true confounder labels and (b) confounder labels provided by SpRAy.

Panel of Figure 25 — Figure 25: Decision boundaries and confidence regions for Squares after applying RR-ClArC to the students. The optimal decision boundary corresponds to a vertical line at 0.5 foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.

Panel of Figure 26 — Figure 26: A selection of counterfactual explanations generated during the CFKD process for asymmetric Camelyon17. From left to right, each group shows the original image, its corresponding counterfactual, and the pixel-wise difference between the two. Panel (a) shows samples where the original was classified as No Tumor and the counterfactual as Tumor by the student, while panel (b) shows the reverse case.

Panel of Figure 27 — Figure 27: Decision boundaries and confidence regions for Squares after applying CFKD to the students. The optimal decision boundary corresponds to a vertical line at 0.5 foreground intensity. Panels (a) and (b) depict results for symmetric and asymmetric Squares, respectively.

Group Distributionally Robust Optimization

See all 9 papers

Feb 26, 2025

Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradients

Machine learning models are inherently bound to the distribution of the training data, often exploiting non-causal shortcuts. As a result, achieving robustness to spurious correlations remains a challenge. While existing approaches rely on data manipulation or re-weighting strategies to achieve robustness, they typically require dense group labels, multiple training domains, or specialized pre-processing. We propose Invariance Pair Guidance (IPG), a method to mitigate reliance on spurious correlations using a sparse set of counterfactual pairs. Unlike other methods demanding extensive supervision, IPG utilizes a novel dual-update mechanism to dynamically correct the optimization trajectory. We generate input pairs that isolate the spurious attribute to define the invariance, a characteristic that should not affect the outcome of the model. Based on these pairs, we define a corrective gradient that complements the traditional gradient descent approach. The correction adapts via a predefined invariance condition. Experiments on ColoredMNIST, Waterbirds-100, and CelebA datasets demonstrate the effectiveness of our approach and its robustness to group shifts, supported by a theoretical convergence analysis. IPG offers a data-efficient and theoretically grounded path to robustness.

Martin Surner, Abdelmajid Khelil, Ludwig Bothmann

Institute for Data and Process Science, Landshut University of Applied Sciences, Landshut, Germany. · Computer Science Department, Landshut University of Applied Sciences, Landshut, Germany. · Department of Statistics, LMU Munich, Munich, Germany. +1

Spurious Correlation Robustness Invariant Representation Learning

Ask PDF

May 13, 2026

Shortcut Mitigation via Spurious-Positive Samples

Shortcut mitigation strategies commonly rely on training data annotations, group-balanced held-out data or the presence of all groups, i.e., all combinations of (spurious) attributes and classes, in the training data. However, these requirements are rarely met in practice. We instead propose a method for targeted model analysis to identify a small set of instances in which the model relies on spurious attributes. Using that set and following ``this feature should not be used for prediction'' reasoning, we identify highly relevant neurons in an intermediate layer and regularize their impact. This ensures that models learn to depend on informative features rather than being right for the wrong reasons, thereby improving robustness without requiring additional balanced held-out data or annotations.

Phuong Quynh Le, Jörg Schlötterer, Sari Sadiya +2

University of Marburg · Goethe University Frankfurt

Neural Network Robustness Spurious Correlation Robustness Shortcut Learning Spurious Correlation Robust Classification

Ask PDF

Feb 6, 2026

Rank-Constrained Adaptation for Reliable Real-World Performance

Deep learning models trained to optimize average accuracy often exhibit systematic failures on particular subpopulations. In real-world settings like healthcare, the subpopulations most affected by such disparities are frequently unlabeled, partially observed, or not known in advance. Existing group-robust methods typically assume prior knowledge of the relevant subgroups, using group annotations for training, validation, or model selection. We propose Misclassification Aware Rank-Limited Adaptation (MARLA), a parameter-efficient method for improving worst group performance without explicit subgroup annotations. MARLA leverages an ERM-trained model by calculating the model's misclassification probability scores on a held-out adaptation set to identify a low-dimensional subspace where errors concentrate. We then learn a rank-restricted additive correction to the classifier logits within that subspace. Across seven real-world datasets, we evaluate group robustness under three settings: no knowledge of subgroup relevance, partial knowledge of subgroup relevance, and full knowledge of subgroup relevance. MARLA improves worst-group performance while remaining fast and parameter-efficient, with data-guided hyperparameter selection.

Abinitha Gourabathina, Hyewon Jeong, Teya Bergamaschi +2

Department of Electrical Engineering & Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA · Harvard School of Medicine, Harvard, Boston, MA, USA.

Distribution Shift Robustness Low-Rank Adaptation

Ask PDF

Abstract

Group Distributionally Robust Optimization

Explore similar work

Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradients

Shortcut Mitigation via Spurious-Positive Samples

Rank-Constrained Adaptation for Reliable Real-World Performance