cs.LGApr 6, 2026
SaveReproducibility study on how to find Spurious Correlations, Shortcut Learning, Clever Hans or Group-Distributional non-robustness and how to fix them
Organizations: Machine Learning Group Technische Universität Berlin
Abstract
Deep Neural Networks (DNNs) are increasingly utilized in high-stakes domains like medical diagnostics and autonomous driving where model reliability is critical. However, the research landscape for ensuring this reliability is terminologically fractured across communities that pursue the same goal of ensuring models rely on causally relevant features rather than confounding signals. While frameworks such as distributionally robust optimization (DRO), invariant risk minimization (IRM), shortcut learning, simplicity bias, and the Clever Hans effect all address model failure due to spurious correlations, researchers typically only reference work within their own domains. This reproducibility study unifies these perspectives through a comparative analysis of correction methods under challenging constraints like limited data availability and severe subgroup imbalance. We evaluate recently proposed correction methods based on explainable artificial intelligence (XAI) techniques alongside popular non-XAI baselines using both synthetic and real-world datasets. Findings show that XAI-based methods generally outperform non-XAI approaches, with Counterfactual Knowledge Distillation (CFKD) proving most consistently effective at improving generalization. Our experiments also reveal that the practical application of many methods is hindered by a dependency on group labels, as manual annotation is often infeasible and automated tools like Spectral Relevance Analysis (SpRAy) struggle with complex features and severe imbalance. Furthermore, the scarcity of minority group samples in validation sets renders model selection and hyperparameter tuning unreliable, posing a significant obstacle to the deployment of robust and trustworthy models in safety-critical areas.
Figures & tables
Figure 1: Research scope overview. A : Sketch illustrating how a confounder that is spuriously correlated with the class labels due to group imbalance can lead to Clever Hans behavior. Majority groups (green area) generalize well, while unseen minority group data (red area) is systematically misclassified. The few training samples in the minority group can easily be overfitted and hence do not prevent the classifier from learning a Clever Hans strategy. B : Conceptual visualization of the selected correction methods. C : Overview of the datasets used in our experiments, highlighting the distinction between the causal (x-axis) and the confounding (y-axis) feature. D : Identification of valid and Clever Hans decision strategies with the help of LRP and SpRAy; corresponding annotation of whole data point clusters with group labels. E : Application of the correction methods, followed by an evaluation and comparison of the correction quality. This specific example shows the decision boundary and confidence regions of the classifier trained on biased Squares before and after applying RR-ClArC. The ideal decision boundary would be a vertical line in the center, perfectly separating the samples by the value of the causal feature.
Figure 2: Selected samples from the CelebA dataset paired with their respective heatmaps as an overlay over the original image. The classifier was trained to distinguish between images of smiling and non-smiling persons. A watermark was added in the bottom right of each image, but the transparency of the watermark is correlated with the class labels. For images with the label Smiling , the watermark is bolder in most cases, and for images with the label Not Smiling , it is usually more transparent. Thus, the model learns to exploit the watermark as a Clever Hans feature, which can clearly be seen in the heatmap overlays: For the image on the very right in the Not Smiling row, we can see that the model correctly considered the mouth of the person to contribute to the Not Smiling class. However, the bold watermark actually had a negative contribution, possibly leading to a false classification. Conversely, for the Smiling class, a bold watermark acts as a strong positive indicator, sometimes leading the model to ignore the shape of the mouth completely, as e.g. in the first and second images in the bottom row.
Figure 3: Result of a Spectral Relevance Analysis on Class ’0’ samples of the Colored MNIST dataset, where 50% of samples from Class ’0’ are colored red, while all other samples are colored white. The spectral embedding is visualized with the help of t-SNE. It is clearly visible that the samples are arranged in two distinct clusters, representing the two decision strategies learned by the model: classification by coloring (left side), which constitutes a Clever Hans effect, and classification by the shape of the depicted digit (right side), which is the correct, causal concept that we actually want our model to base its decision on.
Figure 4: Sketch illustrating how A-ClArC and P-ClArC use a CAV to (a) add the confounder to or (b) remove the confounder from all samples. During training, there are no examples from class B containing the confounder available, so the model strongly associates the confounder with class A, resulting in a Clever Hans decision boundary. During inference, if there are new data points from class B that do contain the confounder, they will get misclassified. To fix this Clever Hans behavior, both A-ClArC and P-ClArC project all data points onto a hyperplane that is orthogonal to the direction associated with the confounder (captured by the CAV), so that it can no longer be used by the model as a discriminative feature between the two classes. However, in this setting, P-ClArC comes with the advantage that no additional fine-tuning is necessary, because for data points on the hyperplane, the true decision boundary and the confounded decision boundary already result in the same classification. A-ClArC requires fine-tuning, because otherwise, all projected data points from class B would get misclassified, since the true decision boundary and the Clever Hans decision boundary deviate for class B samples containing the confounder.
Figure 5: Counterfactual Explanations generated for source class Not Smiling and target class Smiling . From left to right, each panel shows the original image, its counterfactual, and the per-pixel difference between the original and the counterfactual. (a) shows a true counterfactual explanation, i.e. the Counterfactual Explainer changed the causal feature that distinguishes the source and target class by altering the mouth area in such a way that the depicted person is now actually smiling. (b) shows a false counterfactual, as the causal feature remains unchanged and instead the opacity of the watermark is increased, hinting at the model possibly being affected by Clever Hans.
Figure 6: Conceptual illustration of dataset rebalancing using CFKD. (Left) An imbalanced training set leads to a biased Clever Hans decision boundary, where the model incorrectly relies on a spurious correlation between the confounder and class labels. (Right) CFKD generates ’false’ counterfactuals (indicated by red arrows), for which only the value of the confounder is modified and the causal feature remains unchanged. By augmenting the dataset with these new samples, CFKD populates the sparse minority group, consequently weakening the spurious correlation. Fine-tuning on the augmented dataset hence allows the model to learn a more robust decision boundary.
Figure 7: Generalized depictions of (a) the symmetric and (b) the asymmetric dataset distributions in the training and validation splits. The perfect decision boundary is a vertical line in the center, separating the two classes based on the value of the true, causal feature. Because of the simplicity bias, models trained on these datasets will prefer exploiting the confounder as a shortcut feature instead of basing their decision on the causal feature. Clever Hans "pushes" the decision boundary horizontally into the area of the minority group samples, thereby sacrificing accuracy on the minority groups in exchange for easily classifying majority group samples correctly. Due to their small number, minority group samples can be overfitted in order to achieve perfect training accuracy, with poor generalization for the minority groups as a consequence.
Figure 8: Randomly selected samples of the Squares dataset grouped by their respective foreground and background intensities. Class A contains samples with low foreground intensity ( ) and Class B contains samples with high foreground intensity ( ). In the poisoned versions of this dataset, the background intensity acts as a Clever Hans feature.
Figure 9: Randomly selected samples of the CelebA dataset, where we added a watermark of varying opacity to each sample in order to artificially introduce a confounding feature. The samples are grouped by the two target classes, Smiling and Not Smiling , and by the opacity of their watermark ( as transparent and as opaque)
Figure 10: Randomly selected samples of the CelebA dataset, grouped by the two target classes Not Blond (Class A) and Blond (Class B), and by the confounding feature, which is the gender of the depicted person.
Figure 11: Randomly selected samples of the CAMELYON17 dataset grouped by presence of cancer metastases (classification target) and hospital of origin (confounder). Images from hospital A depict a pink color cast, while images from hospital B tend to be more purple.
Figure 12: Randomly selected samples from the Follicles dataset, grouped by the two target classes Primordial (Class A) and Growing (Class B), and by the confounding feature, which is the size of the depicted follicle. Surrounding tissue is covered by a mask in order to help the model focus on the follicle.
| Dataset | Train accuracy | Validation accuracy | Test accuracy | ||||||
|---|---|---|---|---|---|---|---|---|---|
| empirical | AGA | WGA | empirical | AGA | WGA | empirical | AGA | WGA | |
| Squares | |||||||||
| symmetric | 100.0 | 100.0 | 97.5 | 49.7 | |||||
| Squares | |||||||||
| asymmetric | 100.0 | 100.0 | 92.0 | 68.7 | |||||
| Smiling |
Table 1: Accuracies achieved by the uncorrected student models on training, validation, and test data. The test splits for Squares, CelebA Smiling, and Camelyon17 are balanced, i.e. all data groups have the same size, so here the empirical accuracy and the AGA are equal. In contrast, the data distributions in the test splits for CelebA Blond and Follicles are more similar to the distribution on the validation and training splits.
Figure 13: Decision boundaries and confidence regions of the uncorrected student models trained on (a) symmetric and (b) asymmetric Squares. Training samples from Class A with low foreground intensity are depicted as red triangles, training samples from Class B with high foreground intensity as blue triangles. Likewise, new data in the red area is classified as Class A, and in the blue area as Class B. Color saturation reflects the model’s confidence. A perfect decision boundary would be a vertical line at 0.5 foreground intensity, as that would separate the two classes by the true causal feature, rendering the model completely insensible towards the confounder (the background intensity). Due to Clever Hans, the actual decision boundaries deviate heavily from the perfect solution in both cases.
| Dataset | Uncorrected | DFR | Group DRO | P-ClArC | RR-ClArC | CFKD |
|---|---|---|---|---|---|---|
| average group accuracy (AGA) | ||||||
| Squares symmetric | 79.6 | 94.5 | ||||
| Squares asymmetric | 92.4 | 91.5 | ||||
| Smiling symmetric | 68.7 | 79.6 | ||||
| Smiling asymmetric | 80.5 | 86.6 | ||||
| Blond symmetric | 74.5 | 79.1 | ||||
Table 2: Average group accuracies (AGAs) and worst group accuracies (WGAs) on unseen test data before and after applying various correction methods, using ground-truth group labels for DFR, Group DRO, P-ClArC, and RR-ClArC. Model selection was based on validation AGA.
| Dataset | Uncorrected | DFR | Group DRO | P-ClArC | RR-ClArC | CFKD |
|---|---|---|---|---|---|---|
| average group accuracy (AGA) | ||||||
| Squares symmetric | 86.8 | 94.5 | ||||
| Squares asymmetric | 90.4 | 91.5 | ||||
| Smiling symmetric | 64.6 | 79.6 | ||||
| Camelyon17 asymmetric | 77.5 | 77.0 | ||||
| Follicles | 78.3 | 83.9 | ||||
Table 3: Average group accuracies (AGA) and worst group accuracies (WGA) on unseen test data before and after applying various correction methods, using group labels provided by SpRAy for DFR, Group DRO, P-ClArC, and RR-ClArC. Datasets for which SpRAy did not yield labels of sufficient quality to conduct experiments are omitted. Test metrics were computed using ground-truth labels. Model selection was based on validation AGA.
| Dataset | True Group Sizes | SpRAy Group Sizes | SpRAy Group Label Accuracy |
|---|---|---|---|
| Squares symmetric | 490/10/10/490 | 490/10/11/489 | 100.0/100.0/81.8/99.8 |
| Squares asymmetric | 250/250/490/10 | 248/252/491/9 | 99.2/98.4/99.8/100.0 |
| Smiling symmetric | 490/10/10/490 | 470/30/12/488 | 99.1/20.0/58.3/99.4 |
| Smiling asymmetric | 250/250/490/10 | — | — |
| Blond symmetric | 490/10/10/490 | — | — |
| Blond asymmetric | 250/250/490/10 | — | — |
Table 4: Quality of the confounder labels generated by SpRAy. For each dataset, the table lists the true group sizes, the corresponding group sizes identified by SpRAy, and the alignment between SpRAy-derived and ground-truth labels. The reported SpRAy Label Accuracy denotes the proportion of samples within each group (as identified by SpRAy) whose SpRAy-assigned confounder label matches the true confounder label. Within each cell, values from left to right refer to the data groups , , , .
Figure 14: Examples of SpRAy clustering samples by unintended concepts. (a) In Camelyon17, SpRAy grouped images by the presence of an image artifact rather than by color cast. (b) In CelebA Blond, SpRAy clustered samples according to background color instead of gender.
Figure 15: T-SNE visualizations of the Spectral Embedding for Class B of symmetric Squares. The circled clusters correspond to those manually identified with ViRelAy and used for the results reported in Table 4 . Data points are colored according to cluster assignments obtained through (a) Agglomerative Clustering, (b) K-Means, (c) DBSCAN, and (d) HDBSCAN. Except for DBSCAN, the automatically derived clusters align poorly with the true confounder labels, whereas the manually identified clusters match them almost perfectly.
Figure 16: Comparison of (a) t-SNE and (b) UMAP visualizations of the Spectral Embedding for Class B of symmetric Squares. With UMAP, the two groups tend to appear more distinctly separated, but the majority group often is more fractured. In contrast, t-SNE typically yields smoother, more continuous alignments of samples by confounder value, but the two groups are less clearly delineated. Both techniques provide complementary perspectives, and exploring multiple visualizations proved beneficial for reliable manual clustering with Virelay.
Figure 17: t-SNE visualizations of the spectral embeddings obtained by applying SpRAy to Class B samples of the symmetric Squares dataset using attribution maps from (a) an intermediate layer (layer 6) and (b) the penultimate layer (layer 12) of the biased model. Using layer 6 attribution maps leads to a better alignment of the Spectral Embedding with the confounder values.
Figure 18: Decision boundaries and confidence regions for Squares after applying DFR to the student. The optimal decision boundary corresponds to a vertical line at foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy-derived labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.
Figure 19: Decision boundaries and confidence regions for Squares after applying Group DRO to the student. The optimal decision boundary corresponds to a vertical line at foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy-derived labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.
Figure 20: Impact of projection location on validation and test performance when correcting with P-ClArC. Each data point shows the validation and test AGA for the optimal model at a given layer , selected based on the highest validation AGA. The layer indicates where the projection was inserted into the architecture, with signifying a projection directly on the input image. The charts highlight a significant divergence between validation and test AGA, demonstrating that validation performance is an unreliable estimator for true generalization in our setting.
Figure 21: Decision boundaries and confidence regions for Squares after applying P-ClArC to the students. The optimal decision boundary corresponds to a vertical line at 0.5 foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.
Figure 22: Comparison of the validation and test AGAs on symmetric Squares for the top 20 models, which were selected from a grid search based on their validation performance. The results are shown for corrections using (a) true confounder labels and (b) confounder labels provided by SpRAy.
Figure 23: Comparison of the validation and test AGAs on symmetric CelebA Smiling for the top 20 models, which were selected from a grid search based on their validation performance. The results are shown for corrections using (a) true confounder labels and (b) confounder labels provided by SpRAy.
Figure 24: Comparison of the validation and test AGAs on Follicles for the top 20 models, which were selected from a grid search based on their validation performance. The results are shown for corrections using (a) true confounder labels and (b) confounder labels provided by SpRAy.
Figure 25: Decision boundaries and confidence regions for Squares after applying RR-ClArC to the students. The optimal decision boundary corresponds to a vertical line at 0.5 foreground intensity. Panels (a) and (b) depict results for symmetric Squares using true confounder labels and SpRAy labels, respectively. Panels (c) and (d) show the corresponding results for asymmetric Squares.
Figure 26: A selection of counterfactual explanations generated during the CFKD process for asymmetric Camelyon17. From left to right, each group shows the original image, its corresponding counterfactual, and the pixel-wise difference between the two. Panel (a) shows samples where the original was classified as No Tumor and the counterfactual as Tumor by the student, while panel (b) shows the reverse case.
Figure 27: Decision boundaries and confidence regions for Squares after applying CFKD to the students. The optimal decision boundary corresponds to a vertical line at 0.5 foreground intensity. Panels (a) and (b) depict results for symmetric and asymmetric Squares, respectively.