Adversarial robustness does not entail human-aligned categorization
Authors: Itay Reznik, Raz Lapid, Moshe Sipper, Tal Golan
Organizations: Institute for Applied AI Research, Ben-Gurion University of the Negev · Intuit · Institute for Applied AI Research, Data Science Research Center, Ben-Gurion University of the Negev · Department of Industrial Engineering and Management, School of Brain Sciences and Cognition, Data Science Research Center, Ben-Gurion University of the Negev
Adversarial training can improve some forms of correspondence between artificial and biological vision, but whether robust classifiers acquire human-aligned classification behavior remains unclear. We generated 1,008 controversial stimuli by optimizing latent inputs to a pretrained GAN to elicit high-confidence disagreements between pairs of robust ImageNet classifiers. We compared four adversarially robust ImageNet classifiers - ResNet-101, ViT-B, ConvNeXt-L, and Swin-L - with two-alternative category judgments from 60 participants. We first evaluated each classifier on stimuli optimized to elicit disagreement with another model. Across classifiers, accuracy in predicting individual participants' single-trial category choices was near chance (48.6-51.3%), whereas majority judgments from other participants predicted responses with 85.0-86.8% accuracy. An analysis incorporating model prediction strength and participants' reported confidence found weak positive model-human correspondence for the two convolutional models. On stimuli optimized for disagreement between other model pairs, all four robust classifiers predicted human responses above chance but less accurately than predictions based on other participants' majority judgments; the same pattern held for 17 additional RobustBench models not used in stimulus generation. Across these 17 models, mean model-participant agreement was 67.3%, versus 81.3% mean participant-participant agreement on the same images. These results identify model-specific failures of human alignment and a model-human mismatch that persists across robust classifiers and substantially exceeds disagreement among human observers. Our findings show that adversarial robustness alone does not guarantee human-aligned categorization and suggest that robustness and human alignment should be evaluated as distinct properties of visual recognition systems.
Figures & tables
Figure 1: Controversial-stimulus generation. To synthesize a stimulus, two classifiers were assigned different target classes. CMA-ES sampled candidate inputs to a fixed, pre-trained BigGAN generator, and each generated image was evaluated by both classifiers. The objective favored images for which each classifier assigned high probability to its assigned target class. The resulting objective values were returned to CMA-ES, which iteratively updated its sampling distribution to search the generator input space for increasingly controversial stimuli. The resulting stimuli were then used to test how the conflicting model predictions aligned with human visual categorization.
Model
Parameters (M)
Clean acc. (%)
PGD100 acc. (%)
AutoAttack acc. (%)
ResNet-101
44.5
71.0
46.5
42.2
ViT-B/16
86.6
74.7
52.2
49.7
ConvNeXt-L
197.8
78.8
61.7
60.1
Swin-L
196.5
79.7
63.9
62.3
Table 1: Robust ImageNet classifiers used in the study. Parameters are reported in millions. Clean accuracy is top-1 accuracy on unperturbed images. PGD100 is top-1 accuracy under a 100-step projected gradient descent attack, and AutoAttack is top-1 accuracy under AutoAttack ( Croce and Hein, 2020 ) . Both adversarial evaluations used an ℓ∞ perturbation budget of ϵ=4/255 . All three accuracy values are the ARES-reported results on the same random sample of 1,000 images drawn without replacement from the ImageNet validation set; we independently validated all three results on that image set.
Figure 2: Generated controversial stimuli for all six model pairs. Each panel corresponds to one model pair; rows and columns denote the classes favored by the models on the vertical and horizontal axes, respectively. Diagonal cells are omitted because the two target classes always differed. Across the displayed stimuli, models assigned a mean probability of 0.99 to their target classes after normalization over the two candidate classes. Each cell shows the most controversial (Eq. 6 ) of three stimuli used in the behavioral experiment for that model pair and ordered target-class pair.
Figure 3: Model–human alignment for the four models. Both panels evaluate each model only on stimuli generated to target it. (A) Mean binary model–human agreement. (B) Signed-rank cosine similarity (SRCS). Colored points show participant-level values; black markers and error bars show group means and 95% two-factor bootstrap confidence intervals. The dashed line marks chance-level performance. Gray bars span the lower and upper bounds on the human noise ceiling.
Figure 4: Model–human alignment across levels of human consensus and human confidence. Both panels evaluate each model only on stimuli generated to target it. (A) Mean binary model–human agreement at each level of inter-participant agreement—the proportion of the 10 participants giving the most common category response. A value of 0.5 indicates an equal split, and 1.0 indicates unanimity. The gray region shows the attainable range of mean binary agreement at each level of inter-participant agreement. Even for stimuli classified unanimously by participants, mean binary model–human agreement remained near chance. (B) Mean binary model–human agreement for responses grouped by reported confidence. Gray bars span the lower and upper bounds on the human noise ceiling for each group of responses. These bounds were estimated using responses from participants who viewed the same stimuli, regardless of those participants’ reported confidence. Model–human agreement remained near chance even for responses rated certain . Colored markers and error bars show means and 95% two-factor bootstrap confidence intervals. Dashed lines mark chance level (0.5).
Figure 5: Model–human alignment for the four models on non-targeted stimuli. For both measures, each model is evaluated only on stimuli generated using pairs of the other three models. (A) Mean binary model–human agreement. (B) Signed-rank cosine similarity (SRCS). Colored points show participant-level values; black markers and error bars show group means and 95% two-factor bootstrap confidence intervals. Dashed lines mark chance-level performance. Gray bars span the lower and upper bounds on the human noise ceiling.
Figure 6: Model–human alignment for the 17 additional robust models. All 17 models were evaluated on all 1,008 stimuli; none was used to generate or select the stimuli. (A) Mean binary model–human agreement. (B) Signed-rank cosine similarity (SRCS). Colored points show participant-level values; black markers and error bars show group means and 95% two-factor bootstrap confidence intervals. Dashed lines mark chance-level performance. In each panel, the gray band spans the lower and upper bounds on the human noise ceiling shared by all 17 models. Colors identify Swin (blue), ConvNeXt (red), ViT/XCiT (purple), and CNN/WRN (green) models. Models are ordered from left to right by decreasing mean binary model–human agreement; the model identifiers in B apply to both panels.
Figure S1 : Example trial from the behavioral experiment. An image was presented centrally with the two candidate category labels. Each label was paired with three confidence options— certain , hesitating , and guessing . Participants selected one of the six resulting response buttons to report both their category judgment and confidence. The left-right positions of the two labels were randomized across trials. Participants were unaware that the two category labels represented opposing predictions from two models.
Figure S2 : Opposing category preferences for all 1,008 experimental stimuli. Each panel plots the probabilities assigned by two models to the same target category for the 168 experimental stimuli generated for that model pair. Each dot represents one stimulus. The category shown on both axes is the horizontal-axis model’s target category. P(category∣model) denotes the probability assigned to that category by the model, normalized over the two candidate categories. The dotted diagonal marks equal probabilities between models, and reference lines at 0.5 mark equal probabilities for the two categories. Dots near the bottom-right corner indicate strong disagreement: each model assigned high normalized probability to its own target category.
Figure S3 : Human agreement with opposing model predictions. Each off-diagonal cell shows the percentage of human responses matching the row model’s prediction on stimuli generated using the row and column models. The two percentages for each model pair sum to 100%. Bracketed values give 95% two-factor bootstrap confidence intervals. Responses did not differ significantly from an equal split for any of the six model pairs after FDR correction.
Figure S4 : Relationships among model–human agreement, parameter count, and clean and adversarial classification accuracy. Model–human agreement was quantified using mean binary agreement and signed-rank cosine similarity (SRCS). Clean and PGD100 accuracies were computed on the 1,000-image ARES validation sample (Supplementary Table S1 ). Each point represents one of the 17 additional models. Diagonal panels show histograms; off-diagonal panels show pairwise associations. Lower-triangle annotations report Spearman’s ρ . Two-sided p -values are shown for the six correlations between agreement measures and model properties, with FDR correction across these tests. AutoAttack accuracy was omitted because its rank correlation with PGD100 accuracy was near-perfect (Spearman ρ=.994 ).
Figure S5 : Mean human–human agreement exceeded model–human agreement. At the top, gray points show the 60 participants’ human–human agreement scores. Each score is the mean, across that participant’s 168 images, of the fraction of the other nine raters who made the same category choice. Below, colored points show the 17 additional models’ mean binary agreement with humans, calculated within each participant’s 168-image set and then averaged across all 60 participants. All models were evaluated on the same 1,008 images and none participated in stimulus generation or selection. Model choices were determined by the larger logit for the two candidate categories. Black markers and horizontal error bars show group means and 95% participant-and-stimulus bootstrap confidence intervals, with the 17 models held fixed. Colors identify model families. Points are spread vertically for visibility. The dashed line marks 50% agreement. Mean human–human and model–human agreement were 81.3% and 67.3%, respectively (difference: 14.0 percentage points; 95% confidence interval: 11.9–16.0).
Figure S6 : A targeted classifier could achieve high model–human agreement if it captured the average human observer’s category preferences. We simulated four experimental datasets, each with 60 participants whose category-choice probabilities were determined by ResNet-101 (A), ConvNeXt-L (B), Swin-L (C), or ViT-B/16 (D). Each dataset preserved the original participant–image assignments: 168 images per participant and ten responses per image across 1,008 images. Responses were sampled independently from probabilities that preserved the model’s category preferences, with variability adjusted to match the empirical noise-ceiling bounds on the 504 images targeting that model. We repeated the targeted-model agreement analysis for all four classifiers on each dataset, evaluating each classifier on its own 504 targeted images. Colored points show participant-level binary agreement; black markers and error bars show means and 95% participant-and-stimulus bootstrap intervals. Gray bars span the lower and upper synthetic noise ceilings for each classifier’s targeted images. The bold label identifies the model determining the simulated responses; dashed lines mark 50% agreement. The classifier determining the simulated responses achieved 85.47–87.88% agreement on its own targeted images, whereas all four classifiers showed near-chance agreement with human responses in the experiment (Fig. 3 A).
Adversarial Training (AT) is one of the most effective methods for developing robust deep neural networks (DNNs). However, AT faces a trade-off problem between clean accuracy and adversarial robustness. In this work, we reveal a surprising phenomenon for the first time: Varying input perturbation intensities for training samples near decision boundaries in AT have minimal impact on model robustness. This finding directly exposes the inconsistency between accuracy and robustness score fluctuations, leading us to identify the misalignment between input and latent spaces as a critical driver of the robustness-accuracy trade-off. To mitigate this misalignment for harmonizing accuracy and robustness, we define Robust Alignment as a new AT target, encouraging the model perception to change with input perturbations provided the final label prediction remains unchanged, which can be achieved via two novel ideas. First, we suggest a reduced and fixed perturbation intensity for those boundary samples, which facilitates the model to utilize the perturbations as learnable patterns, instead of noises that complicate decision boundaries meaninglessly. Second, we propose a Domain Interpolation Consistency Adversarial Regularization (DICAR), based on rigorous theoretical derivations, which explicitly introduces semantic alignment between input and latent spaces into AT. Based on these two ideas, we end up with a new Robust Alignment Adversarial Training (RAAT) method, effectively harmonizing accuracy and robustness. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, PreActResNet-18, and WideResNet-28-10 demonstrate the effectiveness of RAAT in improving the trade-off beyond four common baselines and a total of 14 related state-of-the-art (SOTA) works.
Safety-critical applications require classifiers that are both robust and reliable. Adversarial training is a widely adopted defense for improving robustness in deep neural networks; however, its effect on the reliability of predictive uncertainty remains underexplored. We investigate this gap through the lens of selective classification, which has rarely been systematically analyzed alongside adversarial robustness. We introduce a unified benchmark for the robustness-uncertainty trade-off. It standardizes architectures, augmentations, threat models, and evaluation metrics across clean, adversarial, and common-corruption settings. Across a wide range of state-of-the-art adversarial training methods, we uncover a recurring failure mode: several approaches improve robust accuracy while degrading uncertainty ranking, leading to poorer selective behavior. To address this, we propose Evidential Adversarial Training (EV-AT), which models uncertainty through a Dirichlet distribution and combines (i) an evidence-based loss promoting clean accuracy and reliable uncertainty with (ii) a robust evidence-alignment loss matching clean and adversarial predictions in log Dirichlet-parameter space. Extensive experiments show that EV-AT shifts the Pareto frontier of robustness-uncertainty trade-offs beyond prior state-of-the-art adversarial training methods. Our source code is publicly available at https://github.com/NicolasSournac/Robustness_Meets_Uncertainty.EV-AT.
Nicolas Sournac, Ahmed Baha Ben Jmaa, Bertrand Braeckeveldt
Multitel Research & Innovation Center, Artificial Intelligence Department, Belgium · TRAIL – Trusted AI Labs, Belgium
Human-model alignment is critical for trustworthy AI-assisted decision-making systems. Yet, most work evaluates model predictions against single ground-truth labels, overlooking that humans themselves often disagree on labels, a signal of genuine ambiguity. We investigate whether models struggle on the same instances that humans find difficult. We measure this on two vision datasets (FER+ and CIFAR-10H) where multiple human annotations per image capture human disagreement patterns. We evaluate eight pretrained models across three architectures (ResNet, EfficientNet, MobileNetV3) in two parts: first, whether model uncertainty (softmax confidence, entropy) correlates with human disagreement, and second, whether predictive multiplicity measures (inter-model disagreement, Jensen-Shannon divergence) do. We find that it does not: alignment is weak in both dimensions. At the discrete label level, 50.4% of CIFAR-10H images and 33.5% of FER+ images receive multiple valid classifications from humans, while the models converge on only one. These instances represent a critical failure case where humans perceive ambiguity and would request expert review, yet models decide confidently. At the continuous score level, single-model uncertainty correlates weakly with human disagreement (ρ=0.24−−0.55), and predictive multiplicity provides only modest improvement. Widely-used uncertainty quantification methods do not reliably identify instances humans find ambiguous. Model uncertainty should not be treated as a trustworthy signal by default for decision-making in high-stakes scenarios.
Manya Singh, Arjun Pakrashi
School of Computer Science, University College Dublin, Ireland