Adversarial attacks - input perturbations imperceptible to humans that fool neural networks - remain both a persistent failure mode in machine learning, and a phenomenon with mysterious origins. To shed light, we define and analyze a network's perceptual manifold (PM) for a class concept as the space of all inputs confidently assigned to that class by the network. We find, strikingly, that the dimensionalities of neural network PMs are orders of magnitude higher than those of natural human concepts. Since volume typically grows exponentially with dimension, this suggests exponential misalignment between machines and humans, with exponentially many inputs confidently assigned to concepts by machines but not humans. Furthermore, this provides a natural geometric hypothesis for the origin of adversarial examples: because a network's PM fills such a large region of input space, any input will be very close to any class concept's PM. Our hypothesis thus suggests that adversarial robustness cannot be attained without dimensional alignment of machine and human PMs, and therefore makes strong predictions: both robust accuracy and distance to any PM should be negatively correlated with the PM dimension. We confirmed these predictions across 18 different networks of varying robust accuracy. Crucially, we find even the most robust networks are still exponentially misaligned, and only the few PMs whose dimensionality approaches that of human concepts exhibit alignment to human perception. Our results connect the fields of alignment and adversarial examples, and suggest the curse of high dimensionality of machine PMs is a major impediment to adversarial robustness.
Figures & tables
Figure 1: Visualization of our main argument: We show that a network’s perceptual manifold (PM) for any class concept (e.g. cat), defined to be the set of all images confidently perceived by the network as that class, is extremely high dimensional : 3060 out of a total of 3072 for CIFAR10 (large red manifold) and ≈135,000 out of 150,528 for CLIP and ImageNet. In contrast natural images perceived by humans as any class (e.g. dogs, airplanes or cats (blue, green and bright red manifolds)) are only ≈20 dimensional. This indicates that machine and human PMs for any concept are exponentially misaligned : there are exponentially many inputs confidently perceived as any given concept by machines, but not by humans (e.g. the two noise images in the network’s cat perceptual manifold). This exponential misalignment also explains the origin of adversarial examples: e.g. because the network’s cat PM fills up so much of image space, any other input (e.g. dog or airplane) is extremely close to it.
Figure 2: Comparison between the dimensionality (Participation Ratio on the left and Two Nearest Neighbors on the right) of the Perceptual Manifold of a WideResNet-28-10 (clean accuracy of 94.78% and robust accuracy of 0% ) to that of the natural images for each class. In the left plot, the PR of the natural images is ≈10 , which makes it barely visible. The arrows in the right plot indicate that those values are lower bounds. The excessive dimensionality of machine PMs relative to their natural counterparts signals exponential misalignment.
Figure 3: Monte Carlo estimate of the squared distance between a random point sampled from unit hypercube and the boundary of a d -dimensional ellipsoid. The principal axes are sampled from a uniform distribution U[6,30] , so the volume of the full 3072 dimensional ellipsoid is roughly equal to the estimated volume of the non robust model’s PM (when approximating it by an ellipsoid). Error bars cover the ±1σ interval. Theory is the red dashed curve.
Figure 4: Dimensionality comparison (Participation Ratio and Two Nearest Neighbors) of the CLIP Perceptual Manifold versus natural images (LSUN dataset). Arrows indicate that those reported are lower bounds on the actual value, we include a plot of how the predicted PR and 2NN scale with dataset size in Figure 26 and Figure 27 . We include a “Gibberish” control prompt: “kjdbfkw hsafj asjf gjkbg”; note that the natural dimensionality of the gibberish class is undefined (N/A) as no natural images correspond to it.
Natural Manifold
Perceptual Manifold
Class
PR
2NN
PR
2NN
Cat
10
21
2650
≥ 412
Truck
11
23
2652
≥ 420
Table 1: Dimensionality comparison between the Natural Manifold and CLIP’s Perceptual Manifold (PM) in the CIFAR-like setting ( 32×32 inputs). The PM is consistently orders of magnitude larger than the natural data manifold.
Figure 5: Representative samples from CLIP’s Perceptual Manifold for valid descriptions (e.g., “a photo of a bedroom”) and the control prompt “kjdbfkw hsafj asjf gjkbg”. Visually, all samples appear as noise, again indicating exponential misalignment between machine and human perception.
Figure 6: Dimensionality of the Perceptual Manifold (averaged over all CIFAR10 classes) versus L∞ robust accuracy for models from the CIFAR10 L∞ model zoo of RobustBench ( Croce et al., 2021 ) ; the models in legend order come from: ( Ding et al., 2020 ; Wong et al., 2020 ; Zhang et al., 2019a ; MadryLab, 2019 ; Addepalli et al., 2022 ; Zhang et al., 2019b ; Hendrycks et al., 2019 ; Sehwag et al., 2022 ; Rade and Moosavi-Dezfooli, 2021 ; Wu et al., 2020 ; Huang et al., 2022a ; Rebuffi et al., 2021 ; Huang et al., 2022b ; Cui et al., 2024 ; Wang et al., 2023 ; Peng et al., 2023 ; Bartoldson et al., 2024 ) . The left panel displays effective dimensionality (via Participation Ratio), while the right panel displays intrinsic dimensionality (via Two Nearest Neighbors). Note that 2NN values are lower bounds; however, based on the scaling of the predicted intrinsic dimension with dataset size in Figure 20 , the relative ranking will likely stay the same with more samples. We also report the scaling for the PR with samples in Figure 21 , and the full class by class breakdown in Figure 16 and Figure 17 .
Figure 7: Average squared Euclidean distance between a random point x∼U[0,1]D and the perceptual manifold, averaged over CIFAR-10 classes. The color scheme indicates robust accuracy (via RobustBench). We observe a strong correlations between robustness, PM dimension and distance to the PM: models with higher robust accuracy exhibit lower PM dimension, and a larger distance to the PM from random noise. For a class-wise breakdown, see Figure 22 and Figure 23 .
Figure 8: Individual samples from the PMs of all 10 CIFAR10 classes for each of 4 different models (1 model per row) with increasing robust accuracy from top to bottom. In particular, using the same model names as in other figure legends, we show examples from the PMs of “Standard”, “Wong2020Fast”, “Wu2020Adversarial _ extra”, “Peng2023Robust” with L∞ robust accuracies from top to bottom of 0%,43.21%,60.04%,71.07% respectively. In each row we have ordered the classes shown according to the PR dimensionality of their PMs for the corresponding network, from highest dimensionality at the left to lowest at the right. We note that we only see alignment between samples of the PM and human perception in the lowest dimensional PMs of the most robust networks. For example, the recognizable horse, dog and deer samples in the bottom row come from network PMs of PR dimensionality 140 , 133 and 80 , and 2NN dimensionality ≥89 , ≥138 , and ≥82 respectively, while the recognizable dog and horse samples in the third row come from network PMs of PR dimensionality 307 and 289 and 2NN dimensionality ≥176 and ≥183 respectively. In contrast in the second row from the top, despite the network achieving a robust accuracy of 43.21% , its PMs are unaligned to human perception with unrecognizable samples because their PM PR and 2NN dimensionalities are all high (ranging from PR =3016 , and 2NN ≥470 for frog to PR =1397 and 2NN ≥295 for dog). See Figure 24 and Figure 25 for more samples of PMs from all 18 networks studied in this paper and all 10 CIFAR10 classes, again indicating alignment at the level of samples only when PM dimensionality is low.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Comparison of distances to the Perceptual Manifold (PM) initialized from natural images versus random noise. The x -axis displays the typical squared Euclidean distance from a random point x∼U[0,1]D to the ‘truck’ class PM; the y -axis displays the distance from natural images (excluding the target class) to the same PM. Each point represents a different model. The dashed line denotes the identity y=x . For the legend, refer to Figure 7
Figure 10: Full colored lines: density of the spectrum of the covariance matrix of the PM of different model. Dashed black line: density of the spectrum of the covariance of natural images. Both refer to class 9 of CIFAR10, i.e. trucks. For the plots of all classes refer to Figure 28
Figure 11: Alignment Score analysis between natural image subspaces and Model PM subspaces. k is defined as the number of eigenvectors required to explain 90% of the variance in natural images. (a) Comparison with fixed k=m . We observe a very small increase in alignment as robust accuracy improves, though values remain close to the random baseline (red dashed line). (b) Alignment as the model subspace dimension m increases. Even robust models (yellow) fail to capture the natural subspace significantly faster than random chance.
Figure 12: Radially averaged Power Spectral Density (PSD) of PM samples compared to natural CIFAR-10 images. The PSD is computed for each class separately and then averaged. Note the log-log scale: robust models show a steeper slope closer to natural images, but higher overall power. Black dashed line is the PSD of actual CIFAR10 images, red dashed line is the expected k−2 line
Figure 13: Comparison of the Participation Ratio (PR) of the Perceptual Manifold for a standard ResNet-50 (clean accuracy of 76.52%, robust accuracy of 0%) against natural images across 9 different ImageNet classes. Note that the estimated PR of the perceptual manifold represents a lower bound. The PR of natural images is ≈20 , rendering it virtually imperceptible compared to the ≥130,000 dimensions occupied by the machine PM.
Figure 14: Participation Ratio of the Perceptual Manifold (averaged over 9 ImageNet classes) versus L∞ robust accuracy for models from the RobustBench ImageNet model zoo ( Croce et al., 2021 ) . The evaluated models, in legend order, are from ( Xu et al., 2025 ; Liu et al., 2023 ; Singh et al., 2023 ; Rodríguez-Muñoz et al., 2024 ; Peng et al., 2023 ; Debenedetti et al., 2023 ; Mo et al., 2022 ) . Full class-by-class breakdowns and dataset size scaling are provided in Figure 29 and Figure 30 , respectively.
Figure 15: Average squared Euclidean distance between a random initialization x∼U[0,1]D and the Perceptual Manifold as a function of the Participation Ratio, averaged over 9 ImageNet classes. Node colors indicate L∞ robust accuracy as reported by RobustBench. See Figure 31 for class-wise breakdowns.
Figure 16: Participation Ratio of the Perceptual Manifold (conditioned on each CIFAR10 class) versus L∞ robust accuracy for models from RobustBench ( Croce et al., 2021 ) . The robust accuracy is the one reported on the RobustBench leaderboard.
Figure 17: Two Nearest Neighbor dimension of the Perceptual Manifold (conditioned on each CIFAR10 class) versus L∞ robust accuracy for models from RobustBench ( Croce et al., 2021 ) . The robust accuracy is the one reported on the RobustBench leaderboard. Note that the values for the 2NN are lower bounds, however, based on the scaling of the predicted intrinsic dimension with dataset size in Figure 20 , we are confident the relative ranking will stay the same
Figure 18: Barplot of the Participation Ratio (PR) for all 10 CIFAR10 classes for each model
Figure 19: Barplot of the Two Nearest Neighbor (2NN) dimensionality for all 10 CIFAR10 classes for each model
Figure 20: Convergence of Two Nearest Neighbors (2NN) estimates with sample size. We compute 2NN estimates on subsamples of the full dataset ( N=106 ) to analyze scaling behavior. Due to the curse of dimensionality, a convergence plateau is not observable within the available sample limit; instead, the estimates scale as ∼lnN . However, the stratification of curves—corresponding to models with varying degrees of adversarial robustness—remains consistent across all scales. This suggests that the relative ranking of model manifold dimensionality is invariant to sample size even prior to convergence.
Figure 21: Convergence of Participation Ratio (PR) estimates with sample size. We compute PR estimates on subsamples of the full dataset ( N=106 ) to analyze scaling behavior. The estimates for all classes plateau as N approaches 106 , indicating that the dataset size is sufficient for stable dimensionality estimation. The ordered stratification of curves corresponding to models of increasing robustness shows once again that more robust models yield lower PR values
Figure 22: Typical distance between a random point and the class-conditional PM as a function of the Participation Ratio for each class. Error bars represent the ±σ interval.
Figure 23: Typical distance between a random point and the class-conditional PM as a function of the Two Nearest Neighbor dimension for each class. Error bars represent the ±σ interval.
Figure 24: pt 1. Samples from the PM for all CIFAR-10 classes across the 9 models whose PM have the highest average (over classes) dimensionality. Rows are sorted in descending order top to bottom by the model’s average PR, which is correlated with the robust accuracy. Columns (classes) are sorted by the PR of the corresponding PM descending from left to right.
Figure 25: pt 2. Samples from the PM for all CIFAR-10 classes across the 9 models whose PM have the lowest average (over classes) dimensionality. Rows are sorted in descending order top to bottom by the model’s average PR, which is correlated with the robust accuracy. Columns (classes) are sorted by the PR of the corresponding PM descending from left to right.
Figure 26: How the computed Participation Ratio scales with dataset size. 106 samples are actually not enough for the estimation of the PM’s dimension to converge, so value needs to be interpreted as lower values. That of natural images, on the other hand, has converged
Figure 27: How the computed Two Nearest Neighbor dimension scales with dataset size. 106 samples are actually not enough for the estimation of the PM’s dimension to converge, so value needs to be interpreted as lower values. That of natural images, on the other hand, has converged
Figure 28: Full colored lines: density of the spectrum of the covariance matrix of the PM of different model. Dashed black line: density of the spectrum of the covariance of natural images.
Figure 29: Participation Ratio of the Perceptual Manifold (each subplot conditioned on a different ImageNet class) versus L∞ robust accuracy (as reported in RobustBench ( Croce et al., 2021 ) )
Figure 30: Convergence of Participation Ratio (PR) estimates with sample size. We compute PR estimates on subsamples of the full dataset ( N=106 ) to analyze scaling behavior. The estimates for all robust models plateau as N approaches 106 , while that for the standard model doesn’t and is therefore to be taken as a lower bound. The ordered stratification of curves corresponding to models of increasing robustness shows once again that more robust models yield lower PR values
Figure 31: Typical distance between a random point and the class-conditional PM as a function of the Participation Ratio for 9 ImageNet Classes. The color scheme report the robust accuracy
Figure 32: pt 1. Samples from the PM of 9 ImageNet classes across the 7 models whose PM have the highest average (over classes) dimensionality. Rows are sorted in descending order top to bottom by the model’s average PR, which is correlated with the robust accuracy. Columns (classes) are sorted by the PR of the corresponding PM descending from left to right.
Figure 33: pt 2. Samples from the PM of 9 Imagenet classes across the 6 models whose PM have the lowest average (over classes) dimensionality. Rows are sorted in descending order top to bottom by the model’s average PR, which is correlated with the robust accuracy. Columns (classes) are sorted by the PR of the corresponding PM descending from left to right.
Several theoretical works have tried to explain the adversarial vulnerability of deep neural networks through properties of high-dimensional geometry. However, the assumptions underlying these works are rarely examined empirically, and systematic evidence remains limited. In this work, we present a systematic study of the role of input dimensionality in both the emergence and the targeted control of adversarial examples. We first analyse the scope and limitations of existing theoretical frameworks based on concentration of measure, showing that real image classes exhibit strong empirical localization, beyond what such theories typically assume. We then conduct an extensive empirical evaluation across hierarchical image datasets spanning a wide range of input dimensionalities and diverse neural architectures. Our results consistently show that adversarial examples become easier to construct as dimensionality increases. We also investigate how input dimensionality affects the additional difficulty of crafting targeted adversarial examples. In particular, we provide theoretical arguments showing that high-dimensional geometry implies that enforcing a specific target label entails only a limited additional distortion compared to untargeted attacks. We corroborate this insight through extensive experiments, demonstrating that the gap between targeted and untargeted perturbations remains small and further narrows as input dimensionality increases. While, taken together, our findings establish high input dimensionality as a fundamental factor underlying the emergence and targeted control of adversarial examples, whether this phenomenon primarily arises from the interplay between high-dimensional geometry and data distributions or from the architectural properties of deep neural networks remains an open question.
Thanks to their extensive capacity, over-parameterized neural networks exhibit superior predictive capabilities and generalization. However, having a large parameter space is considered one of the main suspects of the neural networks' vulnerability to adversarial example -- input samples crafted ad-hoc to induce a desired misclassification. Relevant literature has claimed contradictory remarks in support of and against the robustness of over-parameterized networks. These contradictory findings might be due to the failure of the attack employed to evaluate the networks' robustness. Previous research has demonstrated that depending on the considered model, the algorithm employed to generate adversarial examples may not function properly, leading to overestimating the model's robustness. In this work, we empirically study the robustness of over-parameterized networks against adversarial examples. However, unlike the previous works, we also evaluate the considered attack's reliability to support the results' veracity. Our results show that over-parameterized networks are robust against adversarial attacks as opposed to their under-parameterized counterparts.
Srishti Gupta, Zhang Chen, Luca Demetrio +9
Sapienza University of Rome, Rome, Italy. · University of Cagliari, Cagliari, Italy. · Northwestern Polytechnical University, Xi’An, China. +1
Deep neural networks (NNs) for computer vision are vulnerable to adversarial attacks, such that minuscule malicious changes to inputs may induce unintuitive outputs. One key approach to verify and mitigate such robustness issues is to falsify expected output behavior, which enables local robustness guarantees and adversarial training. Due to the black-box nature of NNs, current attacks only falsify a class of the final output, such as flipping from \text{stop_sign} to \neg \text{stop_sign}. In this work we argue that robustness analysis should focus more generally on logical inconsistencies in concept-level reasoning rather than solely on output misclassification. We formalize this perspective through concept-based properties, i.e., constraints over human-interpretable concepts such as \text{red} \land \text{octogonal} \implies \text{stop_sign} and study the falsification of such properties as a principled form of adversarial attack. Furthermore, an implementation of concept-based properties on already trained NNs is proposed using techniques from concept-based explainable artificial intelligence (C-XAI). For that we formally prove that attacks on concept-based properties are expected to have a reduced search space compared to simple class falsification, whilst arguably be more aligned with intuitive robustness targets. The practical effectiveness of our approach for logically informed adversarial training is shown in an empirical study across multiple models and datasets, including both vision and a security-relevant code processing task. Our results demonstrate that concept-level falsification generalizes classical adversarial attacks: Retraining on our novel attacks maintains robustness gains as compared to classical adversarial retraining whilst better eliminating semantic biases.
Raik Dankworth, Felix Mächtle, Thomas Eisenbarth +1
Institute of Software Engineering and Programming Languages University of Luebeck · Institute of IT Security University of Luebeck