Representational Geometry as a Diagnostic for Adversarial Robustness
Authors: Brandon Ismalej, Alex Cloninger, Gal Mishne
Organizations: Department of Computer Science, California State University, Northridge, Northridge, California, USA · Department of Mathematics, Halıcıo˘glu Data Science Institute, University of California San Diego, La Jolla, California, USA
Adversarial training is a common technique for making neural networks more robust to small, intentionally crafted input perturbations, and robustness is commonly measured through accuracy under attack. By exploring inter-network representations across different adversarial training objectives, we can analyze whether multiple networks organize their latent representations similarly. We study Multilayer Perceptrons trained on MNIST under three adversarial training regimes and compare the resulting population training dynamics and geometry to those of a vanilla network. We use M-PHATE to embed data samples across training epochs and examine how the geometry of the data changes during training under adversarial objectives. On penultimate-layer representations we measure displacement between clean and perturbed points, distributional distance through maximum mean discrepancy, and nearest-neighbor recovery. Our visualizations and quantitative analyses show that adversarial training leads to structural changes in how networks organize internal data representations, and robust accuracy and representational stability can rank the same defenses in opposite ways. These findings support a discussion of re-evaluating how practitioners approach the auditing of model robustness by treating latent representations and their training trajectories as diagnostic artifacts across sets of networks.
Figures & tables
Figure 1: Clean accuracy and adversarial performance gaps for FGSM, ℓ2 and ℓ∞ attacks across training perturbation budgets. A gap of 0 means clean accuracy matches accuracy under attack and larger gaps mean a larger drop under attack.
Figure 2: Normalized displacement between matched clean and perturbed representations in 3D M-PHATE space; larger values mean the perturbation moved representations further relative to the network’s own scale. Rows: trained network. Columns: budget and attack used to perturb, ordered by ascending vanilla displacement. Each cell represents the network trained at the budget and perturbations of that attack at the same budget. Mean ± s.d., three seeds.
Figure 3: (a) Proportion of perturbed test points whose clean counterpart lies within its ten nearest neighbors in 3D PHATE space. (b) MMD permutation test p -values between clean and perturbed points in 3D PHATE space. Recovery near 1 and p near 1 both mean the perturbation left the representation essentially unchanged. For both panels, rows denote the trained network and applied evaluation attack, and columns denote the perturbation budget. Mean ± s.d. across three seeds.
Figure 4: M-PHATE embeddings of h1 (top) and h2 (bottom) across all 50 training epochs at the High budget, one network per column. Each panel embeds the full trajectory of activations for the clean test batch, colored by epoch.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Value
Architecture
MLP, 784→128→64→10 , ReLU
Optimizer
Adam
Learning rate η
0.001
Mini-batch size B
64
Epochs
50
Loss
cross-entropy
Appendix
Table 1: Training configuration, identical for all networks and budgets.
Budget
FGSM ε
l2ε
l∞ε
Extreme Low
2/255
0.007843
0.125
2/255
0.007843
Low
4/255
0.015686
0.250
4/255
0.015686
Medium
8/255
0.031373
0.500
8/255
0.031373
High
16/255
0.062745
1.000
16/255
0.062745
Extreme High
32/255
0.125490
2.000
32/255
0.125490
Appendix
Table 2: Perturbation budgets. FGSM and l∞ use the same pixel-scale ε , given both as a fraction of 255 and as a decimal; l2 uses a separate scale.
Budget
Attack
ε
mean ∥δ∥2
mean ∥δ∥∞
mean ∥δ∥1
Extreme Low
FGSM
0.007843
0.1693
0.0078
3.6696
l2
0.125000
0.0965
0.0169
1.4967
l∞
0.007843
0.0411
0.0071
0.6388
Low
FGSM
0.015686
0.3348
0.0157
7.2238
l2
0.250000
0.1918
0.0339
2.9683
l∞
0.015686
0.0820
0.0143
1.2742
Appendix
Table 3: Measured perturbation magnitudes, averaged over the 1000-image evaluation batch. δ=xadv−x is measured after clamping to [0,1] , so these are the perturbations actually applied rather than the nominal budgets.
Budget
Trained
Clean
Accuracy under attack
Gap (clean − attacked)
FGSM
l2
l∞
FGSM
l2
l∞
Extreme Low
Vanilla
0.893 ± 0.002
0.855 ± 0.002
0.862 ± 0.007
0.882 ± 0.005
0.039 ± 0.002
0.031 ± 0.006
0.011 ± 0.003
FGSM
0.900 ± 0.003
0.869 ± 0.007
0.871 ± 0.006
0.890 ± 0.004
0.031 ± 0.004
0.030 ± 0.003
0.010 ± 0.002
l2
0.892 ± 0.005
0.863 ± 0.003
0.867 ± 0.003
0.882 ± 0.002
0.030 ± 0.005
0.025 ± 0.004
0.010 ± 0.004
l∞
0.895 ± 0.007
0.862 ± 0.001
0.868 ± 0.004
0.884 ± 0.007
0.033 ± 0.008
0.027 ± 0.004
0.011 ± 0.002
Low
Vanilla
0.893 ± 0.002
0.806 ± 0.002
0.828 ± 0.004
0.868 ± 0.008
0.087 ± 0.000
0.066 ± 0.005
0.025 ± 0.008
Appendix
Table 4: Complete cross-attack accuracy grid. Every network is evaluated on the clean batch and under all three attacks at that budget; attacks are generated white-box against the network being evaluated. Mean ± s.d. over seeds 0, 1, 2.
Trained
Attack
Mean Disp.
Median Clean–Clean
Norm. Disp.
kNN Rec.
MMD 2
p
Extreme Low Budget
Vanilla
FGSM
1.144 ± 0.041
30.977 ± 0.787
0.0370 ± 0.0021
1.000 ± 0.000
0.00017 ± 0.00003
1.000 ± 0.000
Vanilla
l2
0.890 ± 0.038
30.977 ± 0.787
0.0288 ± 0.0018
1.000 ± 0.000
0.00007 ± 0.00002
1.000 ± 0.000
Vanilla
l∞
0.376 ± 0.016
30.977 ± 0.787
0.0122 ± 0.0008
1.000 ± 0.000
0.00001 ± 0.00000
1.000 ± 0.000
FGSM
FGSM
0.871 ± 0.012
30.196 ± 0.449
0.0288 ± 0.0001
1.000 ± 0.000
0.00011 ± 0.00001
1.000 ± 0.000
FGSM
l2
0.751 ± 0.009
30.196 ± 0.449
0.0249 ± 0.0001
1.000 ± 0.000
0.00005 ± 0.00001
1.000 ± 0.000
Appendix
Table 5: Geometry in the raw 64-dimensional penultimate (h2) space. Mean displacement and median clean–clean pairwise distance are reported so normalized values can be recomputed as their ratio. Mean ± s.d. over seeds 0, 1, 2.
Trained
Attack
Mean Disp.
Median Clean–Clean
Norm. Disp.
kNN Rec.
MMD 2
p
Extreme Low Budget
Vanilla
FGSM
0.00169 ± 0.00028
0.02985 ± 0.00033
0.0565 ± 0.0088
0.899 ± 0.037
0.00125 ± 0.00076
0.432 ± 0.405
Vanilla
l2
0.00150 ± 0.00033
0.02980 ± 0.00031
0.0503 ± 0.0105
0.912 ± 0.039
0.00126 ± 0.00099
0.480 ± 0.448
Vanilla
l∞
0.00097 ± 0.00027
0.02969 ± 0.00034
0.0328 ± 0.0087
0.921 ± 0.026
0.00087 ± 0.00059
0.588 ± 0.365
FGSM
FGSM
0.00113 ± 0.00015
0.02971 ± 0.00026
0.0381 ± 0.0050
0.953 ± 0.018
0.00040 ± 0.00023
0.891 ± 0.139
FGSM
l2
0.00119 ± 0.00020
0.02969 ± 0.00025
0.0400 ± 0.0067
0.936 ± 0.014
0.00059 ± 0.00042
0.755 ± 0.305
Appendix
Table 6: Geometry in the 3-component M-PHATE embedding of the joint clean/perturbed tensor. Columns match Table 5 . Mean ± s.d. over seeds 0, 1, 2.
Figure 5: M-PHATE embeddings of training trajectories for layers h1 (top block) and h2 (bottom block), with one row per network and one column per perturbation budget. Each point is one clean test image at one epoch, colored by epoch. The vanilla row is identical across all five columns, since that network never sees an attack.
Figure 6: Embeddings of the vanilla network’s penultimate layer. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across attacks (rows) and perturbation budgets (columns).
Figure 7: Embeddings under the FGSM attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Figure 8: Embeddings under the l2 attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Figure 9: Embeddings under the l∞ attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Adversarial Training (AT) is one of the most effective methods for developing robust deep neural networks (DNNs). However, AT faces a trade-off problem between clean accuracy and adversarial robustness. In this work, we reveal a surprising phenomenon for the first time: Varying input perturbation intensities for training samples near decision boundaries in AT have minimal impact on model robustness. This finding directly exposes the inconsistency between accuracy and robustness score fluctuations, leading us to identify the misalignment between input and latent spaces as a critical driver of the robustness-accuracy trade-off. To mitigate this misalignment for harmonizing accuracy and robustness, we define Robust Alignment as a new AT target, encouraging the model perception to change with input perturbations provided the final label prediction remains unchanged, which can be achieved via two novel ideas. First, we suggest a reduced and fixed perturbation intensity for those boundary samples, which facilitates the model to utilize the perturbations as learnable patterns, instead of noises that complicate decision boundaries meaninglessly. Second, we propose a Domain Interpolation Consistency Adversarial Regularization (DICAR), based on rigorous theoretical derivations, which explicitly introduces semantic alignment between input and latent spaces into AT. Based on these two ideas, we end up with a new Robust Alignment Adversarial Training (RAAT) method, effectively harmonizing accuracy and robustness. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, PreActResNet-18, and WideResNet-28-10 demonstrate the effectiveness of RAAT in improving the trade-off beyond four common baselines and a total of 14 related state-of-the-art (SOTA) works.
As demands for resource efficiency and safety in modern neural networks intensify, substantial research effort has gone into model compression and adversarial robustness. Yet despite progress on each in isolation, a systematic understanding of how compressibility shapes robustness remains elusive. In this paper, we develop a principled framework to analyze how different forms of structured compressibility - such as neuron-level and spectral compressibility - affect adversarial robustness. We show that structured compressibility can induce a small number of highly sensitive directions in the representation space, which adversaries can exploit to construct effective perturbations. Our analysis yields a robustness bound that reveals how neuron and spectral compressibility impact ℓ∞ and ℓ2 robustness via their effects on the learned representations. Crucially, the vulnerabilities we identify arise irrespective of how compressibility is achieved - whether via regularization, architectural bias, or learning dynamics. Through empirical evaluations across synthetic and realistic tasks, we confirm our theoretical predictions, and further demonstrate that these vulnerabilities persist under adversarial training and transfer learning, and contribute to the emergence of universal adversarial examples. Our findings show a fundamental tension between structured compressibility and robustness and highlight new pathways for designing models that are efficient and safe.
Melih Barsbey, Antônio H. Ribeiro, Umut Şimşekli +1
Department of Computing, Imperial College London, UK · Department of Information Technology, Uppsala University, Sweden · INRIA, CNRS, Département d’Informatique de l’Ecole Normale Supérieure / PSL, France
Adversarial attacks - input perturbations imperceptible to humans that fool neural networks - remain both a persistent failure mode in machine learning, and a phenomenon with mysterious origins. To shed light, we define and analyze a network's perceptual manifold (PM) for a class concept as the space of all inputs confidently assigned to that class by the network. We find, strikingly, that the dimensionalities of neural network PMs are orders of magnitude higher than those of natural human concepts. Since volume typically grows exponentially with dimension, this suggests exponential misalignment between machines and humans, with exponentially many inputs confidently assigned to concepts by machines but not humans. Furthermore, this provides a natural geometric hypothesis for the origin of adversarial examples: because a network's PM fills such a large region of input space, any input will be very close to any class concept's PM. Our hypothesis thus suggests that adversarial robustness cannot be attained without dimensional alignment of machine and human PMs, and therefore makes strong predictions: both robust accuracy and distance to any PM should be negatively correlated with the PM dimension. We confirmed these predictions across 18 different networks of varying robust accuracy. Crucially, we find even the most robust networks are still exponentially misaligned, and only the few PMs whose dimensionality approaches that of human concepts exhibit alignment to human perception. Our results connect the fields of alignment and adversarial examples, and suggest the curse of high dimensionality of machine PMs is a major impediment to adversarial robustness.
Alessandro Salvatore, Stanislav Fort, Surya Ganguli
Department of Applied Physics, Stanford University, CA, USA · Aisle, San Francisco, CA, USA