Representational Geometry as a Diagnostic for Adversarial Robustness
Authors: Brandon Ismalej, Alex Cloninger, Gal Mishne
Organizations: Department of Computer Science, California State University, Northridge, Northridge, California, USA · Department of Mathematics, Halıcıo˘glu Data Science Institute, University of California San Diego, La Jolla, California, USA
Adversarial training is a common technique for making neural networks more robust to small, intentionally crafted input perturbations, and robustness is commonly measured through accuracy under attack. By exploring inter-network representations across different adversarial training objectives, we can analyze whether multiple networks organize their latent representations similarly. We study Multilayer Perceptrons trained on MNIST under three adversarial training regimes and compare the resulting population training dynamics and geometry to those of a vanilla network. We use M-PHATE to embed data samples across training epochs and examine how the geometry of the data changes during training under adversarial objectives. On penultimate-layer representations we measure displacement between clean and perturbed points, distributional distance through maximum mean discrepancy, and nearest-neighbor recovery. Our visualizations and quantitative analyses show that adversarial training leads to structural changes in how networks organize internal data representations, and robust accuracy and representational stability can rank the same defenses in opposite ways. These findings support a discussion of re-evaluating how practitioners approach the auditing of model robustness by treating latent representations and their training trajectories as diagnostic artifacts across sets of networks.
Figures & tables
Figure 1: Clean accuracy and adversarial performance gaps for FGSM, ℓ2 and ℓ∞ attacks across training perturbation budgets. A gap of 0 means clean accuracy matches accuracy under attack and larger gaps mean a larger drop under attack.
Figure 2: Normalized displacement between matched clean and perturbed representations in 3D M-PHATE space; larger values mean the perturbation moved representations further relative to the network’s own scale. Rows: trained network. Columns: budget and attack used to perturb, ordered by ascending vanilla displacement. Each cell represents the network trained at the budget and perturbations of that attack at the same budget. Mean ± s.d., three seeds.
Figure 3: (a) Proportion of perturbed test points whose clean counterpart lies within its ten nearest neighbors in 3D PHATE space. (b) MMD permutation test p -values between clean and perturbed points in 3D PHATE space. Recovery near 1 and p near 1 both mean the perturbation left the representation essentially unchanged. For both panels, rows denote the trained network and applied evaluation attack, and columns denote the perturbation budget. Mean ± s.d. across three seeds.
Figure 4: M-PHATE embeddings of h1 (top) and h2 (bottom) across all 50 training epochs at the High budget, one network per column. Each panel embeds the full trajectory of activations for the clean test batch, colored by epoch.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Value
Architecture
MLP, 784→128→64→10 , ReLU
Optimizer
Adam
Learning rate η
0.001
Mini-batch size B
64
Epochs
50
Loss
cross-entropy
Appendix
Table 1: Training configuration, identical for all networks and budgets.
Budget
FGSM ε
l2ε
l∞ε
Extreme Low
2/255
0.007843
0.125
2/255
0.007843
Low
4/255
0.015686
0.250
4/255
0.015686
Medium
8/255
0.031373
0.500
8/255
0.031373
High
16/255
0.062745
1.000
16/255
0.062745
Extreme High
32/255
0.125490
2.000
32/255
0.125490
Appendix
Table 2: Perturbation budgets. FGSM and l∞ use the same pixel-scale ε , given both as a fraction of 255 and as a decimal; l2 uses a separate scale.
Budget
Attack
ε
mean ∥δ∥2
mean ∥δ∥∞
mean ∥δ∥1
Extreme Low
FGSM
0.007843
0.1693
0.0078
3.6696
l2
0.125000
0.0965
0.0169
1.4967
l∞
0.007843
0.0411
0.0071
0.6388
Low
FGSM
0.015686
0.3348
0.0157
7.2238
l2
0.250000
0.1918
0.0339
2.9683
l∞
0.015686
0.0820
0.0143
1.2742
Appendix
Table 3: Measured perturbation magnitudes, averaged over the 1000-image evaluation batch. δ=xadv−x is measured after clamping to [0,1] , so these are the perturbations actually applied rather than the nominal budgets.
Budget
Trained
Clean
Accuracy under attack
Gap (clean − attacked)
FGSM
l2
l∞
FGSM
l2
l∞
Extreme Low
Vanilla
0.893 ± 0.002
0.855 ± 0.002
0.862 ± 0.007
0.882 ± 0.005
0.039 ± 0.002
0.031 ± 0.006
0.011 ± 0.003
FGSM
0.900 ± 0.003
0.869 ± 0.007
0.871 ± 0.006
0.890 ± 0.004
0.031 ± 0.004
0.030 ± 0.003
0.010 ± 0.002
l2
0.892 ± 0.005
0.863 ± 0.003
0.867 ± 0.003
0.882 ± 0.002
0.030 ± 0.005
0.025 ± 0.004
0.010 ± 0.004
l∞
0.895 ± 0.007
0.862 ± 0.001
0.868 ± 0.004
0.884 ± 0.007
0.033 ± 0.008
0.027 ± 0.004
0.011 ± 0.002
Low
Vanilla
0.893 ± 0.002
0.806 ± 0.002
0.828 ± 0.004
0.868 ± 0.008
0.087 ± 0.000
0.066 ± 0.005
0.025 ± 0.008
Appendix
Table 4: Complete cross-attack accuracy grid. Every network is evaluated on the clean batch and under all three attacks at that budget; attacks are generated white-box against the network being evaluated. Mean ± s.d. over seeds 0, 1, 2.
Trained
Attack
Mean Disp.
Median Clean–Clean
Norm. Disp.
kNN Rec.
MMD 2
p
Extreme Low Budget
Vanilla
FGSM
1.144 ± 0.041
30.977 ± 0.787
0.0370 ± 0.0021
1.000 ± 0.000
0.00017 ± 0.00003
1.000 ± 0.000
Vanilla
l2
0.890 ± 0.038
30.977 ± 0.787
0.0288 ± 0.0018
1.000 ± 0.000
0.00007 ± 0.00002
1.000 ± 0.000
Vanilla
l∞
0.376 ± 0.016
30.977 ± 0.787
0.0122 ± 0.0008
1.000 ± 0.000
0.00001 ± 0.00000
1.000 ± 0.000
FGSM
FGSM
0.871 ± 0.012
30.196 ± 0.449
0.0288 ± 0.0001
1.000 ± 0.000
0.00011 ± 0.00001
1.000 ± 0.000
FGSM
l2
0.751 ± 0.009
30.196 ± 0.449
0.0249 ± 0.0001
1.000 ± 0.000
0.00005 ± 0.00001
1.000 ± 0.000
Appendix
Table 5: Geometry in the raw 64-dimensional penultimate (h2) space. Mean displacement and median clean–clean pairwise distance are reported so normalized values can be recomputed as their ratio. Mean ± s.d. over seeds 0, 1, 2.
Trained
Attack
Mean Disp.
Median Clean–Clean
Norm. Disp.
kNN Rec.
MMD 2
p
Extreme Low Budget
Vanilla
FGSM
0.00169 ± 0.00028
0.02985 ± 0.00033
0.0565 ± 0.0088
0.899 ± 0.037
0.00125 ± 0.00076
0.432 ± 0.405
Vanilla
l2
0.00150 ± 0.00033
0.02980 ± 0.00031
0.0503 ± 0.0105
0.912 ± 0.039
0.00126 ± 0.00099
0.480 ± 0.448
Vanilla
l∞
0.00097 ± 0.00027
0.02969 ± 0.00034
0.0328 ± 0.0087
0.921 ± 0.026
0.00087 ± 0.00059
0.588 ± 0.365
FGSM
FGSM
0.00113 ± 0.00015
0.02971 ± 0.00026
0.0381 ± 0.0050
0.953 ± 0.018
0.00040 ± 0.00023
0.891 ± 0.139
FGSM
l2
0.00119 ± 0.00020
0.02969 ± 0.00025
0.0400 ± 0.0067
0.936 ± 0.014
0.00059 ± 0.00042
0.755 ± 0.305
Appendix
Table 6: Geometry in the 3-component M-PHATE embedding of the joint clean/perturbed tensor. Columns match Table 5 . Mean ± s.d. over seeds 0, 1, 2.
Figure 5: M-PHATE embeddings of training trajectories for layers h1 (top block) and h2 (bottom block), with one row per network and one column per perturbation budget. Each point is one clean test image at one epoch, colored by epoch. The vanilla row is identical across all five columns, since that network never sees an attack.
Figure 6: Embeddings of the vanilla network’s penultimate layer. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across attacks (rows) and perturbation budgets (columns).
Figure 7: Embeddings under the FGSM attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Figure 8: Embeddings under the l2 attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Figure 9: Embeddings under the l∞ attack. Joint M-PHATE embeddings of clean (circles) and perturbed (crosses) penultimate-layer representations of 1000 MNIST test images, colored by digit class, across perturbation budgets (rows) and training regimes (columns).
Department of Computing, Imperial College London, UK · Department of Information Technology, Uppsala University, Sweden · INRIA, CNRS, Département d’Informatique de l’Ecole Normale Supérieure / PSL, France