Why Does Robustness Reduce Superposition?

Adam Elimadi

cs.LGAug 23, 2026

Why Does Robustness Reduce Superposition?

Authors: Adam Elimadi

Organizations: Independent Researcher

Abstract

The study of adversarial examples and their origins remains an open area of research. Mechanistic interpretability, and superposition in particular, offers new avenues for approaching this problem. Gorton & Lewis (2025) demonstrate that adversarial examples arise from superposition and show empirically that adversarial training reduces superposition, yet provide no mechanistic account of why this occurs. We present an empirical explanation inspired by the feature taxonomy of Ilyas et al. (2019), tracing the following chain of causalities: adversarial training abandons non-robust features, leading to fewer total features to represent, resulting in less superposition.

Figures & tables

Panel of Figure 1 — Figure 1: Superposition and representational power fall together as adversarial training discards features.

Panel of Figure 2 — Figure 2: Adversarial training drops more features than standard training, and the dropped features are the ones with the most costly interference geometry.

Figure 3: Left: Features dropped at each $S\in\mathcal{S}$ under structured data. The adversarially trained model drops exactly $|\mathcal{N}|=70$ at every sparsity level. Right: Mean interference of kept versus dropped features across $\mathcal{S}$ for the adversarially trained model under structured data. Dropped features carry positive net interference; kept features carry negative net interference.

Explore similar work

Supervised Learning Has a Necessary Geometric Blind Spot: Theory, Consequences, and Minimal Repair
Apr 23, 2026Vishal RajputAdversarial Training Representation Geometry
1KU Leuven, Belgium.
Probabilistic Adversarial Training
Sep 30, 2026Andi Zhang, Xingyu Zhao, Siddartha KhastgirAdversarial Training Adversarial Robustness
WMG, University of Warwick · Wuhan University
Reducing information dependency does not cause training data privacy. Adversarially non-robust features do
Jul 14, 2026Rasmus Torp, Shailen K. Smith, Adam BreuerAdversarial Training Neural Network Robustness
Department of Computer Science Dartmouth College · Department of Computer Science & Department of Government Dartmouth College

Apr 23, 2026, cs.LGJ/K move · Enter open · S save

Supervised Learning Has a Necessary Geometric Blind Spot: Theory, Consequences, and Minimal Repair

Vishal Rajput

1KU Leuven, Belgium.

Adversarial Training Representation Geometry Distribution Shift Robustness Neural Network Robustness Adversarial Robustness Corruption Robustness+1

Open paper AskPDF arXiv

PGD adversarial training, the standard robustness method, can reduce Jacobian Frobenius norm yet worsen clean-input geometry (e.g., TDI 1.336 vs. ERM 1.093). We show this is not an implementation artifact but a theorem-level consequence of supervised learning. We prove that any encoder minimizing supervised loss must retain non-zero sensitivity along directions correlated with training labels, including directions that are nuisance at test time. This holds across proper scoring rules, architectures, and dataset sizes. We call this the geometric blind spot of supervised learning. This theorem unifies four empirical phenomena often treated separately: non-robust features, texture bias, corruption fragility, and the robustness-accuracy tradeoff. It also explains why suppressing sensitivity in one adversarial direction can redistribute sensitivity elsewhere. We introduce Trajectory Deviation Index (TDI), a diagnostic of geometric isotropy. Unlike CKA, intrinsic dimension, or Jacobian Frobenius norm alone, TDI captures the failure mode above. In our experiments, PGD attains low Frobenius norm but high TDI, while PMH attains the lowest TDI with one additional training term and no architectural changes. Across seven tasks, BERT/SST-2, and ImageNet ViT-B/16 (backbone family underlying CLIP/DINO/SAM), the blind spot is measurable and repairable. It appears at foundation-model scale, worsens with model scale and task-specific fine-tuning, and is substantially reduced by PMH. PMH also leads on non-Gaussian corruption types (blur/brightness/contrast) without corruption-specific training.

Abstract

Explore similar work

Supervised Learning Has a Necessary Geometric Blind Spot: Theory, Consequences, and Minimal Repair

Probabilistic Adversarial Training

Reducing information dependency does not cause training data privacy. Adversarially non-robust features do