cs.LGNov 5, 2026

On Lipschitz Explosion in Deep Neural Networks with Normalization: Consequences for Optimization and Adversarial Robustness

Authors: Ashkan Soleymani, Reyhaneh Hosseinpourkhoshkbari, Hadi Daneshmand, Patrick Jaillet

Organizations: Massachusetts Institute of Technology · The University of Texas at Dallas · University of Virginia

Abstract

Can a network look stable at every layer and still be exponentially unstable as a whole? The Lipschitz constant of a neural network is a basic stability quantity appearing in optimization guarantees, generalization bounds, and robustness certificates. Deep networks naturally admit two such notions, input Lipschitzness, measuring sensitivity to data perturbations, and parameter Lipschitzness, measuring sensitivity to weight perturbations. We prove that, for deep networks with normalization layers, with batch normalization as the canonical example, both Lipschitz constants can grow exponentially with depth and hence with parameter dimension, even under strong per-layer norm control such as ∥Wℓ∥2≤1\|W_\ell\|_2\le 1, and already for linear activations. Thus, a mechanism widely used to stabilize training can create severe worst-case instabilities that are invisible locally, i.e., from layerwise norm bounds alone. Parameter-Lipschitz explosion turns Lipschitz-dependent nonsmooth optimization guarantees into exponential-in-dimension bounds for deep normalized networks, while input-Lipschitz explosion makes worst-case Lipschitz-based generalization and robustness certificates vacuous. The latter also yields a concrete rank-separation mechanism for adversarial vulnerability. The theory predicts that perturbations introducing new singular directions should be amplified much more strongly than perturbations at the same noise level that remain within the input's original singular subspace. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 support this prediction, showing that rank-creating perturbations cause substantially sharper drops in accuracy, confidence, and margins than same-subspace perturbations.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Lipschitz Thinking: Ten Years of Certifiable-by-Design Robust Neural Networks

    Oct 5, 2026Fabio Brau, Giorgio Piras, Maura Pintor +1Neural Network RobustnessCertified Robustness

  2. Theoretical Lower Bounds on the Robustness of Deep ReLU Networks

    Feb 21, 2026Věra KůrkováNeural Network RobustnessAdversarial Robustness

  3. Convex training of Lipschitz-regularized shallow neural networks

    Jun 17, 2026Chao Yin, Antoine Lesage-LandryAdversarial TrainingShallow Neural Networks