Adversarial Training Without Input Gradients via Low-Rank Householder Expansions

Tiana C. Johnson; Donsub Rim

cs.LGAug 27, 2026

Adversarial Training Without Input Gradients via Low-Rank Householder Expansions

Authors: Tiana C. Johnson, Donsub Rim

Abstract

This work concerns adversarial training against the small-norm adversarial examples that arise from the inherent input instability of a trained deep neural network. Examples in this class are small as measured in the relative $\ell^2$ -norm, and therefore lie in the neighborhood of the input on which the model acts approximately linearly, the regime in which the perturbation remains imperceptible. We first show that such examples can be computed directly from the trained network parameters, without input gradient iterations, by means of a linearization called the low-rank Householder expansion (LRHE). The expansion describes the composed affine map rather than any individual layer, and the directions it identifies are read from the activation pattern already available in the forward pass. We then propose a simple adversarial training scheme built on this construction. No differentiation with respect to the input is performed at any point: training requires only additional forward evaluations, with weight parameters updated by the standard backward pass, and the inner maximization of the usual min-max formulation is eliminated entirely. That such a regularizer exists is our main finding: the methods that dispense with the inner search all obtain their local geometry by differentiating with respect to the input, and we show this is not necessary. The regularizer costs the equivalent of $2.8$ PGD steps per epoch, an $8.7\times$ reduction relative to 40-step adversarial training on MNIST and below the cost of 3-step training. The resulting models match three-step PGD adversarial training for relative $\ell^2$ budgets $\varepsilon \le 0.02$ and 40-step training for $\varepsilon \le 0.012$ , falling away beyond, consistent with the locality of the expansion.

Figures & tables

Figure 1. Diagram of a forward pass that computes the output basis

(\phi_{\ell})_{\ell}

( 3.5 ). The computation of the output basis involves additional partial forward passes that omit the nonlinearities. The output basis is then used in the regularization term

\mathcal{R}

( 3.12 ) to penalize large changes in the output logits in the direction of the output basis.

Figure 2. An illustration relating LRHE input basis to adversarial examples. The surface denotes the model logit output for the incorrect class, which increases quickly in one of the input-basis directions. In the linear regime, when the input is perturbed in the direction of $\psi_{1}$ the logit changes in the direction of $\phi_{1}$ resulting in a misclassification.

Figure 3. A diagram of the CNN architecture used for the MNIST experiments (left). The feedforward form of the same architecture, where the max-pooling layers are converted into a ReLU network, is shown alongside (right). Four new ReLUs are added to the feedforward form, which implies additional output basis vectors

\phi_{\ell}

will be computed.

Panel of Figure 4 — Figure 4. Adversarial examples found for the standard CNN model using PGD with relative $\ell^{2}$ norm $\varepsilon=0.1$ .

$\alpha$	Adv. acc.
$0.5$	99.04 (0.44)
$1.0$	98.84 (0.64)
$1.5$	98.70 (0.78)
$2.0$	98.16 (1.32)
$2.5$	97.90 (1.58)
$3.0$	96.45 (3.03)

Table 1. Adversarial accuracy of the clean model after LRHE basis attack for each step-size

\alpha

( 5.1 ). Difference from clean accuracy of 99.48% is in the parentheses.

Panel of Figure 5 — Figure 5. LRHE basis attack results. Six adversarial examples found for varying step-size $\alpha$ . The original image and correctly predicted label is shown, alongside the perturbed input and the perturbation. The corresponding logit scores and the changes are also shown below the examples.

Panel of Figure 6 — Figure 6. Orthogonal complement projection of adversarial examples leads to corrections. Original test image, the adversarial example, and the projected example are shown, along with the differences. Output logit scores and corresponding changes are also shown.

Panel of Figure 7 — Figure 7. LRHE input and output bases for original and adversarially trained models. Top two examples are from the LRHE of the original model, and bottom two examples from the adversarially trained model.

		Original	LRHE-AT	PGD-40-AT	PGD-3-AT
	Test	99.48	99.53	99.46	99.43
$\ell^{2}$ -rel. adv.	$\varepsilon=0.002$	99.45 ( 0 0.03)	99.48 ( 0 0.05)	99.40 ( 0 0.06)	99.34 ( 0 0.09)
	$\varepsilon=0.004$	99.42 ( 0 0.06)	99.40 ( 0 0.13)	99.36 ( 0 0.10)	99.31 ( 0 0.12)
	$\varepsilon=0.006$	99.34 ( 0 0.14)	99.38 ( 0 0.15)	99.28 ( 0 0.18)	99.26 ( 0 0.17)
	$\varepsilon=0.008$	99.25 ( 0 0.23)	99.31 ( 0 0.22)	99.25 ( 0 0.21)	99.25 ( 0 0.18)
	$\varepsilon=0.010$	99.17 ( 0 0.31)	99.27 ( 0 0.26)	99.20 ( 0 0.26)	99.19 ( 0 0.24)

Table 2. Adversarial accuracy (percentage) under relative

\ell^{2}

perturbations (

\lVert{\delta x}\rVert_{2}\leq\varepsilon\|x\|_{2}

; with

\|x\|_{2}\approx 9.2

on MNIST,

\varepsilon=0.150

corresponds to an absolute radius

\|\delta x\|_{2}

\approx 1.4

) and under

\ell^{\infty}

perturbations. All defended models were trained at

\varepsilon=0.075

in the relative

\ell^{2}

norm; the

\ell^{\infty}

block therefore measures cross-norm transfer. Parenthesized values give accuracy lost relative to each model’s own clean accuracy.

	LRHE-AT
Test accuracy	99.53
White-box adv. acc.	94.94 (4.59)
Transfer from second LRHE-AT model	97.28 (2.72)
Transfer from original model	98.47 (1.53)

Table 3. Black-box attack for the adversarially trained model. Adversarial accuracy (percentage) computed for

\ell^{2}

-relative

\varepsilon=0.075

PGD steps	1	2	5	10	20	40
Wall time (s)	7.8	10.6	18.8	32.3	58.0	110.5

Table 4. Wall time per epoch during PGD adversarial training. Batch size is 30. The wall time was measured using standard PyTorch implementation on NVIDIA GeForce RTX 3090 with 24GB of memory.

Explore similar work

May 14, 2026cs.LG

Fast Adversarial Attacks with Gradient Prediction

Generating adversarial examples at scale is a core primitive for robustness evaluation, adversarial training, and red-teaming, yet even "fast" attacks such as FGSM remain throughput-limited by the cost of a backward pass. We introduce a family of attacks that eliminates the backward pass by predicting the input gradient from forward-pass hidden states via a lightweight linear regression. Theoretically, we derive exact affine conditional gradient means, showing optimality in the (idealized) NTK regime. Empirically, our methods work when applied to practical finite-width models; we recover much of FGSM's attack performance while using only a small fraction of the time, corresponding to a

532\%

increase in throughput. These results suggest gradient prediction as a simple and general route to significantly faster adversarial generation under realistic wall-clock constraints.

Kamil Ciosek, Aleksandr V. Petrov, Nicolò Felicioni +1

Spotify

Adversarial Attacks Neural Network Robustness Adversarial Examples

Ask PDF

Jun 19, 2026cs.LG

Robustness Cannot be Reduced to Regularization: Studying Adversarial Training Beyond the Linear Case

The vulnerability of ML models to adversarial examples has recently emerged as a major concern. While adversarial training is one of the most effective countermeasures to this issue, its high computational cost remains an obstacle to practical deployment. Recent progress in reducing this cost has relied, in the case of linear models, on a formal equivalence between the adversarial risk and a simpler form of regularized risk. This enabled significantly more efficient training procedures, which naturally raises the question of whether such an equivalence can be extended beyond linear models. In this work, we formally show that no such equivalence is possible for two-layer networks. Our proofs proceed via a reduction to key properties that fundamentally separate the adversarial risk from any simple regularized risk which would only exhibit a weak form of data dependence. Beyond this setting, we provide empirical evidence on Wide-ResNets indicating that the same type of impossibility persists in deeper and more expressive architectures.

David A. R. Robin, Rafael Pinot, Yann Chevaleyre

LAMSADE, Dauphine · Universit´e Paris Dauphine PSL Research University · LPSM, Jussieu +1

Adversarial Training Neural Network Robustness Adversarial Robustness

Ask PDF

May 19, 2025cs.LG

A Few Large Shifts: Layer-Inconsistency Based Minimal Overhead Adversarial Example Detection

Deep neural networks (DNNs) are highly susceptible to adversarial examples---small, malicious perturbations that can cause incorrect predictions. We introduce a lightweight, plug-in detector that uses internal layer-wise inconsistencies within the target model and requires only benign data for fitting and calibration. The approach is motivated by the A Few Large Shifts Assumption, an empirical hypothesis that adversarial perturbations often produce large, localized growth in representation changes across a small number of consecutive layers, connecting adversarial behavior to layer-wise Lipschitz continuity. We develop two complementary scores---Recovery Testing (RT) for intermediate-layer inconsistency and Logit-layer Testing (LT) for augmentation-induced output instability---and fuse them through RLT. Across CIFAR-10, CIFAR-100, and ImageNet, RLT achieves strong detection performance under standard attacks with substantially lower overhead than detector families requiring external encoders or reference-set retrieval. We further study its behavior under adaptive attacks, at low false-positive operating points, and under benign distribution shifts. The code is available here: https://github.com/c0510gy/AFLS-AED.

Sanggeon Yun, Ryozo Masukawa, Hyunwoo Oh +2

Department of Computer Science University of California, Irvine · Whiting School of Engineering Johns Hopkins University

Adversarial Attacks Adversarial Examples Adversarial Example Detection

Ask PDF