Adversarial training is often plagued by "pseudo-attacks", perturbations that significantly increase the loss value but fail to truly alter the model's confidence structure. To address this, we propose two complementary approaches to directly attack and defend against confidence structure deformation. First, building upon the traditional loss maximization perspective, we introduce a hybrid objective that integrates cross-entropy and KL divergence, encouraging adversarial examples to simultaneously disrupt prediction accuracy and distribution consistency. Second, we geometrically reconstruct adversarial training by generating perturbations that maximize the distribution distance on the confidence manifold. Theoretical analysis shows that the optimal perturbation direction aligns with the principal eigenvectors of the Fisher information matrix. We also propose an efficient algorithm to avoid high-dimensional matrix computations. Experiments on multiple datasets and model architectures demonstrate that our method outperforms existing methods in adversarial robustness and robust transferability, particularly excelling in structural perturbations within high-confidence regions.
Figures & tables
Figure 1: Most previous methods add perturbation noise to a single discriminant value f(x) for adversarial training (top), while our SCP adds structured perturbations to the statistical manifold {p(y∣x;θ)∣x∈X} for adversarial training (bottom).
Dataset
Method
ResNet-18
ResNet-34
ResNet-50
STD ↑
CW-10 ↑
AA ↑
ASR ↓
STD ↑
CW-10 ↑
AA ↑
ASR ↓
STD ↑
CW-10 ↑
AA ↑
ASR ↓
CIFAR-10
PGD [ 8 ]
78.87%
46.20%
62.37%
20.92%
77.74%
50.26%
64.24%
17.37%
75.38%
48.30%
62.49%
17.10%
TRADES [ 12 ]
80.72%
48.74%
64.93%
19.56%
82.47%
49.15%
67.31%
18.38%
80.20%
50.09%
64.69%
19.34%
MIFGSM [ 29 ]
81.92%
50.98%
64.11%
21.74%
81.35%
53.30%
65.06%
20.02%
78.86%
49.39%
61.78%
21.66%
NIFGSM [ 30 ]
81.28%
52.07%
65.08%
19.93%
82.63%
52.26%
67.39%
18.44%
81.33%
54.91%
67.13%
17.46%
DIFGSM [ 31 ]
81.51%
50.18%
64.23%
21.20%
80.76%
53.31%
67.03%
17.00%
79.61%
52.47%
66.00%
17.10%
Table 1: Comparison of adversarial training methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet. Red : best; blue : second best.
Dataset
Method
Model
STD ↑
AA ↑
ASR ↓
CIFAR-10
RPF [ 35 ]
PreActResNet-18
84.21%
53.89%
36.01%
RAMP [ 36 ]
PreActResNet-18
82.10%
66.60%
18.88%
GAWP [ 37 ]
PreActResNet-18
88.07%
68.89%
21.78%
LOAT [ 38 ]
ResNet-18
77.97%
55.66%
28.61%
RAT [ 39 ]
ResNet-18
81.87%
60.19%
26.48%
AR-AT [ 40 ]
ResNet-18
86.80%
62.45%
28.05%
Table 2: Comparison with recent adversarial training baselines on CIFAR-10. Red : best; blue : second best.
Dataset
Method
DenseNet121
ViT-B/16
STD ↑
CW-10 ↑
AA ↑
ASR ↓
STD ↑
CW-10 ↑
AA ↑
ASR ↓
CIFAR-10
TRADES [ 12 ]
76.24%
48.56%
62.27%
18.32%
71.60%
47.37%
48.55%
32.19%
MIFGSM [ 29 ]
77.35%
49.79%
61.03%
21.10%
74.14%
50.36%
52.39%
29.34%
DIFGSM [ 31 ]
77.75%
49.21%
62.68%
19.38%
70.90%
47.43%
48.53%
31.55%
PIFGSM [ 33 ]
78.87%
49.41%
63.33%
19.70%
68.22%
45.13%
46.32%
32.10%
SPP (Ours)
81.62%
60.38%
69.15%
15.28%
68.12%
44.55%
45.37%
33.40%
Table 3: Comparison of DenseNet121 and ViT-B/16 on CIFAR-10. Red : best; blue : second best.
Figure 2: Diagnostic analysis of gradient variation on CIFAR-10.
Method
DenseNet121
ViT-B/16
PGD
TRADES
PIFGSM
PGD
TRADES
PIFGSM
TRADES [ 12 ]
92.94%
93.17%
92.63%
91.68%
92.29%
90.58%
MIFGSM [ 29 ]
92.41%
92.54%
90.59%
92.03%
92.58%
90.94%
DIFGSM [ 31 ]
92.80%
92.99%
91.37%
92.02%
92.87%
91.26%
PIFGSM [ 33 ]
93.03%
92.61%
91.52%
91.83%
92.12%
90.73%
SPP (Ours)
94.23%
94.33%
93.89%
91.13%
91.68%
90.22%
Table 4: Cross-model transfer results on CIFAR-10. AutoAttack-L2 adversarial examples are generated on ResNet-18 source models and evaluated on DenseNet-121 and ViT-B/16 target models. Each entry reports robust accuracy ↑ . The best and second-best results in each column are highlighted in red and blue , respectively.
Method
Median λmax↓
P(λmax≤0.01)↑
P(λmax≤0.1)↑
PGD
0.1438
33.96%
47.62%
TRADES
0.3011
24.36%
40.93%
PIFGSM
0.1332
33.10%
47.84%
MIFGSM
0.2257
26.83%
43.39%
SPP
0.1131
32.61%
48.98%
SCP
0.0287
45.03%
56.06%
Table 5: FIM maximum eigenvalue statistics on CIFAR-10 using ResNet-18.
Figure 3: 3D visualization comparison of confidence shift on CIFAR-10 (left: CW, right: SCP). The X - Y plane represents t-SNE projected features, and the Z -axis represents the prediction confidence.
Method
Space complexity
Time complexity
SPP
O(d)
O(d)
SCP (direct)
O(d2)
O(Kd2+d3)
SCP (indirect)
O(dK)
O(dK2+K3)
Table 6: Time and space complexity of SPP and SCP with direct and indirect decomposition.
Loss Type
Clean Acc ↑
AA ↑
ASR ↓
CE only
78.87%
62.37%
20.92%
KL only
80.72%
64.93%
19.56%
CE + KL (SPP)
81.90%
70.15%
14.35%
Table 7: Comparison of the hybrid loss SPP(CE+KL) with PGD(CE) and TRADES(KL) on CIFAR10 using ResNet-18.
Figure 4: SPP lambda ablation on CIFAR-10 using ResNet-18.
Building on a probabilistic perspective in which adversarial examples arise from the overlap between a distance-based distribution pdis and a victim-classifier-induced distribution pvic, we start from a simple intuition: adversarial examples become harder to generate when these two distributions are pushed apart, as their overlap becomes smaller, thereby increasing robustness. This intuition naturally motivates a KL-based robustness objective. We then prove that KL(pdis∥pvic)−logZvic is a lower bound on probabilistic robustness (PR), where Zvic denotes the normalizing constant of pvic. Since PR is generally intractable to compute directly, maximizing this KL-based lower bound provides a tractable surrogate objective for improving PR. We further show that this objective recovers a scaled form of adversarial training, offering a probabilistic interpretation of adversarial training and a principled route to robustness improvement. We call the resulting method probabilistic adversarial training. Experiments show that it consistently improves PR, and ablation studies demonstrate that the induced scaling factor can even enhance the PR of non-probabilistic adversarial training methods.
Transfer-based adversarial attacks rely on surrogate models to craft perturbations, yet often overfit the surrogate's decision boundary. To address this problem, we propose Inverse Knowledge Distillation (IKD), a simple and attack-agnostic mechanism that maximizes the prediction-distribution discrepancy between benign and adversarial samples on the surrogate model. IKD uses a CE/KL-equivalent soft-label objective to push adversarial predictions away from a fixed benign prediction anchor and enrich the attack with Fisher-sensitive surrogate directions. We prove that, under a matched fixed-anchor implementation, soft-label cross-entropy and KL divergence differ only by a constant entropy term and therefore induce identical gradients, Hessians, and adversarial optimization trajectories. Our information-geometric analysis further derives a quantitative lower bound on dominant Fisher-subspace overlap between surrogate and target models from local same-task stability and a Fisher eigengap, and establishes a sufficient target-margin crossing condition under oriented gradient coherence and target smoothness. This analysis connects IKD's surrogate Fisher sensitivity to cross-model transfer. In contrast, mean squared error uses a different Euclidean pullback in output probability space. IKD integrates seamlessly with standard gradient-based attacks without modifying their optimization pipelines. Extensive ImageNet experiments demonstrate consistent black-box gains across CNN, ViT, and defended models, while ablations confirm CE and KL equivalence and the pronounced disadvantage of MSE. These results establish IKD as an effective and lightweight component for improving adversarial transferability. Code is available at https://github.com/ImmortalTing/IKD.
Wenyuan Wu, Yuan Sun, Yingke Chen +4
College of Computer Science, Sichuan University, China · Department of Computer and Information Sciences, Northumbria University, UK · School of Artificial Intelligence, Sichuan University, China
Fast Adversarial Training (FAT) has proven effective in enhancing model robustness by encouraging networks to learn perturbation-invariant representations. However, FAT often suffers from catastrophic overfitting (CO), where the model overfits to the training attack and fails to generalize to unseen ones. Moreover, robustness oriented optimization typically leads to notable performance degradation on clean inputs, and such degradation becomes increasingly severe as the perturbation budget grows. In this work, we conduct a comprehensive analysis of how guidance strength affects model performance by modulating perturbation and supervision levels across distinct confidence groups. The findings reveal that low confidence samples are the primary contributors to CO and the robustness accuracy trade off. Building on this insight, we propose a Distribution-aware Dynamic Guidance (DDG) strategy that dynamically adjusts both the perturbation budget and supervision signal. Specifically, DDG scales the perturbation magnitude according to the sample confidence at the ground truth class, thereby guiding samples toward consistent decision boundaries while mitigating the influence of learning spurious correlations. Simultaneously, it dynamically adjusts the supervision signal based on the prediction state of each sample, preventing overemphasis on incorrect signals. To alleviate potential gradient instability arising from dynamic guidance, we further design a weighted regularization constraint. Extensive experiments on standard benchmarks demonstrate that DDG effectively alleviates both CO and the robustness accuracy trade off.
Mengnan Zhao, Lihe Zhang, Bo Wang +3
Anhui University, Anhui, China · Dalian University of Technology, Liaoning, China · Zhejiang University, Zhejiang, China +1