Deep neural networks (DNNs) have achieved remarkable performance in many tasks, yet they often behave as opaque black boxes. Explanation-guided learning (EGL) methods steer DNNs using human-provided explanations or supervision on model attributions. These approaches improve interpretability but typically assume benign inputs and incur heavy annotation costs. In contrast, both predictions and saliency maps of DNNs could dramatically alter facing imperceptible perturbations or unseen patterns. Adversarial training (AT) can substantially improve robustness, but it does not guarantee that model decisions rely on semantically meaningful features. In response, we propose Explanation-Guided Adversarial Training (EGAT), a unified framework that integrates the strength of AT and EGL to simultaneously improve prediction performance, robustness, and explanation quality. EGAT generates adversarial examples on the fly while imposing explanation-based constraints on the model. By jointly optimizing classification performance, adversarial robustness, and attributional stability, EGAT is not only more resistant to unexpected cases, including adversarial attacks and out-of-distribution (OOD) scenarios, but also offer human-interpretable justifications for the decisions. We further formalize EGAT within the Probably Approximately Correct learning framework, demonstrating theoretically that it yields more stable predictions under unexpected situations compared to standard AT. Empirical evaluations on OOD benchmark datasets show that EGAT consistently outperforms competitive baselines in both clean accuracy and adversarial accuracy +37% while producing more semantically meaningful explanations, and requiring only a limited increase +16% in training time.
Figures & tables
Fig. 1: Overview of the proposed EGAT framework: For each original input, EGAT optimizes a complementary objectives of four terms: (1) Traditional classification loss ( Lcls , in gray) is simply calculated by cross entropy; (2) Adversarial training based loss ( Ladv , in red) promotes the robustness for both predictions and explanations; (3) Explanation guided learning loss ( Legl , in purple) adopts explanations as extra supervision signals; (4) Regularizer ( Lreg , in yellow) incorporates mixup strategy to further regulate model behaviors.
Model
Caltech101
LabelMe
SUN09
VOC2007
cAcc
aAcc
Comp
Suff
cAcc
aAcc
Comp
Suff
cAcc
aAcc
Comp
Suff
cAcc
aAcc
Comp
Suff
ERM
98.68
38.51
0.48
38.69
76.17
4.24
13.97
12.78
81.25
7.04
18.63
27.16
84.88
12.96
18.47
18.77
IGR
99.33
71.68
1.47
39.54
77.33
4.00
13.21
9.04
79.32
12.38
14.25
31.28
86.59
15.55
19.94
15.72
IGN
99.29
76.54
2.25
13.82
74.29
1.88
12.37
9.67
83.61
6.66
10.95
25.95
86.37
11.29
20.61
11.19
DMADA
99.05
69.27
0.73
47.36
76.41
0.93
18.29
35.62
81.26
6.97
15.38
40.20
86.26
10.54
8.35
37.90
IRM
95.35
43.48
2.65
30.31
76.41
5.22
7.16
22.14
80.46
10.07
12.23
37.43
85.22
9.60
9.05
38.77
TABLE I: Performance comparison across domains on VLCS (top) and Terra Incognita (bottom). Metrics include clean accuracy (cAcc), adversarial accuracy (aAcc), Comprehensiveness (Comp), and Sufficiency (Suff). We highlight winner and runner-up .
Fig. 2: Adversarial robustness across different attack methods (FGSM, MI-FGSM, PGD, SCA ) and datasets (VLCS and Terra Incognita). The figures show accuracy under varying perturbation budgets, ranging from 0 (clean) to 0.04. EGAT (in red solid curves) shows significantly more stable predictions than baselines under strong adversarial attacks.
Train on
Caltech101
LabelMe
SUN09
VOC2007
Ranking
Test on
LabelMe
SUN09
VOC2007
Caltech101
SUN09
VOC2007
Caltech101
LabelMe
VOC2007
Caltech101
LabelMe
SUN09
Avg. ± Var.
ERM
28.11 [5]
34.66 [5]
48.01 [2]
91.59 [4]
56.40 [4]
66.55 [5]
73.45 [2]
57.54 [6]
63.14 [5]
94.34 [6]
52.64 [6]
71.49 [4]
4.50 ±1.75
IGR
33.01 [3]
43.04 [1]
52.96 [1]
86.28 [6]
54.28 [6]
64.44 [6]
69.91 [3]
60.37 [5]
58.14 [6]
97.87 [2]
62.50 [5]
79.80 [1]
3.75 ±4.19
IGN
26.65 [6]
33.52 [6]
43.70 [5]
88.49 [5]
55.61 [5]
70.18 [1]
67.25 [4]
64.38 [3]
63.51 [4]
97.34 [5]
64.15 [4]
76.57 [2]
4.17 ±2.14
DRE
30.94 [4]
34.75 [4]
45.22 [4]
93.36 [2]
62.34 [2]
67.40 [3]
34.75 [6]
62.34 [4]
70.48 [2]
97.58 [4]
67.40 [1]
61.45 [6]
3.50 ±2.25
SGDrop
34.19 [2]
40.85 [2]
42.96 [6]
92.47 [3]
64.63 [1]
67.22 [4]
40.85 [5]
64.63 [2]
75.76 [1]
98.67 [1]
67.22 [2]
61.85 [5]
2.83 ±2.81
TABLE II: Clean accuracy and corresponding ranking (in brackets) under OOD evaluation across different train-test settings. The average and variance of rankings across various train-test settings are disclosed in the last column.
Model
Caltech101
LabelMe
SUN09
VOC2007
IGR
99.33
33.01
43.04
52.96
SGDrop
99.31
34.19
40.85
42.96
EGAT (base)
99.66
38.15
39.04
46.59
w SG
99.54
40.51
44.50
50.20
w IG
99.68
35.79
42.44
50.94
w Finer-CAM
99.47
38.49
44.44
45.16
TABLE III: Sensitivity analysis under OOD evaluation: models are trained on VOC2007 and tested on all others. EGAT (base) uses Grad-CAM; SG/IG/Finer-CAM replace the explanation module. The last three rows vary the guide map quality.
Model
Caltech101
LabelMe
SUN09
VOC2007
cAcc
aAcc
Comp
Suff
cAcc
aAcc
Comp
Suff
cAcc
aAcc
Comp
Suff
cAcc
aAcc
Comp
Suff
ERM
98.68
38.51
0.48
38.69
76.17
4.24
13.97
12.78
81.25
7.04
18.63
27.16
84.88
12.96
18.47
18.77
w/o EGL
99.64
97.34
3.40
23.68
74.05
55.66
26.28
8.23
82.15
55.23
23.49
25.22
86.29
47.59
24.81
19.29
w/o AT
99.98
63.27
28.53
28.22
80.18
11.32
14.02
14.82
82.58
7.42
19.90
25.62
86.90
17.22
19.21
15.15
EGAT
99.66
98.23
9.65
1.42
78.74
55.66
19.71
13.76
83.61
52.19
16.40
34.19
86.66
45.18
25.70
29.64
TABLE IV: Ablation results on VLCS (top) and Terra Incognita (bottom).
Fig. 3: Adversarial accuracy vs. training time per epoch. Red points denote Pareto-optimal methods and the red curve denotes the Pareto front.
Fig. 4: Visualization of explanation heatmaps under clean and adversarial conditions. We present Grad-CAM heatmaps of four models (columns: ERM, DRE, SGDrop, and EGAT) for three images before and after PGD perturbations. These visualizations illustrate where each model attends when making predictions and how their explanations change under adversarial attacks.
The adversarial robustness of attributions is a fundamental requirement for reliable explainability in deep learning, yet existing approaches typically rely on computationally expensive explicit regularization. In this work, we show that attribution robustness can arise implicitly from the learning dynamics of standard stochastic gradient descent. We theoretically motivate this effect through connections between parameter-space and input-space curvature, and validate it across architectures, datasets, and attribution methods, with negligible computational overhead. In contrast, we prove that such robustness gains often does not transfer to attention-based attribution under softmax normalization, due to inherent entropy constraints, and we validate this limitation experimentally. Finally, we show that replacing softmax attention with kernel-based attention restores the robustness gains in transformer models. Our results highlight learning dynamics as a principled and practical mechanism for robust explainability, and reveal fundamental limitations of attention-based attribution under normalization.
Amir Mehrpanah, Matteo Gamba, Hossein Azizpour
Department of Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden · Department of Computer Science, Brown University, USA · Science for Life Laboratory, Stockholm, Sweden
Gradient-based saliency methods such as Vanilla Gradient (VG) and Integrated Gradients (IG) are widely used to explain image classifiers, yet the resulting maps are often noisy and unstable, limiting their usefulness in high-stakes settings. Most prior work improves explanations by modifying the attribution algorithm, leaving open how the training procedure shapes explanation quality. We take a training-centered view and first provide a curvature-based analysis linking attribution stability to how smoothly the input-gradient field varies locally. Guided by this connection, we study adversarial training and identify a consistent trade-off: it yields sparser and more input-stable saliency maps, but can degrade output-side stability, causing explanations to change even when predictions remain unchanged and logits vary only slightly. To mitigate this, we propose augmenting adversarial training with a lightweight feature-map smoothing block that applies a differentiable Gaussian filter in an intermediate layer. Across FMNIST, CIFAR-10, and ImageNette, our method preserves the sparsity benefits of adversarial training while improving both input-side stability and output-side stability. A human study with 65 participants further shows that smoothed adversarial saliency maps are perceived as more sufficient and trustworthy. Overall, our results demonstrate that explanation quality is critically shaped by training, and that simple smoothing with robust training provides a practical path toward saliency maps that are both sparse and stable.
Adversarial Training (AT) is one of the most effective methods for developing robust deep neural networks (DNNs). However, AT faces a trade-off problem between clean accuracy and adversarial robustness. In this work, we reveal a surprising phenomenon for the first time: Varying input perturbation intensities for training samples near decision boundaries in AT have minimal impact on model robustness. This finding directly exposes the inconsistency between accuracy and robustness score fluctuations, leading us to identify the misalignment between input and latent spaces as a critical driver of the robustness-accuracy trade-off. To mitigate this misalignment for harmonizing accuracy and robustness, we define Robust Alignment as a new AT target, encouraging the model perception to change with input perturbations provided the final label prediction remains unchanged, which can be achieved via two novel ideas. First, we suggest a reduced and fixed perturbation intensity for those boundary samples, which facilitates the model to utilize the perturbations as learnable patterns, instead of noises that complicate decision boundaries meaninglessly. Second, we propose a Domain Interpolation Consistency Adversarial Regularization (DICAR), based on rigorous theoretical derivations, which explicitly introduces semantic alignment between input and latent spaces into AT. Based on these two ideas, we end up with a new Robust Alignment Adversarial Training (RAAT) method, effectively harmonizing accuracy and robustness. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, PreActResNet-18, and WideResNet-28-10 demonstrate the effectiveness of RAAT in improving the trade-off beyond four common baselines and a total of 14 related state-of-the-art (SOTA) works.