Why the Maximum Second Derivative of Activations Matters for Adversarial Robustness

Yunrui Yu; Hang Su; Jun Zhu

cs.LGMar 25, 2026

Why the Maximum Second Derivative of Activations Matters for Adversarial Robustness

Authors: Yunrui Yu, Hang Su, Jun Zhu

Organizations: 1Tsinghua University, Beijing, China.

Abstract

This work investigates the critical role of activation function curvature -- quantified by the maximum second derivative $\max|σ''|$ -- in adversarial robustness. Using the Recursive Curvature-Tunable Activation Family (RCT-AF), which enables precise control over curvature through parameters $α$ and $β$ , we systematically analyze this relationship. Our study reveals a fundamental trade-off: insufficient curvature limits model expressivity, while excessive curvature amplifies the normalized Hessian diagonal norm of the loss, leading to sharper minima that hinder robust generalization. This results in a non-monotonic relationship where optimal adversarial robustness consistently occurs when $\max|σ''|$ falls within 4 to 10, a finding that holds across diverse network architectures, datasets, and adversarial training methods. We provide theoretical insights into how activation curvature affects the diagonal elements of the hessian matrix of the loss, and experimentally demonstrate that the normalized Hessian diagonal norm exhibits a U-shaped dependence on $\max|σ''|$ , with its minimum within the optimal robustness range, thereby validating the proposed mechanism.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of RCT-AF activation functions with tunable rectification strength via $\alpha$ parameter ( $\alpha=1,5,10,15,20$ ). (a) $\beta=0$ : Baseline form. (b) $\beta=1$ : First-order variant. (c) $\beta=2$ : Second-order variant. Across all configurations, increasing $\alpha$ systematically strengthens the asymmetric treatment of negative versus positive inputs, enabling precise control over the activation function’s rectification characteristics.

Panel of Figure 2 — Figure 2: Second derivative analysis of symmetric RCT-AF activation functions for $\alpha\in\{1,5,10,15,20\}$ . (a) $\beta=0$ : $\sigma^{\prime\prime}_{\beta=0}(x)$ yields a single extremum at $x=0$ with value $\alpha/4$ . (b) $\beta=1$ : $\sigma^{\prime\prime}_{\beta=1}(x)$ produces three critical points: a maximum at $x=0$ ( $\alpha/2$ ) and two symmetric minima. (c) $\beta=2$ : $\sigma^{\prime\prime}_{\beta=2}(x)$ exhibits five critical points: a central maximum at $x=0$ ( $\alpha$ ) and four symmetric extremal points. All functions are even and symmetric about the $y$ -axis.

Panel of Figure 3 — Figure 3: Robustness analysis of RCT-AF activations under DAJAT adversarial training on CIFAR-10 with ResNet-18. (a) Robust accuracy vs. $\alpha$ for $\beta=0,1,2$ , with $\alpha$ varying from 1 to 50 in steps of 1. The curves differ significantly across $\beta$ values, indicating that $\alpha$ alone does not determine robustness. (b) Robust accuracy vs. $\max_{x}\sigma^{\prime\prime}(x)$ for the same models, where $\max_{x}\sigma^{\prime\prime}(x)=\alpha/4$ for $\beta=0$ , $\alpha/2$ for $\beta=1$ , and $\alpha$ for $\beta=2$ . When $\max_{x}\sigma^{\prime\prime}(x)<15$ , the curves nearly overlap, showing that maximum curvature, not $\beta$ , primarily governs robustness.

Figure 4: Adversarial robustness vs. maximum second derivative

\max|\sigma^{\prime\prime}|

for ResNet-18 and WideResNet-28-10 trained with TRADES on CIFAR-10 using RCT-AF (

\beta=1

). Both architectures exhibit the same inverted-U relationship, with robustness peaking when

\max|\sigma^{\prime\prime}|

lies between 4 and 10. The wider network (WideResNet-28-10) achieves higher absolute robust accuracy but follows an identical dependence on activation curvature, demonstrating that the identified optimal curvature range is architecture-agnostic.

Figure 5: Comparison of adversarial robustness vs.

\max|\sigma^{\prime\prime}|

across three adversarial training methods—DAJAT, DKL, and TRADES—on CIFAR-10 with ResNet-18 and RCT-AF (

\beta=1

). All methods exhibit the same qualitative behavior: robustness improves with curvature up to an optimum (

\max|\sigma^{\prime\prime}|\approx 4\text{--}10

) and declines thereafter.

Figure 6: Adversarial robustness vs. maximum second derivative

\max|\sigma^{\prime\prime}|

evaluated on CIFAR-100 using ResNet-18 trained with DAJAT and RCT-AF (

\beta=1

). Despite the increased complexity of CIFAR-100 (100 classes), the non-monotonic relationship persists, and optimal performance consistently occurs within

\max|\sigma^{\prime\prime}|\in[4,10]

Panel of Figure 7 — Figure 7: (a) Measured normalized Hessian diagonal norm $\|\operatorname{diag}(\nabla^{2}_{\theta}\hat{L})/p\|_{2}$ versus $\max|\sigma^{\prime\prime}|$ for models trained with standard (non-adversarial) training using the SGD optimizer on CIFAR-10 using ResNet-18 ( $\beta=0,1,2$ ). Here, $p$ denotes the total number of model parameters. The plot reveals that when $\max|\sigma^{\prime\prime}|$ lies between 4 and 10, $\|\operatorname{diag}(\nabla^{2}_{\theta}\hat{L})/p\|_{2}$ reaches its minimum values across all $\beta$ values. When $\max|\sigma^{\prime\prime}|$ exceeds 10, further increases lead to marked rises in the norm, indicating sharper loss landscapes. Conversely, when $\max|\sigma^{\prime\prime}|$ falls below 4, further reductions in curvature also lead to increased norm, suggesting that insufficient curvature can similarly sharpen the landscape. (b) Clean accuracy versus $\max|\sigma^{\prime\prime}|$ for the same set of models shown in (a) (standard training, ResNet-18 on CIFAR-10). When $\max|\sigma^{\prime\prime}|$ falls below 4, further reductions in curvature lead to a clear decrease in clean accuracy, demonstrating that insufficient curvature limits model capacity due to weak rectification and near-linear behavior. Together, these panels—measured from the same models under identical conditions—illustrate the dual effects of activation curvature: insufficient curvature impairs expressivity, while excessive curvature sharpens the loss landscape.

Activation Function	Standard Accuracy	Robust Accuracy	$\max\|\sigma^{\prime\prime}\|$
ReLU	85.23	51.37	$\infty$
Leaky ReLU	84.52	51.24	$\infty$
ELU	78.39	43.11	1.0
GELU	84.88	51.69	0.798
Swish	82.53	48.08	0.5
Mish	82.63	48.75	0.362

Table 1: Comparison of activation functions in adversarial training on CIFAR-10 with ResNet-18. All models are trained with DAJAT and evaluated using AutoAttack. RCT-AF uses optimal parameters:

\max|\sigma^{\prime\prime}|=7

. Note: Leaky ReLU uses a negative slope of 0.01.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Activation Function	GPU Memory (GB)	Training Time (min)
ReLU	0.73	5.88
LeakyReLU	0.73	5.92
ELU	0.73	5.83
SELU	0.73	5.88
GELU	0.99	7.92
Mish	1.25	6.98

Appendix

Table 2: Computational overhead comparison of activation functions under identical training settings (ResNet-18, CIFAR-10, batch size 128, 20 epochs). GPU memory usage and training time are measured on an NVIDIA 2080Ti.

Loss Landscape Sharpness

See all 19 papers

How Learning Dynamics Drive Adversarially Robust Generalization?
Oct 10, 2024Yuelin Xu, Xiao ZhangAdversarial Training PAC-Bayesian Generalization Bounds
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Adversarial Rademacher Complexity of Deep Neural Networks
Nov 27, 2022Jiancong Xiao, Yanbo Fan, Ruoyu Sun +1Neural Network Generalization Neural Network Robustness
The Chinese University of Hong Kong, Shenzhen Shenzhen, China · Nanjing University Suzhou, China
S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights
Mar 1, 2026Gaojie Jin, Xinping Yi, Wei Huang +2Adversarial Training Adversarial Robustness
Department of Computer Science, University of Exeter, Exeter, UK · National Mobile Communications Research Laboratory, Southeast University, Nanjing, China · Department of Computer Science, University of Liverpool, Liverpool, UK

Oct 10, 2024J/K move · Enter open · S save

How Learning Dynamics Drive Adversarially Robust Generalization?

Yuelin Xu, Xiao Zhang

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Adversarial Training PAC-Bayesian Generalization Bounds Neural Network Robustness Adversarial Robustness Neural Network Training Dynamics

Open paper AskPDF arXiv

Despite being widely adopted as a canonical framework for learning robust models, adversarial training suffers from robust overfitting. Existing empirical and theoretical explorations fail to provide a satisfactory mechanistic interpretation of the phenomenon. By modeling adversarial training with momentum SGD as a discrete-time dynamical system, we propose a PAC-Bayesian analytical framework that proves time-resolved robust generalization bounds. Specifically, our framework tracks the closed-form evolution of the posterior mean and covariance under both stationary and non-stationary transient regimes, connecting the model's robust generalization performance to learning rate, local loss geometry, and mini-batch stochastic gradients. By estimating the key quantities associated with the bound, we illustrate the underlying mechanism of robust overfitting. Our framework also shows how adversarial weight perturbation reduces robust generalization gaps by suppressing dominant loss-curvature modes, while suggesting that excessive penalization can be sub-optimal for optimization.

Abstract

Loss Landscape Sharpness

Explore similar work

How Learning Dynamics Drive Adversarially Robust Generalization?

Adversarial Rademacher Complexity of Deep Neural Networks

S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights