On the Escaping Efficiency of Distributed Adversarial Training Algorithms

Ying Cao; Kun Yuan; Ali H. Sayed

cs.LGSep 14, 2025

On the Escaping Efficiency of Distributed Adversarial Training Algorithms

Authors: Ying Cao, Kun Yuan, Ali H. Sayed

Organizations: Institute of Electrical and Micro Engineering in EPFL, Lausanne · Center for Machine Learning Research (CMLR) in Peking University

Abstract

Adversarial training has been widely studied in recent years due to its role in improving model robustness against adversarial attacks. This paper focuses on comparing different distributed adversarial training algorithms--including centralized and decentralized strategies--within multi-agent learning environments. Previous studies have highlighted the importance of model flatness in determining robustness. To this end, we develop a general theoretical framework to study the escaping efficiency of these algorithms from local minima, which is closely related to the flatness of the resulting models. We show that when the perturbation bound is sufficiently small (i.e., when the attack strength is relatively mild) and a large batch size is used, decentralized adversarial training algorithms--including consensus and diffusion--are guaranteed to escape faster from local minima than the centralized strategy, thereby favoring flatter minima. However, as the perturbation bound increases, this trend may no longer hold. In the simulation results, we illustrate our theoretical findings and systematically compare the performance of models obtained through decentralized and centralized adversarial training algorithms. The results highlight the potential of decentralized strategies to enhance the robustness of models in distributed settings.

Figures & tables

Panel of Figure 1 — Fig. 1: Flatness visualization of the final models trained on the CIFAR-10 dataset. Wider valleys correspond to flatter minima. The terminology random in the title of each plots mean the randomly generated graph. From these figures, we observe that when adversarial perturbations are bounded by $128/255$ under the $\ell_{2}$ norm or by $3/255$ under $\ell_{\infty}$ norm–i.e., when the attack strength is relatively mild–decentralized adversarial training methods consistently yield flatter solutions compared to the centralized counterpart. However, when the attack strength increases to $\epsilon=8/255$ under the $\ell_{\infty}$ norm, this principle could be broken. For example, in panels (c) and (f), the models obtained via diffusion are noticeably sharper than those obtained via centralized training.

Panel of Figure 2 — Fig. 2: Flatness visualization of the final models trained on the CIFAR-100 dataset. The observed phenomenon is consistent with Fig. 1 .

Panel of Figure 3 — Fig. 3: Flatness visualization of the models trained on the Tiny ImageNet-200 dataset. The observed phenomenon is consistent with Fig. 1 .

dataset	graph	K	norm	$\epsilon$	local batch	method	best		final
dataset	graph	K	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-10	random	16	$\ell_{\infty}$	$8/255$	128	centralized	71.11	35.89	71.44	35.86
						consensus	80.95	45.16	82.74	43.05
						diffusion	84.02	44.46	84.38	38.71
					256	centralized	65.95	33.93	66.35	34.06
						consensus	81.18	43.15	82.05	42.75

TABLE I: Clean and robust accuracy of the obtained models evaluated using AutoAttack on CIFAR-10

dataset	graph	K	norm	$\epsilon$	local batch	method	best		final
dataset	graph	K	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-100	random	16	$\ell_{\infty}$	$8/255$	128	centralized	46.95	16.28	47.09	15.47
						consensus	58.24	23.70	58.85	21.52
						diffusion	58.92	20.92	57.67	18.86
					256	centralized	39.96	15.32	40.06	15.40
						consensus	55.33	22.49	57.09	21.07

TABLE II: Clean and robust accuracy of the obtained models evaluated using AutoAttack on CIFAR-100

dataset	graph	K	norm	$\epsilon$	local batch	method	best		final
dataset	graph	K	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
Tiny ImageNet-200	random	16	$\ell_{\infty}$	$3/255$	128	centralized	47.11	22.22	46.11	19.67
						consensus	57.34	32.54	56.19	27.75
						diffusion	54.92	27.11	54.03	25.50
			$\ell_{2}$	$128/255$	128	centralized	47.13	24.41	46.32	22.44
						consensus	60.01	37.60	58.27	31.67

TABLE III: Clean and robust accuracy of the obtained models evaluated using AutoAttack on Tiny ImageNet-200

Panel of Figure 7 — Fig. 4: The evolution of the training error on CIFAR-10 using a random graph topology with $K=16$ nodes. We observe that under strong adversarial attacks, specifically when the perturbation bound is $\epsilon=8/255$ , the optimization performance of centralized models in the large-batch setting can be bad.

Panel of Figure 8 — Fig. 5: The evolution of the training error on CIFAR-100 using the random graph structure with $K=16$ nodes. Similar to Figure 4 , when $\epsilon=8/255$ , the optimization performance of centralized models are bad.

Panel of Figure 9 — Fig. 6: The evolution of the training error on Tiny ImageNet-200 using the random graph structure with $K=16$ nodes.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Data	Neural Network	Initial Learning Rate(LR)	Weight Decay	Optimizer	K	B	Epoch	LR Milestones
CIFAR-10	WideResNet-28-10	0.1	5e-4	SGD	8	128	350	[140, 210, 280]
					16	128	350	[140, 210, 280]
					16	256	400	[200, 280, 360]
					32	128	400	[200, 280, 360]
				SGD Momentum	16	128	350	[140, 210, 280]
				SGD Momentum	16	256	400	[200, 280, 360]

Appendix

TABLE IV: Training Hyperparameter Settings

Data	K	B	Method	Hardware	Epoch Time
CIFAR-10	8	128	Centralized	1*A100	220s
			Consensus	1*A100	270s
			Diffusion	1*H100	139s
	16	128	Centralized	1*H100	106s
			Consensus	1*H100	164s
			Diffusion	1*H100	162s

Appendix

TABLE V: The training time of each epoch

Panel of Figure 15 — Fig. 7: Flatness visualization of the final models trained on the CIFAR-10 dataset using the ring topology with $K=\{8,16,32\}$ . Under relatively mild attacks, i.e., $\ell_{\infty}$ perturbations bounded by $\epsilon=3/255$ and $\ell_{2}$ perturbations bounded by $128/255$ , decentralized adversarial training methods, including consensus and diffusion, converge to flatter models than the centralized strategy. However, when the attack strength increases to $\epsilon=8/255$ with the $\ell_{\infty}$ norm, diffusion can yield sharper models than centralized training. These observations remain consistent across different network sizes.

Panel of Figure 16 — Fig. 8: Flatness visualization of the final models trained on the CIFAR-10 dataset using the grid topology with $K=16$ . In (c), diffusion yields a sharper model than centralized. Combining these results with those in Figures 1 and 7 , we conclude that a similar phenomenon holds across different graph topologies.

Panel of Figure 17 — Fig. 9: Flatness visualization of the final models trained on the CIFAR-100 dataset using the ring topology with $K=\{8,16,32\}$ . The results shows a similar phenomenon to that observed in Figure 7 .

Panel of Figure 18 — Fig. 10: Flatness visualization of the final models trained on the CIFAR-100 dataset using the grid topology with $K=16$ . The results are similar to Figure 8 .

Panel of Figure 19 — Fig. 11: Flatness visualization of the best models trained on the CIFAR-10 dataset using the random topology with $K=16$ . The results are similar to Figure 1 .

Panel of Figure 20 — Fig. 12: Flatness visualization of the best models trained on the CIFAR-100 dataset using the random topology with $K=16$ nodes. The results are similar to Figure 2 .

Panel of Figure 21 — Fig. 13: Flatness visualization of the best models trained on the Tiny ImageNet-200 dataset using the random topology with $K=16$ nodes. The results are similar to Figure 3 .

Panel of Figure 22 — Fig. 14: Flatness visualization of the best models trained on the CIFAR-10 dataset using the ring topology with $K=\{8,16,32\}$ . The results are similar to Figure 7 .

Panel of Figure 23 — Fig. 15: Flatness visualization of the best models trained on the CIFAR-10 dataset using the grid topology with $K=16$ . The results are similar to Figure 8 .

Panel of Figure 24 — Fig. 16: Flatness visualization of the best models trained on the CIFAR-100 dataset using the ring topology with $K=\{8,16,32\}$ . The results are similar to Figure 9 .

Panel of Figure 25 — Fig. 17: Flatness visualization of the best models trained on the CIFAR-100 dataset using the grid topology with $K=16$ . The results are similar to Figure 10 .

dataset	graph	K	norm	$\epsilon$	local batch	method	best		final
dataset	graph	K	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-10	ring	8	$\ell_{\infty}$	$8/255$	128	centralized	78.36	37.32	78.68	33.84
						consensus	84.63	44.99	84.97	38.64
						diffusion	85.44	39.96	84.85	37.38
				$3/255$	128	centralized	86.49	62.04	86.45	61.97
						consensus	91.53	71.40	91.17	69.50

Appendix

TABLE VI: Clean and robust accuracy on CIFAR-10 for models trained using a ring topology

dataset	graph	K	norm	$\epsilon$	local batch	method	best		final
dataset	graph	K	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-10	grid	16	$\ell_{\infty}$	$8/255$	128	centralized	71.11	35.89	71.44	35.86
						consensus	80.82	44.85	82.99	42.96
						diffusion	84.55	44.67	84.85	39.65
				$3/255$	128	centralized	84.91	60.97	84.50	59.55
						consensus	90.18	71.77	91.00	69.22

Appendix

TABLE VII: Clean and robust accuracy on CIFAR-10 for models trained using a grid topology

dataset	graph	K	norm	$\epsilon$	local batch	method	best		final
dataset	graph	K	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-100	ring	8	$\ell_{\infty}$	$8/255$	128	centralized	40.51	15.23	48.97	14.30
						consensus	59.08	20.55	57.68	19.19
						diffusion	57.02	19.29	56.92	18.91
				$3/255$	128	centralized	56.73	31.10	57.28	31.28
						consensus	67.60	40.17	67.38	39.56

Appendix

TABLE VIII: Clean and robust accuracy on CIFAR-100 for models trained using a ring topology

dataset	graph	K	norm	$\epsilon$	local batch	method	best		final
dataset	graph	K	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-100	grid	16	$\ell_{\infty}$	$8/255$	128	centralized	46.95	16.28	47.09	15.47
						consensus	58.63	23.04	59.40	21.50
						diffusion	60.25	21.63	58.03	19.35
				$3/255$	128	centralized	55.13	29.07	54.57	28.27
						consensus	68.55	42.99	68.02	39.18

Appendix

TABLE IX: Clean and robust accuracy on CIFAR-100 for models trained using a grid topology

Panel of Figure 30 — Fig. 18: The evolution of the training error on CIFAR-10 using the ring graph with $K=\{8,16,32\}$ nodes. The centralized method exhibits particularly high training error when $\epsilon=8/255$ .

Panel of Figure 31 — Fig. 19: The evolution of the training error on CIFAR-10 using a grid graph with $K=16$ nodes. The observed phenomenon is similar to Figure 18 .

Panel of Figure 32 — Fig. 20: The evolution of the training error on CIFAR-100 using the ring graph with $K=\{8,16,32\}$ nodes. The observed phenomenon is similar to Figure 18 .

Panel of Figure 33 — Fig. 21: The evolution of the training error on CIFAR-100 using a grid graph with $K=16$ nodes. The observed phenomenon is similar to Figure 18 .

Panel of Figure 34 — Fig. 22: Flatness visualization of models in the pretrained initialization scenario. Basically, when initialized from models pretrained by the centralized approach, decentralized converges to flatter regions than centralized ones.

dataset	norm	$\epsilon$	local batch	method	best		final
dataset	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-10	$\ell_{\infty}$	$8/255$	128	centralized	75.67	39.34	83.88	38.86
				consensus	85.23	42.87	84.89	40.18
				diffusion	76.53	41.34	84.75	40.36
			256	centralized	76.87	38.70	83.27	37.25
				consensus	83.60	39.98	83.35	39.04

Appendix

TABLE X: Clean and robust accuracy of models trained using PGD and the SGD momentum optimizer.

dataset	norm	$\epsilon$	local batch	method	best		final
dataset	norm	$\epsilon$	local batch	method	clean( $\%$ )	AA( $\%$ )	clean( $\%$ )	AA( $\%$ )
CIFAR-10	$\ell_{\infty}$	$8/255$	128	centralized	81.59	43.47	81.68	40.10
				consensus	82.46	45.54	83.05	41.20
				diffusion	82.54	45.39	83.07	41.46
			256	centralized	80.07	42.20	80.16	37.51
				consensus	81.21	43.83	82.03	40.52

Appendix

TABLE XI: Clean and robust accuracy of the models trained with Trades.

Explore similar work

Feb 2, 2026cs.LG

Decentralized SGD with Controlled Disagreement Finds Flatter Minima

Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which uses a time-dependent scaling mechanism to maintain consensus errors throughout the training. We show that adaptive consensus changes the stationary variance of disagreement modes by balancing two effects: it preserves consensus-error magnitude through weaker graph damping while still allowing curvature-dependent damping to shape the disagreement directions. This balance can produce a stronger Hessian-weighted loss-envelope penalty around the deployed model, even when normalized Hessian alignment is weaker than in standard DSGD. Empirical results on image classification show that DSGD-AC reaches flatter solutions and higher test accuracy than standard DSGD and even centralized SGD. Together, these results support consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.

Zesen Wang, Mikael Johansson

Division of Decision and Control Systems KTH Royal Institute of Technology

Decentralized Learning Stochastic Gradient Descent Quadratic Slack Penalty Flat Minima Disagreement

Ask PDF

Apr 29, 2026cs.CV

Robust Alignment: Harmonizing Clean Accuracy and Adversarial Robustness in Adversarial Training

Adversarial Training (AT) is one of the most effective methods for developing robust deep neural networks (DNNs). However, AT faces a trade-off problem between clean accuracy and adversarial robustness. In this work, we reveal a surprising phenomenon for the first time: Varying input perturbation intensities for training samples near decision boundaries in AT have minimal impact on model robustness. This finding directly exposes the inconsistency between accuracy and robustness score fluctuations, leading us to identify the misalignment between input and latent spaces as a critical driver of the robustness-accuracy trade-off. To mitigate this misalignment for harmonizing accuracy and robustness, we define Robust Alignment as a new AT target, encouraging the model perception to change with input perturbations provided the final label prediction remains unchanged, which can be achieved via two novel ideas. First, we suggest a reduced and fixed perturbation intensity for those boundary samples, which facilitates the model to utilize the perturbations as learnable patterns, instead of noises that complicate decision boundaries meaninglessly. Second, we propose a Domain Interpolation Consistency Adversarial Regularization (DICAR), based on rigorous theoretical derivations, which explicitly introduces semantic alignment between input and latent spaces into AT. Based on these two ideas, we end up with a new Robust Alignment Adversarial Training (RAAT) method, effectively harmonizing accuracy and robustness. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, PreActResNet-18, and WideResNet-28-10 demonstrate the effectiveness of RAAT in improving the trade-off beyond four common baselines and a total of 14 related state-of-the-art (SOTA) works.

Yanyun Wang, Qingqing Ye, Li Liu +2

1HK PolyU · 2HKUST (GZ) · 2HKUST (GZ

Adversarial Training Harmonization

Ask PDF

May 8, 2026cs.LG

\mathsf{VISTA}: Decentralized Machine Learning in Adversary Dominated Environments

Decentralized machine learning often relies on outsourcing computations, such as gradient evaluations, to untrusted worker nodes. Existing robust aggregation methods can mitigate malicious behavior under honest-majority assumptions, but may fail when adversaries control a majority of the workers. We study this adversary-dominated setting through an incentive-oriented framework in which reports are accepted and rewarded only when they are mutually consistent up to a threshold. This turns the adversary from a pure saboteur into a rational agent that trades off increasing estimation error against the risk of rejection and loss of reward. We consider iterative optimization under this model. Unlike one-shot computation, iterative learning requires long-horizon decisions: permissive acceptance rules enable faster early progress but admit more adversarial corruption, while strict rules improve estimation accuracy but cause frequent rejections. We propose \mathsf{VISTA}, an adaptive algorithm that tunes the acceptance threshold using the optimization history. Numerical results show that \mathsf{VISTA} improves convergence over static thresholds. We also provide a rigorous convergence analysis showing that, with suitable incentive-aware adaptation, adversary-dominated decentralized learning can retain the asymptotic convergence behavior of standard SGD without relying on an honest majority.

Hanzaleh Akbari Nodehi, Parsa Moradi, Soheil Mohajer +1

University of Minnesota

Bounded Adversary Adversarial Robustness Generative Control Policies Adversarial Corruption Vista

Ask PDF