Distributionally robust optimization (DRO) provides a principled framework for learning under distribution shift, but its practical use is hindered by the difficulty of evaluating worst-case risks for nonconvex loss functions. We study a penalized DRO formulation in which the adversary may choose any distribution but incurs a Wasserstein penalty for deviating from the empirical distribution. We show that the adversary's problem can be reformulated as an optimization problem over transport maps that push empirical samples to adversarial ones, and we prove that optimal maps are cyclically monotone. We also show that standard adversarial training---based on per-sample local optimization---violates cyclical monotonicity and wastes transport costs unless the adversary is severely restricted. We propose two remedies. First, we introduce multi-start particle ascent, which alternates parallel gradient ascent with reassignment to enforce cyclical monotonicity across samples. Second, we parameterize adversarial maps as gradients of input-convex neural networks, which guarantees cyclical monotonicity by construction. Experiments on robust regression, image classification, and robust control show that our methods consistently outperform standard adversarial training and state-of-the-art baselines, achieving improved robustness and better generalization under distribution shift.
Figures & tables
Figure 1: PA and MPA trajectories over the contours of f(θ,z)−λ∥z−zi∥22 for i=1 (left) and i=2 (right).
Figure 2: Test error vs. perturbation level Δ on multi-class logistic regression.
Method
CIFAR-10
CIFAR-10.1
CIFAR-10.2
CIFAR-10-C (Common Corruption)
Runtime
Clean
PGD
AA
Monge Gap
Clean
Clean
1
2
3
4
5
Avg
(s)
ERM
93.12
14.76
00.20
—
87.12
83.14
91.50
87.95
83.84
77.89
68.41
81.92
—
PA
90.08
52.41
48.86
8.01e-4
80.27
76.16
87.11
84.35
80.21
74.62
69.74
79.20
1320
RO
90.05
53.37
50.26
8.06e-4
80.28
76.81
87.15
85.06
81.75
75.35
70.12
79.89
659
WFR
91.03
53.28
50.37
5.63e-3
81.24
77.09
88.60
85.34
82.93
77.56
71.09
81.10
6396
SDRO
90.00
53.12
49.61
1.37e-3
80.20
76.48
87.12
84.82
80.33
74.70
70.09
79.41
6241
Table 1: Accuracy (%) under adversarial attacks, natural domain shifts and image corruptions. The left section reports results for the original CIFAR-10 test set (Clean, PGD, and AA), along with the Monge gap. The center columns evaluate accuracy on the unseen domains of CIFAR-10.1 and CIFAR-10.2. The right section (CIFAR-10-C) details performance across five increasing levels of corruption severity, followed by the mean accuracy across all five levels and the runtime. Bold values indicate the best results, and underlined values denote the second best.
Figure 3: Mean episode length for λ=1 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Loss landscape of ICNN-DRO on CIFAR-10. Panels (a) and (b) correspond to test sample #4731 (Deer), while panels (c) and (d) detail test sample #5118 (Bird). For each sample, the left panel (a, c) displays the 2D decision regions. The dashed circle marks the ℓ2 budget ε=0.5 ; orange and blue indicate correct and incorrect classification, respectively; and point markers specify the exact coordinates of the initial clean image (black circle) and the final adversarial image (black triangle). The right panel (b, d) plots the corresponding 3D cross-entropy surface where the red curve traces the loss along the adversarial direction.
Method
Original
Easier Environments
Harder Environments
Original
Light
Long
Soft Gravity
Heavy
Short
Strong Gravity
(a) λ=0.1
ERM
186.4±1.9
188.7±1.9
191.4±1.2
217.3±1.9
178.6±1.5
155.6±0.8
157.6±1.1
PA
382.9±7.4
395.1±7.4
266.9±4.1
543.3±7.1
289.0±5.8
205.1±1.7
207.0±2.8
RO
380.1±5.1
398.3±5.5
270.5±3.4
556.6±5.3
295.1±2.1
209.0±1.6
212.4±1.2
WFR
507.3±7.6
532.9±8.0
434.5±5.8
756.8±7.2
387.0±4.6
313.4±2.5
323.9±3.0
Appendix
Table 2: Mean episode length ± standard error over 1,000 deterministic rollouts across different λ . Panel (a) reports λ=0.1 ; panel (b) reports λ=0.5 and panel (c) reports λ=1 . The leftmost column corresponds to the original training configuration, the middle group to easier physics variants (Light, Long, Soft Gravity), and the right group to harder variants (Heavy, Short, Strong Gravity). Bold entries indicate the best result per column.
Figure 5: Test loss vs. perturbation level Δ on uncertain least squares.
Accuracy (%)
Transfer
Model
Clean ↑
Adv. ↑
Drop ↓
Ovlp.
% src
Lbl mis.%
R1-ICNN (src)
92.38
74.29
18.09
2,336
—
—
ViT1
89.37
49.39
39.98
1,903
81.46
56.96
ViT2
78.22
53.54
24.68
1,696
72.60
54.13
ViT3
82.32
54.38
27.95
1,763
75.47
55.30
ViT4
92.42
57.29
35.12
1,839
78.72
55.41
Appendix
Table 3: Cross-architecture transfer of the ICNN adversary trained against R1. Drop : Clean-Adv. gap; Ovlp : counts examples fooled by both the source and the target; % src err : percentage of source-model failures that transfer to the target; Lbl mis.% : percentage of overlap examples on which source and target predict different incorrect labels.
Regularization strength λ
Method
2
5
10
15
30
ERM
0
0
0
0
0
Baselines
PA
1.24\times10−2
1.15\times10−2
3.01\times10−3
7.93\times10−4
7.97\times10−4
WFR
1.13\times10−2
1.02\times10−2
1.73\times10−3
5.68\times10−4
5.58\times10−4
SDRO
1.83\times10−0
1.04\times10−0
4.52\times10−3
1.27\times10−3
1.29\times10−3
NN-DRO
1.37\times10−2
1.21\times10−2
3.54\times10−3
3.28\times10−3
3.16\times10−3
Appendix
Table 4: Monge gap on CIFAR10.
Regularization strength λ
Method
0.1
0.5
1
ERM
0
0
0
Baselines
PA
4.87\times10−2
2.01\times10−2
1.99\times10−2
WFR
1.87\times10−2
1.41\times10−2
1.13\times10−2
SDRO
2.18\times10−2
2.03\times10−2
1.93\times10−3
NN-DRO
2.17\times10−2
2.07\times10−2
1.98\times10−3
Appendix
Table 5: Monge gap on robust control (CartPole).
Method
rout
Total params
CIFAR-10
Wall-clock Runtime (s)
Clean
PGD ε=0.5
Complex backbone [1024, 512, 512, 256, 128, 64]
ICNN-DRO-Q
8
31,693,314
91.37
57.36
10854
ICNN-DRO-Q
32
31,767,042
91.30
57.41
10908
ICNN-DRO-Q
128
32,061,954
91.21
57.58
11325
ICNN-DRO
8
8,665,602
91.45
57.28
6439
Appendix
Table 6: Adversary capacity versus robustness. Best results per column are in bold, and second-best results are underlined. PGD cells are shaded green for configurations satisfying our robustness criterion (PGD accuracy ≥56% ), with darker green indicating stronger robustness, and red otherwise, with darker red indicating a larger shortfall. Runtime cells are shaded green only for configurations satisfying this robustness criterion, with darker green indicating lower computational cost. A runtime marked with † corresponds to a configuration with PGD accuracy below 56% . Although such configurations may be substantially faster, we do not regard them as successful accelerations of ICNN training because they fail our minimum robustness criterion. The objective of this ablation is specifically to reduce the approximately 104 s wall-clock training time of the full-width quadratic ICNN while retaining at least 56% PGD accuracy.
Method
K
CIFAR-10
CIFAR-10.1
CIFAR-10.2
CIFAR-10-C (Common Corruption)
Clean
PGD ε=0.5
AA ε=0.5
Clean
Clean
1
2
3
4
5
Avg
ICNN-DRO (fixed λ=30 )
50
91.20
59.83
58.81
82.57
79.98
89.28
86.85
84.10
80.00
74.06
82.85
ICNN-DRO (fixed λ=30 )
20
91.41
58.11
57.71
82.56
80.12
89.20
86.79
84.01
79.92
73.95
82.77
ICNN-DRO + λ -annealing
20
91.48
59.94
58.92
82.60
80.10
89.31
86.93
84.15
80.04
74.12
82.91
Appendix
Table 7: Penalty annealing versus fixed-penalty adversarial training on CIFAR-10. The K=20 fixed- λ baseline is reproduced from Table 1 for comparison. Best results per column are in bold, and the second best are underlined.
We study finite-sample statistical performance guarantees for distributionally robust optimization (DRO) with optimal transport (OT) and OT-regularized divergence model neighborhoods. Specifically, we derive concentration inequalities for supervised learning via DRO-based adversarial training, as commonly employed to enhance the adversarial robustness of machine learning models. Our results apply to a wide range of OT cost functions, beyond the p-Wasserstein case studied by previous authors. In particular, our results are the first to: 1) cover soft-constraint norm-ball OT cost functions; soft-constraint costs have been shown empirically to enhance robustness when used in adversarial training, 2) apply to the combination of adversarial sample generation and adversarial reweighting that is induced by using OT-regularized f-divergence model neighborhoods; the added reweighting mechanism has also been shown empirically to further improve performance. In addition, even in the p-Wasserstein case, our bounds exhibit better behavior as a function of the DRO neighborhood size than previous results when applied to the adversarial setting.
Jeremiah Birrell, Xiaoxi Shen
Department of Mathematics Texas State University San Marcos, TX, USA
Wasserstein distributionally robust optimization (WDRO) provides a framework for evaluating model robustness through the worst-case expectation in the Wasserstein ball around a reference distribution. We prove that ordering hypotheses by increasing WDRO risk is equivalent to ordering them by decreasing robust accuracy if and only if the loss function is a positive multiple of the zero-one loss plus a label-dependent offset. For a class of loss functions including cross-entropy, we show that a hypothesis with perfect robust accuracy can conversely have a higher WDRO risk than another hypothesis with near zero robust accuracy. To evaluate robust accuracy directly, we develop Wasserstein distributional attacks (WDA), which add a weight generation stage after candidate support search to optimize the allocation of empirical mass under the Wasserstein constraint, rather than assigning each data point's full mass to one returned attack point as existing methods do. We further introduce a support-search algorithm for WDA that finds nearby misclassified inputs. On CIFAR-10, CIFAR-100, and ImageNet, WDA using our search algorithm achieves the lowest adversarial accuracy in most tested settings. Our code is available at https://github.com/OLab-Repo/WDA.
Bach C. Le, Tung V. Dao, Binh T. Nguyen +1
College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam
We study distributionally robust optimization (DRO) for robust inference when the worst-case distribution is continuous, leading to significant computational challenges due to the infinite-dimensional nature of the optimization problem. Unlike traditional discrete DRO approaches, which often suffer from scalability issues, limited generalization, and costly worst-case inference, our framework exploits Brenier's theorem to characterize the least favorable distribution as the pushforward of a transport map from a continuous reference measure. This characterization motivates our study of the minimax problem in Wasserstein space. We propose an iterative algorithmic framework with multiple variants and establish global convergence guarantees under mild assumptions, deriving complexity bounds in terms of subgradient evaluations and inexact Jordan-Kinderlehrer-Otto updates. Numerical results with neural network-based transport maps demonstrate that the proposed method enables both stable training of robust classifiers and effective worst-case inference for classification tasks.
Linglingzhi Zhu, Yunqin Zhu, Yao Xie
1H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology