Distributionally robust linear regression through the lens of adversarial training

Elis Stefansson; David Vävinggren; Antônio H. Ribeiro

stat.MLSep 30, 2026

Distributionally robust linear regression through the lens of adversarial training

Authors: Elis Stefansson, David Vävinggren, Antônio H. Ribeiro

Organizations: Uppsala University, Sweden · Science for Life Laboratory, Sweden

Abstract

Distributionally robust optimization (DRO) studies parameter estimation under uncertainty in the underlying probability distribution and has emerged as a principled framework for analyzing robustness and generalization. In particular, Wasserstein DRO, with distributional uncertainty induced by the Wasserstein distance, generalizes several popular regularizers. This paper studies Wasserstein DRO linear regression, unifying square-root Lasso and adversarial linear regression as important special cases. We prove that many properties of these two special cases carry over to this general method. In particular, we show (i) deterministic and non-asymptotic in-sample error bounds $O(n^{-1/2})$ in general and $O(n^{-1})$ under design matrix and sparsity conditions; (ii) insensitivity to the noise level, also known as the pivotal property; and (iii) solution equivalences for small and large ambiguity sets. The key proof step is to recast the method into a quadratic form, mimicking adversarial linear regression. We also show that the method can be solved efficiently, and we validate our findings through numerical simulations.

Figures & tables

Figure 1 : Mean computation times over 10 repetitions for the saddle-point solver (solid) and the

\eta

-trick solver (dashed) in the fast-rate experiment.

Panel of Figure 2 — Figure 2 : Fast rates (left) and slow rates (right) for $p\in\{2,3,6,\infty\}$ . Both show a clear convergence toward the predicted rates $O(n^{-1})$ and $O(n^{-1/2})$ , respectively. The curves are mean over 10 runs with $\pm 1$ standard deviation bands.

Figure 3 : Large

\delta

(bottom) and small

\delta

(top) in the overparametrized regime, with zero solution and interpolation for

\delta\geq\delta_{L}

and

\delta\leq\delta_{S}

, respectively.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure A.1 : Additional plots for the fast rate simulations in Section 8 showing that the conditions in Theorem 3 hold. Concretely, the left plot shows that the RE condition is satisfied for $\kappa=1/\sqrt{6}$ (for all $n$ ) since it indeed lower-bounds the minimum eigenvalue $\lambda_{\min}(G)$ of $G=X^{T}X/n$ (see ( A.30 ) and the corresponding derivation for details). Moreover, the right plot shows that $\delta\geq\bar{\delta}$ (which it should be with high probability). Therefore, the conditions in Theorem 3 hold. Both plots show median over 10 runs with shaded bands showing minimum and maximum across the repetitions.

Figure A.2 : Convergence for

e_{q}:=\frac{\|\boldsymbol{\varepsilon}\|_{q}}{n^{1/q}}

in the fast rate simulations from Section 8 . We see that all

e_{q}

quickly stabilizes toward a constant as

n

increases. Since

e_{q}

converges to a constant, so do

B

and

C

in Theorem 3 . The plot shows the median over 10 repeats with shaded bands being the 10th-90th percentile range.

Panel of Figure 6 — Figure A.3 : We plot the error of the saddle-point solver and the $\eta$ -trick solver for the fast rate setup (left). We run 10 repeats and plot the mean with $\pm 1$ standard deviation bands. We also plot the robust risk for the obtained estimate for both solvers (right). Here we plot the median over 10 repeats with 10-90 percentiles. We note that both output near-identical results.

Panel of Figure 7 — Figure A.4 : Computation time (left) and time ratio (right) of the saddle-point solver and the $\eta$ -trick solver for the fast rate setup. We see that $\eta$ -trick solver is, in general, faster, particularly for large $n$ and $p\in\{2,\infty\}$ . Both methods are slower for $p\in\{3,6\}$ , since the robust risk then lacks a closed-form expression. Both plots show mean over 10 runs.

Explore similar work

Wasserstein Distributionally Robust Regret Optimization
Apr 15, 2025Lukas-Benedikt Fiechtner, Jose BlanchetDistributionally-Robust Optimization Robust Optimization
Institute of Computational and Mathematical Engineering Stanford University · Institute of Computational and Mathematical Engineering Department of Management Science and Engineering Stanford University
Distributionally-Robust Learning to Optimize
May 7, 2026Vinit Ranjan, Jisun Park, Bartolomeo StellatoDistributionally-Robust Optimization Convex Optimization
Department of Operations Research and Financial Engineering, Princeton University. · Department of Operations Research and Financial Engineering, Princeton University, and Research Institute of Mathematics, Seoul National University. · Research Institute of Mathematics, Seoul National University.
Distributionally Robust Linear Regression With Block Lewis Weights
Jun 30, 2026Naren Sarayu Manoj, Kumar Kshitij PatelDistributionally-Robust Optimization Least Squares
TTIC · Institute for Foundations of Data Science, Yale University

Apr 15, 2025, math.OCJ/K move · Enter open · S save

Wasserstein Distributionally Robust Regret Optimization

Lukas-Benedikt Fiechtner, Jose Blanchet

Institute of Computational and Mathematical Engineering Stanford University · Institute of Computational and Mathematical Engineering Department of Management Science and Engineering Stanford University

Distributionally-Robust Optimization Robust Optimization Wasserstein Distance Convex Loss Minimax Rate Regret

Open paper AskPDF arXiv

Distributionally robust optimization (DRO) is widely used for decision-making under uncertainty, but its adversarial focus on worst-case loss can lead to overly conservative policies. To mitigate this, we study ex-ante Distributionally Robust Regret Optimization (DRRO) with Wasserstein ambiguity sets, designed to balance robustness with upside potential. We develop a theory of Wasserstein DRRO (WDRRO) paralleling Wasserstein DRO. Under smoothness and regularity, WDRRO selects among ERM optima by a first-order gradient-discrepancy rule. If the ERM optimizer is unique, first-order sensitivity vanishes and a second-order expansion governs deviations. For convex quadratics ERM and DRRO coincide for any radius. We then study regimes where these assumptions fail: nondifferentiable max-affine losses, discrete references, and larger radii, where WDRRO can differ from ERM and WDRO. We show that computing WDRRO regret is NP-hard even without bilinear terms. Nevertheless, we develop exact algorithms, a tractable convex relaxation with guarantees, and experiments showing tightness and loss-dependent behavior.

Abstract

Explore similar work

Wasserstein Distributionally Robust Regret Optimization

Distributionally-Robust Learning to Optimize

Distributionally Robust Linear Regression With Block Lewis Weights