How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

Kuo-Wei Lai; Guanghui Wang; Molei Tao; Vidya Muthukumar

stat.MLMar 5, 2026

How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

Authors: Kuo-Wei Lai, Guanghui Wang, Molei Tao, Vidya Muthukumar

Organizations: School of Electrical and Computer Engineering, Georgia Institute of Technology · College of Computing, Georgia Institute of Technology · School of Mathematics, Georgia Institute of Technology · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology

Abstract

Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work (Vardi and Shamir, 2021) showed that the implicit bias does not exist in the worst-case, or corresponds exactly to the minimum- $\ell_2$ -norm interpolating solution under exactly orthogonal data (Boursier et al., 2022). Our work interpolates between these two extremes and shows that, for sufficiently high-dimensional random data, the implicit bias approximates the minimum- $\ell_2$ -norm solution with high probability with a gap on the order $Θ(\sqrt{n/||λ||_1})$ , where $n$ is the number of training examples and $λ$ denotes the spectrum of the data covariance matrix. Our results are obtained through a novel primal-dual analysis that carefully tracks the evolution of predictions, data-span coefficients, as well as their interactions, and show that the ReLU activation pattern quickly stabilizes with high probability over random data.

Figures & tables

	Orthogonal data	High dimensional data	Worst-case data
ReLU models ( $k=1$ ) $h(\boldsymbol{x})\coloneqq\sum_{k=1}^{m}s_{k}\sigma(\boldsymbol{w}_{k}^{\top}\boldsymbol{x})$	Implicit bias characterization ( Boursier et al., 2022 ) $\boldsymbol{w}^{(\infty)}=\underset{\boldsymbol{w}\in\{\boldsymbol{w}\mathrel{\mathop{\ordinarycolon}}\boldsymbol{X}\boldsymbol{w}=\boldsymbol{y}\}}{\arg\min}\left\\|\boldsymbol{w}-\boldsymbol{w}^{(0)}\right\\|_{2}$	Global convergence only ( Dana et al., 2025 )	No implicit bias in general ( Vardi and Shamir, 2021 ) $\left\\|\boldsymbol{w}^{(\infty)}-\boldsymbol{w}^{(0)}\right\\|_{2}\leq 2\cdot\left\\|\boldsymbol{w}^{\star}-\boldsymbol{w}^{(0)}\right\\|_{2}$
		This work $\left\\|\boldsymbol{w}^{(\infty)}-\boldsymbol{w}^{\star}\right\\|_{2}\asymp\sqrt{\frac{n}{\left\\|\boldsymbol{\lambda}\right\\|_{1}}}$
Linear models $h(\boldsymbol{x})\coloneqq\boldsymbol{w}^{\top}\boldsymbol{x}$	Implicit bias coincides with maximum $\ell_{2}$ margin SVM ( Hsu et al., 2021 )		$\boldsymbol{w}^{(\infty)}=\underset{\boldsymbol{w}\in\{\boldsymbol{w}\mathrel{\mathop{\ordinarycolon}}\boldsymbol{X}\boldsymbol{w}=\boldsymbol{y}\}}{\arg\min}\left\\|\boldsymbol{w}-\boldsymbol{w}^{(0)}\right\\|_{2}$ ( Engl et al., 1996 )

Table 1: Our results contextualized with related literature.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: We illustrate the prediction dynamics of gradient descent for a single ReLU model under different random initializations when $d$ is comparable with $n$ . In both cases, with sufficiently small step size, the final solution converges to a linear minimum- $\ell_{2}$ -norm interpolator on some subset of the training examples, i.e. of the form $\boldsymbol{w}_{\text{linear-MNI},S}=\boldsymbol{X}_{S}^{\top}(\boldsymbol{X}_{S}\boldsymbol{X}_{S}^{\top})^{-1}\tilde{\boldsymbol{y}}_{S}$ , where $\tilde{y}_{S,i}=\max\{y_{i},0\}$ . In contrast to the high-dimensional regime, different initializations lead to different subsets $S$ , indicating that ReLU training implicitly performs an example “selection” process, that is initialization-dependent, rather than fitting all positively-labeled samples. The experiment uses $n=10$ , $d=50$ , $\boldsymbol{x}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ , $y\sim\mathcal{N}(0,1)$ , $\boldsymbol{w}^{(0)}\sim\mathcal{N}(\boldsymbol{0},2\times 10^{-6}\boldsymbol{I})$ , and $\eta=10^{-4}$ .

Panel of Figure 4 — Figure 4: Simulation illustrating Theorem 3 . In the high-dimensional regime and under our “all-positive” initialization, after the first gradient step, examples with positive labels remain active while examples with negative labels become inactive, consistent with Lemma 13 . The blue region shows primal variables that remain positive over training, whereas the red region corresponds to dual variables that are sufficiently negative and remain unchanged. As training proceeds, $\boldsymbol{w}_{\oplus}$ fits all positively labeled examples and $\boldsymbol{w}_{\ominus}$ fits all negatively labeled examples. The experiment uses $n=10$ , $d=2000$ , features $\boldsymbol{x}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ , and labels satisfying $|y|\sim\mathcal{U}(0.1,1)$ with $\mathrm{sign}(y)$ uniformly distributed over $\{\pm 1\}$ .

Panel of Figure 5 — Figure 5: Simulation with random initialization in the high-dimensional regime, which violates our initialization assumption in Theorem 3 . Under random initialization, the sufficient conditions of Lemma 13 are violated at the first gradient step. As a result, positively labeled examples do not all remain in the active (blue) regime (e.g., example no. 5), nor do negatively labeled examples consistently enter the inactive (red) regime (e.g., example no. 7). Consequently, during training, this model fails to converge to a global minimum. The experiment uses $n=10$ , $d=2000$ , features $\boldsymbol{x}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ , and labels satisfying $|y|\sim\mathcal{U}(0.1,1)$ with $\mathrm{sign}(y)$ uniformly distributed over $\{\pm 1\}$ .

Panel of Figure 6 — Figure 6: Simulation with all-positive initialization outside the high-dimensional regime. When the data dimension is not sufficiently large, the feature vectors are no longer approximately orthogonal. As a result, the clear separation into active (blue) and inactive (red) regimes observed in Figures 4 and 5 disappears. Consequently, the gradient dynamics become highly coupled across examples and are no longer analytically tractable using our high-dimensional arguments. The experiment uses $n=10$ , $d=15$ , features $\boldsymbol{x}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ , and labels satisfying $|y|\sim\mathcal{U}(0.1,1)$ with $\mathrm{sign}(y)$ uniformly distributed over $\{\pm 1\}$ .

Panel of Figure 7 — Figure 7: Failure of stable activation patterns in multiple ReLU models. We illustrate the training dynamics of a multiple ReLU model when multiple neurons share the same sign. In this setting, the sufficient conditions of Lemma 15 are violated, and positive primal variables do not necessarily remain in the active (blue) regime throughout training (e.g. training example no. 0). As a result, the activation pattern becomes unstable, and the resulting primal dynamics are no longer tractable. The experiment uses $n=10$ , $d=2000$ , $m=4$ , with neuron signs $s_{1}=s_{2}=1$ and $s_{3}=s_{4}=-1$ , features $\boldsymbol{x}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$ , and labels satisfying $|y|\sim\mathcal{U}(0.1,1)$ with $\mathrm{sign}(y)$ uniformly distributed over $\{\pm 1\}$ .

High-Dimensional Linear Regression

See all 16 papers

Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse
Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural Networks ReLU Neural Networks
Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences
Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias
May 26, 2026James Town, Etienne Boursier, Ben Lewis +2ReLU Neural Networks Neural Network Optimization
University of Warwick · INRIA LMO, Université Paris-Saclay
Estimating Implicit Regularization in Deep Learning
May 6, 2026Joseph H. Rudoler, Kevin Tan, Giles Hooker +1Neural Network Optimization Implicit Bias
University of Pennsylvania

Jul 3, 2026J/K move · Enter open · S save

Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

Shuang Liang, Tom Jacobs, Guido Montúfar

Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences

Shallow Neural Networks ReLU Neural Networks Implicit Bias Neural Network Training Dynamics Stochastic Gradient Descent

Open paper AskPDF arXiv

We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression. In a mean-field regime, the training dynamics are approximated by a Wasserstein gradient flow that converges to a unique stationary measure. We characterize the structure of this stationary measure and the predictor it represents. We show that, despite the network being infinitely overparameterized, the learned predictor admits an effectively finite representation: the input weights and biases align along finitely many directions, leading to an effective width collapse. In particular, the solution function is continuous piecewise affine, with affine regions determined by the cells of a finite hyperplane arrangement. The number of learned directions, and hence hyperplanes, is bounded above by

2\mathcal{P}-1

, where

\mathcal{P}

denotes the number of linear dichotomies realizable on the training inputs. We further establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data. Consequently, the complexity of the learned predictor is governed by the combinatorial geometry of the training data.

Abstract

High-Dimensional Linear Regression

Explore similar work

Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias

Estimating Implicit Regularization in Deep Learning