Implicit Bias in Deep Linear Discriminant Analysis

Jiawen Li

cs.LGMar 3, 2026

Implicit Bias in Deep Linear Discriminant Analysis

Authors: Jiawen Li

Organizations: School of Computer Science and Engineering, University of New South Wales, Kensington, Sydney 2052, New South Wales, Australia

Abstract

While the Implicit Bias(or Implicit Regularization) of standard loss functions has been studied, the optimization geometry induced by discriminative metric-learning objectives remains largely unexplored.To the best of our knowledge, this paper presents an initial theoretical analysis of the implicit regularization induced by the Deep LDA,a scale invariant objective designed to minimize intraclass variance and maximize interclass distance. By analyzing the gradient flow of the loss on a L-layer diagonal linear network, we prove that under balanced initialization, the network architecture transforms standard additive gradient updates into multiplicative weight updates, which demonstrates an automatic conservation of the (2/L) quasi-norm.

Figures & tables

Figure 1: The Optimization in Simplex and the Implicit Bias

Figure 2: The Simulated Result for DeepLDA in DLNs.

Dataset	$L=1$	$L=2$	$L=5$	$L=10$	$L=20$
Simulated ( $d\!=\!5$ )	0.466%	0.226%	0.157%	0.194%	0.562%
Iris ( $d\!=\!4$ )	0.076%	0.031%	0.020%	0.017%	0.018%
Wine ( $d\!=\!13$ )	0.239%	0.106%	0.071%	0.083%	0.209%
Breast Cancer ( $d\!=\!30$ )	0.038%	0.016%	0.010%	0.009%	0.009%

Table 1: Quasi-Norm Conservation: Maximum Deviation

Figure 3: Depth-induced feature sparsity across datasets

Implicit Regularization

Implicit regularization studies how training algorithms and model architectures favor some solutions over others without explicit penalty terms.

See all 46 papers

Estimating Implicit Regularization in Deep Learning
May 6, 2026Joseph H. Rudoler, Kevin Tan, Giles Hooker +1Neural Network Optimization Implicit Bias
University of Pennsylvania
Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization
Mar 9, 2026Chaewon Moon, Dongkuk Si, Chulhee YunSharpness-Aware Minimization Implicit Bias
Kim Jaechul Graduate School of AI, KAIST · Mobilint, Inc.
Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization
Jul 14, 2026Jiajie Zhao, Jianxing Wang, Junjie Yang +2Deep Linear Networks Implicit Bias
School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Artificial Intelligence, Shanghai Jiao Tong University

May 6, 2026J/K move · Enter open · S save

Estimating Implicit Regularization in Deep Learning

Joseph H. Rudoler, Kevin Tan, Giles Hooker, Konrad P. Kording

University of Pennsylvania

Neural Network Optimization Implicit Bias Implicit Regularization

Open paper AskPDF arXiv

Deep learning systems are known to exhibit implicit regularization (alt. implicit bias), favoring simple solutions instead of merely minimizing the loss function. In some cases, we can analytically derive the implicit regularization -- connecting it to an equivalent penalty that augments the learning objective. However, modern deep learning systems are complex, carrying modifications to the training procedure and architecture (e.g. early stopping, minibatching, dropout) whose effects are not always directly interpretable. Although estimating the resulting implicit regularization could aid theorists in algorithm design and practitioners in interpreting their hyperparameter choices, this problem has received little direct attention. It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization. Our method works on networks with known regularization, recovering popular explicit penalties like

\ell_1

and

\ell_2

. It also replicates known implicit effects, like the quadratic weight penalty induced by early stopping in gradient descent, demonstrating that it can be used to test theories of implicit regularization. Crucially, because our method is empirical, it can handle implicit regularization in arbitrary networks. We demonstrate this use by characterizing the effects of dropout in deep networks, showing implicit

\ell_2

effects in this popular method. Our work shows that practitioners can use gradient matching to understand regularization in networks with implicit biases that are too complicated to derive analytically.

Abstract

Implicit Regularization

Explore similar work

Estimating Implicit Regularization in Deep Learning

Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization

Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization