Overview

Implicit bias is the tendency of a model and its training procedure to select some solutions or representations over others, even without an explicit regularizer; these preferences shape generalization, extrapolation, and sometimes fairness. Researchers derive these preferences from gradient descent, gradient flow, and stochastic optimization in settings such as separable classification, deep linear and ReLU networks, matrix factorization, and attention, often finding max-margin, sparse, or low-rank solutions. They vary initialization, depth, parameterization, optimizer, step size, and noise, and test consequences in tasks including length generalization, shortcut learning, and demographic representations in language and vision-language models. Current work is moving beyond asymptotic results in simplified models toward finite-time dynamics and realistic architectures, but the selected bias can shift during training and improve one notion of generalization while worsening another; fair outputs can also conceal biased internal representations.

Latest papers

CardsList

36 papers7 papers in the last four weeks, up 75% on the four weeks before. 0.0% of all new papers.

  1. How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

    Mar 5, 2026Kuo-Wei Lai, Guanghui Wang, Molei Tao +1High-Dimensional Linear RegressionRandom Feature Methods

  2. To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

    Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network GeneralizationMuon Optimizer

  3. Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective

    Jan 10, 2026Nicola Aladrah, Emanuele Ballarin, Matteo Biagetti +3Implicit BiasNeural Network Training Dynamics

  4. Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

    Dec 16, 2025Michael Murray, Tenzin Chan, Kedar Karhadker +1Sample ComplexityNeural Network Memorization