Implicit Bias

Ask about this topic

Overview

Implicit bias is the tendency of a model and its training procedure to select some solutions or representations over others, even without an explicit regularizer; these preferences shape generalization, extrapolation, and sometimes fairness. Researchers derive these preferences from gradient descent, gradient flow, and stochastic optimization in settings such as separable classification, deep linear and ReLU networks, matrix factorization, and attention, often finding max-margin, sparse, or low-rank solutions. They vary initialization, depth, parameterization, optimizer, step size, and noise, and test consequences in tasks including length generalization, shortcut learning, and demographic representations in language and vision-language models. Current work is moving beyond asymptotic results in simplified models toward finite-time dynamics and realistic architectures, but the selected bias can shift during training and improve one notion of generalization while worsening another; fair outputs can also conceal biased internal representations.

Latest papers

CardsList

30 papers4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. RAGing Against Misogyny: Improving Implicit Misogyny Explanations

    Oct 26, 2026Ivana Crescenzi, Arianna Muti, Maarten Sap +2Implicit BiasSocial Bias in Language Models

    Department of Computing Sciences, Bocconi University, Milan, Italy · Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA · Center for Language and Cognition Groningen, University of Groningen, The Netherlands

  2. A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability

    Oct 16, 2026Quan Nguyen, Robin Yadav, Param Mody +1Neural Network OptimizationReLU Neural Networks

    University of British Columbia · Canada CIFAR AI Chair (Amii)

  3. From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon

    Oct 8, 2026Jihwan Kim, Dogyoon Song, Chulhee YunAdaptive Gradient MethodsClassification

    Seoul National University · KAIST InnoCORE LLM · University of California, Davis +1

  4. The Implicit Bias of Hyperbolic Representation Learning for Multiclass Data: A Busemann Risk Perspective

    Oct 5, 2026Xingrun Li, Sho Kuno, Yusuke Mukuta +2Riemannian OptimizationImplicit Bias

    The University of Tokyo · RIKEN

  5. Learning the identity: a case study of how SGD selects among functional decompositions

    Sep 30, 2026Andy Arditi, Weian Xie, David Bau +1Deep Linear NetworksImplicit Bias

    Northeastern University · Massachusetts Institute of Technology

  6. On the Two Faces of Adam in Separable Linear Classification

    Sep 27, 2026Chen Fan, Csaba SzepesváriImplicit BiasClassification

    University of Alberta

  7. High-Dimensional Learning Dynamics of Attention-Indexed Models

    Sep 3, 2026Yizhou Xu, Margarita Sagitova, Lenka Zdeborová +1Implicit BiasAttention Mechanisms

    1Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · 2Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

  8. Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

    Aug 5, 2026Han BaoLogistic RegressionImplicit Bias

    The Institute of Statistical Mathematics · Tohoku University · RIKEN AIP

  9. Variable Selection in the Context of AI Fairness

    Aug 4, 2026Ivan Luciano Danesi, Chiara Frigerio, Fabio Maccaferri +2Algorithmic FairnessGroup Fairness

    UniCredit S.p.A., Milan, Italy. · 2Universit`a Cattolica del Sacro Cuore, Milan, Italy. · 3Cetif, Milan, Italy.

  10. From Minds to Models: The Intersection of Psychology and LLM Behaviours

    Jul 30, 2026Oliver Guidetti, Reza RyanSocial Bias in Language ModelsImplicit Bias

    School of Psychology, Faculty of Science, Medicine and Health, University of Wollongong, Wollongong, NSW 2522, Australia. · School of Electrical Engineering, Computing and Mathematical Sciences, Faculty of Science and Engineering, Curtin University, Bentley, WA 6102, Australia.

  11. Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

    Jul 21, 2026Subham Singh, Ashutosh Mishra, Subha RautSoftmax AttentionRotary Positional Embeddings

    Department of Mathematics and Statistics, Mississippi State University · Independent Researcher

  12. Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization

    Jul 14, 2026Jiajie Zhao, Jianxing Wang, Junjie Yang +2Deep Linear NetworksImplicit Bias

    School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Artificial Intelligence, Shanghai Jiao Tong University

  13. The Anatomy of Implicit Bias: Information Allocation in Neural Network Training

    Jul 8, 2026Zhang Gongyue, Wang Zhiyong, Liu Donghan +3Neural Network OptimizationImplicit Bias

    1State Key Laboratory of Robotics and Systems, Harbin Institute of Technology Shenzhen, Shenzhen, 518055, China.

  14. The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System

    Jul 6, 2026Thomas HofmannEdge of StabilityGradient Descent

    Department of Computer Science, ETH Zurich

  15. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

    Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural NetworksReLU Neural Networks

    Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences

  16. Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima

    Jun 18, 2026Md Sakir Ahmed, Kumaresh Sarmah, Hemen DuttaInformation GeometryImplicit Bias

    Department of Electronics and Communication Technology Gauhati University Guwahati, Assam, India · Department of Mathematics Gauhati University Guwahati, Assam, India

  17. Vision-Language Models Suppress Female Representations Under Ambiguous Input

    May 29, 2026Arnau Marin-Llobet, Simon Henniger, Mahzarin R. BanajiVLM EvaluationVLM Interpretability

    School of Engineering and Applied Sciences · Department of Psychology

  18. The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

    May 21, 2026Connall Garrod, Jonathan P. Keating, Christos ThrampoulidisDeep Linear NetworksImplicit Bias

    Mathematical Institute, University of Oxford · Department of Electrical and Computer Engineering, University of British Columbia

  19. Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning

    May 19, 2026Tom Jacobs, Guido MontufarImplicit BiasNeural Network Training Dynamics

    CISPA Helmholtz Center

  20. Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisions

    May 12, 2026Jagdish Tripathy, Marcus BuckmannLLM AuditingImplicit Bias

    Bank of England

  21. Estimating Implicit Regularization in Deep Learning

    May 6, 2026Joseph H. Rudoler, Kevin Tan, Giles Hooker +1Neural Network OptimizationImplicit Bias

    University of Pennsylvania

  22. Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints

    Mar 24, 2026Shengping Xie, Zekun Wu, Quan Chen +1Implicit BiasGradient Descent

    School of Mathematical Sciences, Peking University

  23. Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime

    Mar 11, 2026Reza Ghane, Danil Akhtiamov, Babak HassibiGradient DescentImplicit Bias

    Electrical Engineering Department, California Institute of Technology, Pasadena, CA 91125 USA · Computing + Mathematical Sciences Department, California Institute of Technology, Pasadena, CA 91125 USA

  24. Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization

    Mar 9, 2026Chaewon Moon, Dongkuk Si, Chulhee YunSharpness-Aware MinimizationImplicit Bias

    Kim Jaechul Graduate School of AI, KAIST · Mobilint, Inc.

  25. Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts

    Mar 7, 2026Truong Xuan Khanh, Truong Quynh HoaNeural Network Training DynamicsRepresentation Learning

    H&K Research Studio, Clevix LLC, Hanoi, Vietnam

  26. How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

    Mar 5, 2026Kuo-Wei Lai, Guanghui Wang, Molei Tao +1High-Dimensional Linear RegressionRandom Feature Methods

    School of Electrical and Computer Engineering, Georgia Institute of Technology · College of Computing, Georgia Institute of Technology · School of Mathematics, Georgia Institute of Technology +1

  27. Implicit Bias and Loss of Plasticity in Matrix Completion: Depth Promotes Low-Rankness

    Mar 5, 2026Baekrok Shin, Chulhee YunImplicit BiasNeural Network Training Dynamics

    Kim Jaechul Graduate School of AI, KAIST

  28. To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

    Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathNeural Network GeneralizationMuon Optimizer

    Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London

  29. Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective

    Jan 10, 2026Nicola Aladrah, Emanuele Ballarin, Matteo Biagetti +3Implicit BiasNeural Network Training Dynamics

    Department of Mathematics, Informatics and Geoscience, University of Trieste, Via Valerio 12/1, 34127 Trieste, Italy · Area Science Park, Padriciano, 34149 Trieste, Italy · McGovern Institute, MIT, Main Street, Cambridge, MA 02139, USA

  30. Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

    Dec 16, 2025Michael Murray, Tenzin Chan, Kedar Karhadker +1Sample ComplexityNeural Network Memorization

    Mathematical Sciences, University of Bath · Department of Mathematics, UCLA · Algebraic