cs.LGNov 18, 2026

Bias-Only Training Dynamics in ReLU Networks

Authors: Shreyas Chaudhari, Jose M. F. Moura

Organizations: Department of Electrical and Computer Engineering, Carnegie Mellon University

Abstract

Neural networks with randomly initialized, fixed weights and trainable biases are universal approximators, yet the dynamics of bias-only training remain largely unexplored. We study these dynamics in one-hidden-layer ReLU networks through activation codes, the binary patterns of active neurons that partition the input space into convex polyhedral regions. We show that the gradients with respect to the biases depend on each input only through its activation code, and hence the training dynamics can be characterized with these codes. In particular, as long as the activation codes remain fixed, training reduces to a linear least-squares problem with the codes as features. Training can then be understood as a sequence of least-squares problems, with transitions occurring when neurons change their activation states. We show that the fraction of switching neurons vanishes as the network widens, and the training dynamics approach those of linear least squares on the initial activation codes in the infinite-width limit. Our experiments support these theoretical findings, showing that while activation switches can noticeably affect training in narrower networks, their relative frequency and the associated changes in the dynamics diminish as the width increases.

Figures & tables

Explore similar work

CardsList
  1. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

    Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural NetworksReLU Neural Networks

  2. Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations

    Apr 9, 2026Fabricio Macià, Shu NakamuraShallow Neural NetworksNeural Network Training Dynamics