Bias-Only Training Dynamics in ReLU Networks
Organizations: Department of Electrical and Computer Engineering, Carnegie Mellon University
Abstract
Neural networks with randomly initialized, fixed weights and trainable biases are universal approximators, yet the dynamics of bias-only training remain largely unexplored. We study these dynamics in one-hidden-layer ReLU networks through activation codes, the binary patterns of active neurons that partition the input space into convex polyhedral regions. We show that the gradients with respect to the biases depend on each input only through its activation code, and hence the training dynamics can be characterized with these codes. In particular, as long as the activation codes remain fixed, training reduces to a linear least-squares problem with the codes as features. Training can then be understood as a sequence of least-squares problems, with transitions occurring when neurons change their activation states. We show that the fraction of switching neurons vanishes as the network widens, and the training dynamics approach those of linear least squares on the initial activation codes in the infinite-width limit. Our experiments support these theoretical findings, showing that while activation switches can noticeably affect training in narrower networks, their relative frequency and the associated changes in the dynamics diminish as the width increases.
Figures & tables
| Flipped bits | Kernel | Fit | |||||
| test | drift | to | loss | vs. LS | |||
| 256 | 0.125 | 32 | 0.125 | 0.065 | 0.073 | 0.17 | |
| 1024 | 0.056 | 57 | 0.056 | 0.021 | 0.038 | 0.066 | |
| 4096 | 0.029 | 117 | 0.028 | 0.0077 | 0.018 | 0.021 | |
| 16384 | 0.014 | 231 | 0.014 | 0.0028 | 0.0079 | 0.015 | |
| slope | – | ||||||
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 553 papers