cs.LGFeb 27, 2024

Implicit Bias via Spectral Neural Networks and Non-linear Matrix Sensing

Authors: Hong T. M. Chu, Xiaoyu Dong, Subhro Ghosh, Chi Thanh Lam, Soumendu Sundar Mukherjee

Organizations: Dept of Mathematics, National University of Singapore · School of Computing,National University of Singapore · Statistics and Mathematics Unit, Indian Statistical Institute

Abstract

The phenomenon of implicit bias has attracted interest in recent years as a fundamental aspect of the remarkable generalizing ability of neural networks. In a nutshell, it entails that gradient descent dynamics in many neural nets, even without any explicit regularizer in the loss function, converges to the solution of a regularized learning problem. In this work, we show that this phenomenon extends to non-linear spectral parameterizations in matrix sensing. More precisely, we propose a matrix network architecture called Spectral Neural Networks (abbrv. SNN) where each neuron applies nonlinear activation to the singular values of the input matrix. Conceptually, this entails coordinatizing the space of matrices by their singular values and singular vectors, as opposed to by their entries, a potentially fruitful perspective for matrix learning. We prove the exponentially fast decay of squared loss during the training dynamics of an SNN parameterized matrix sensing model under gradient flow (which is a continuous approximation of the discrete gradient descent), and also that the training implicitly selects an approximate solution that minimizes nuclear norm. We also demonstrate the effectiveness of the SNN architecture in the context of matrix sensing by empirical investigations. We believe that the SNN architecture has the potential to be of wide applicability in a broad class of matrix learning scenarios.

Figures & tables

Explore similar work

CardsList
  1. A Unified Matrix-Spectral Framework for Stability and Interpretability in Deep Learning

    Feb 1, 2026Ronald KatendeNeural Network StabilityNeural Network Robustness

  2. The Spectral Neuron

    Aug 8, 2026Alex ShtoffNeural Network RobustnessNeural Network Interpretability

  3. Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks

    Oct 29, 2026Wiebke Bartolomaeus, Holger RauhutImplicit BiasImplicit Regularization