cs.LGNov 7, 2026

Spectral Bias Reduction in Local Gated ReLU Architectures

Authors: Victorita Dolean

Organizations: Department of Mathematics and Computer Science, Eindhoven University of Technology, PO Box 513, Eindhoven, 5600 MB, The Netherlands

Abstract

Neural networks trained by gradient-based methods learn the low-frequency content of a target function faster than its high-frequency content. This spectral bias is well documented, but it is much less clear which architectural choices mitigate it and which cannot. We study the question for shallow ReLU models linear in their trainable parameters, where each mode is learned at a rate given by a Gram-matrix eigenvalue, and for a general symmetric bilinear form a(⋅,⋅)a(\cdot,\cdot), so that least-squares regression (a=a= the L2L^2 inner product) and variational or physics-informed losses (a=a= the H1H^1 energy form) are two instances of one analysis. Within this framework we propose gated local ReLU dictionaries, obtained by multiplying each ReLU feature by a hat partition-of-unity weight, in the spirit of partition-of-unity finite elements and overlapping Schwarz methods. The resulting Gram matrices are block-banded for any local bilinear form, with a block-Toeplitz interior whose matrix-valued symbol governs the spectrum. The two forms then behave very differently. For L2L^2 regression, the symbol is bounded away from zero, the ratio is uniform in the number of windows, and the measured spread of frequency-dependent learning rates falls from 1.9×1051.9\times10^{5} to 4.54.5 at matched model size. For the H1H^1 stiffness matrix, the symbol vanishes at zero frequency: gating leaves the Θ(N2)Θ(N^2) conditioning intact and inverts the frequency ordering instead of removing the bias. A preconditioner built from the same interior stencil will fix the spectral bias, showing that architecture alone is not enough. Experiments on 11D regression and Poisson problems confirm the dichotomy, and the regression effect persists when the hidden-layer parameters are trained as well.

Figures & tables

Explore similar work

CardsList
  1. Implicit Bias via Spectral Neural Networks and Non-linear Matrix Sensing

    Feb 27, 2024Hong T. M. Chu, Xiaoyu Dong, Subhro Ghosh +2Implicit RegularizationNeural Network Training Dynamics

  2. How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

    Mar 5, 2026Kuo-Wei Lai, Guanghui Wang, Molei Tao +1High-Dimensional Linear RegressionRandom Feature Methods

  3. Spectral bias in physics-informed and operator learning: Analysis and mitigation guidelines

    Feb 22, 2026Siavash Khodakarami, Vivek Oommen, Nazanin Ahmadi Daryakenari +2Spectral BiasPhysics-Informed ML