Neural networks trained by gradient-based methods learn the low-frequency content of a target function faster than its high-frequency content. This spectral bias is well documented, but it is much less clear which architectural choices mitigate it and which cannot. We study the question for shallow ReLU models linear in their trainable parameters, where each mode is learned at a rate given by a Gram-matrix eigenvalue, and for a general symmetric bilinear form a(⋅,⋅), so that least-squares regression (a= the L2 inner product) and variational or physics-informed losses (a= the H1 energy form) are two instances of one analysis. Within this framework we propose gated local ReLU dictionaries, obtained by multiplying each ReLU feature by a hat partition-of-unity weight, in the spirit of partition-of-unity finite elements and overlapping Schwarz methods. The resulting Gram matrices are block-banded for any local bilinear form, with a block-Toeplitz interior whose matrix-valued symbol governs the spectrum. The two forms then behave very differently. For L2 regression, the symbol is bounded away from zero, the ratio is uniform in the number of windows, and the measured spread of frequency-dependent learning rates falls from 1.9×105 to 4.5 at matched model size. For the H1 stiffness matrix, the symbol vanishes at zero frequency: gating leaves the Θ(N2) conditioning intact and inverts the frequency ordering instead of removing the bias. A preconditioner built from the same interior stencil will fix the spectral bias, showing that architecture alone is not enough. Experiments on 1D regression and Poisson problems confirm the dichotomy, and the regression effect persists when the hidden-layer parameters are trained as well.
Figures & tables
K
λmin(h−1Agate,K)
λmax(h−1Agate,K)
λmin(TK+1)
λmax(TK+1)
BH1
24
1.16×10−2
5.285
4.05×10−2
5.293
32.2
26
7.83×10−4
5.330
3.02×10−3
5.330
441
28
4.99×10−5
5.333
1.98×10−4
5.333
6.74×103
210
3.13×10−6
5.333
1.25×10−5
5.333
1.07×105
212
1.96×10−7
5.333
7.84×10−7
5.333
1.70×106
214
1.23×10−8
5.333
4.90×10−8
5.333
2.72×107
Table 1: Gated stiffness matrix at m=1 . The symbol vanishes at ζ=0 , so λmin decays as K−2 while λmax saturates at 16/3 , and the spectral-bias factor grows at the same Θ(K2) rate: conditioning and frequency disparity degrade together.
Form a
L2 inner product
H1 semi-inner-product
Gram matrix
mass GgateK
stiffness Agate,K
Universal scale
K−3 (Lemma 4 , α=3 )
K−1 ( α=1 )
Symbol minimum
mfL2>0
mfH1=0 at ζ=0
λmin scale
Θ(K−3) ; uniform after rescaling
Θ(K−3) ; not uniform after rescaling
Condition no.
κ≤CmL2/cmL2 , K -uniform
κ(Agate,K)=Θ(N2)
Spectral-bias factor Ba
≤MfL2/mfL2 , K -uniform ( 2.14 at m=1 )
Θ(K2) ; ordering inverted
Table 2: The gated dictionary under the two bilinear forms. The Gram matrices share the same block-banded, block-Toeplitz structure; only the symbol differs, and its minimum mfa decides every row below it. The λmin row agrees by coincidence of the two scalings, Θ(K−3)⋅Θ(1) against Θ(K−1)⋅Θ(K−2) .
κ
BH1
m
K
unpre.
Pd
Pt
Pc
unpre.
Pc
1
16
108.4
107.5
13.53
10.68
32.2
2.04
1
32
423.4
423.1
13.88
11.55
116
2.85
1
64
1677
1677
14.07
18.96
441
4.47
4
16
425.3
139.4
28.78
22.74
414
3.11
4
32
852.6
554.4
29.40
24.08
426
4.52
Table 3: Stiffness matrix with u(1)=0 imposed, under the three preconditioners: condition number on the left, spectral-bias factor over the gate-resolved band on the right. Only Pt is uniform in K in both. It gives BH1=1 exactly in every row, and Pd leaves BH1 unchanged at m=1 , so both are omitted on the right.
configuration
S
Ba
κ
measured
predicted
predicted
L2 , global ( N=64 )
1.9×105
—
6.7×107
L2 , gated ( K=31 , m=2 )
4.5
1.54
189
H1 , global ( N=64 )
2.23
—
824
H1 , gated ( K=31 , m=2 )
11.0
10.5
533
H1 , gated +Pt
4.36
1
14.0
Table 4: Measured against predicted, at the sizes of Figures 2 and 4 . Only S comes from the training runs; Ba is read from the symbol over q=1,…,10 , and κ from the assembled matrix, neither needing any training. Ba exists only for the gated, block-Toeplitz dictionaries, and predicts S closely where κ does not. The H1 rows are the reduced system the dynamics run on, with the Dirichlet constraint eliminated.
Figure 1: Condition number against the number of features N . Left, the L2 mass matrix: the global dictionary grows quartically (slope 3.97 ), the gated one stays flat at m=1,2,4 . Right, the H1 stiffness matrix: both grow with the same exponent (slopes ≈2 ). The predictions are Theorems 8 (a) and 11 (b).
Figure 2: L2 regression, linear in parameters. Color is ∣f~q(t)∣/Aq clipped to [0,1] against the dimensionless time τ=μmaxt of ( 46 ), on a log axis shared across panels. Left: the global dictionary, showing the staircase of Rahaman et al. (2019) , spread 1.9×105 . Centre: the gated dictionary, all ten frequencies together, spread 4.5 . Right: the rise times, whose spread is S .
Figure 3: L2 regression with trainable networks, same metric and color scale as Figure 2 . The gated network resolves all ten frequencies; the global network resolves only q=1 within the budget, and open symbols in the right-hand panel mark the frequencies it never reaches.
Figure 4: Poisson / H1 with u(0)=u(1)=0 imposed; same metric and time axis as Figure 2 . Left to right: global (spread 2.23 ); gated (spread 11.0 , ordering inverted, q=1 slowest); gated with the block-Toeplitz preconditioner (spread 4.36 ); the rise times. The rising middle panel is RH1(q)∝q2 of Theorem 11 (c); the preconditioner flattens it, as Theorem 12 (a) requires.
target
global
gated
ratio
equal amplitudes
1.9×105
4.5
4.2×104
increasing amplitudes
5.8×106
6.6
8.9×105
random phases
5.0×103
5.9
8.4×102
high band only ( k=6,…,10 )
6.0
2.8
2.1
two extremes ( k∈{1,10} )
5.2×103
2.9
1.8×103
Table 5: Spread S=maxqτq/minqτq for L2 regression across five targets, at N=64 (global) and K=31 , m=2 (gated). The gated dictionary stays within a factor of seven in every case.
Dept of Mathematics, National University of Singapore · School of Computing,National University of Singapore · Statistics and Mathematics Unit, Indian Statistical Institute
School of Electrical and Computer Engineering, Georgia Institute of Technology · College of Computing, Georgia Institute of Technology · School of Mathematics, Georgia Institute of Technology +1