cs.LGSep 25, 2026

Population loss in shallow ReLU networks: Bias & families of critical points

Authors: Michael Field

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The main result presented is a formula for the population loss in the student-teacher kernel model that is applicable to shallow ReLU networks with bias. This extends previous work of Choo and Saul (2009) and Brutzkus and Globerson (2017). The formula makes essential use of Owen's T-function. The necessary theory of the T-function is given and a high precision coding using MPFR for the T-function, based on an algorithm of Komelj (2023), is available on request. It is shown that various families of spurious minima described in past papers of Arjevani and the author extend to biased networks and that the loss is always strictly decreased when bias is added. The change in landscape geometry caused by adding bias appears to be relatively mild. Only the simplest examples are described in this paper where it is assumed that the number of inputs is equal to the number of neurons (this restriction is for reasons of length). A review of relevant previous results on unbiased networks is included. Aside from Gaussian statistics, the main mathematical tools and ideas come from analytic geometry (analytic and subanalytic sets, the Curve Selection Lemma).

Figures & tables

Figure 1
Figure 1. The region of integration in the x1,x2x_{1},x_{2} -plane with boundary shown in red and bˉ<0<βˉ\bar{b}<0<\bar{\beta} and θ∈(0,π/2]\theta\in(0,\pi/2] . As θ\theta increases, the line LL defined by x1cos⁡(θ)+x2sin⁡(θ)=βˉx_{1}\cos(\theta)+x_{2}\sin(\theta)=\bar{\beta} rotates counter clockwise so that when θ∈(π/2,π)\theta\in(\pi/2,\pi) , LL intersects the negative x1x_{1} -axis.
Figure 2. The region of integration on the strip [γ‾,β‾]×R[\overline{\gamma},\overline{\beta}]\times{\mathbb{R}} , θ\theta near zero and β‾<γ‾\overline{\beta}<\overline{\gamma} .
d=kd=kValue of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0})
660.71830101072557148145×6−20.71830101072557148145\times 6^{-2}
10100.33399917189077800002×10−20.33399917189077800002\times 10^{-2}
20200.10183336077334108245×10−20.10183336077334108245\times 10^{-2}
1001000.49862816276459333028×10−40.49862816276459333028\times 10^{-4}
100010000.53292271608433825762×10−60.53292271608433825762\times 10^{-6}
10610^{6}0.54139203214105478639×10−120.54139203214105478639\times 10^{-12}
Table 1. Numerically computed values of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0}) for a type II critical point with k=dk=d and isotropy ΔSd−1\Delta S_{d-1}
d=kd=kValue of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0})
10100.77142020703795495629×10−20.77142020703795495629\times 10^{-2}
1001000.1220227002721111972×10−20.1220227002721111972\times 10^{-2}
100010000.13745083160523485590×10−30.13745083160523485590\times 10^{-3}
10610^{6}0.14472588806870954343×10−60.14472588806870954343\times 10^{-6}
101210^{12}0.14496749467704066784×10−120.14496749467704066784\times 10^{-12}
Table 2. Numerically computed values of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0}) for a type I critical point with k=dk=d and isotropy ΔSd\Delta S_{d}
Type I  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy ΔSd−1\Delta S_{d-1}  
10100.7468×10−20.7468\times 10^{-2}0.7283×10−20.7283\times 10^{-2}
100010000.1374×10−30.1374\times 10^{-3}0.1769×10−30.1769\times 10^{-3}
10610^{6}0.1473×10−60.1473\times 10^{-6}0.1969×10−60.1969\times 10^{-6}
Table 3. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type I critical point with m=0m=0 and isotropy ΔSd−1\Delta S_{d-1} .
Type I   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy ΔSd−1\Delta S_{d-1}   
10100.3532×10−20.3532\times 10^{-2}0.4842×10−20.4842\times 10^{-2}0.1070×10−10.1070\times 10^{-1}
100010000.1318×10−30.1318\times 10^{-3}0.4658×10−30.4658\times 10^{-3}0.4885×10−30.4885\times 10^{-3}
10610^{6}0.1446×10−60.1446\times 10^{-6}0.2495×10−50.2495\times 10^{-5}0.2513×10−50.2513\times 10^{-5}
Table 4. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type I critical point with m=1m=1 and isotropy ΔSd−1\Delta S_{d-1} .
Type I    
k=d+2k=d+2∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}∂L/∂β‾3\partial\mathcal{L}/\partial\overline{\beta}_{3}
Isotropy ΔSd−1\Delta S_{d-1}    
10100.2760×10−20.2760\times 10^{-2}0.4500×10−20.4500\times 10^{-2}0.4969×10−10.4969\times 10^{-1}0.7548×10−20.7548\times 10^{-2}
100010000.1295×10−30.1295\times 10^{-3}0.4396×10−30.4396\times 10^{-3}0.4006×10−30.4006\times 10^{-3}0.1937×10−30.1937\times 10^{-3}
10610^{6}0.1449×10−60.1449\times 10^{-6}0.2483×10−50.2483\times 10^{-5}0.2420×10−50.2420\times 10^{-5}0.1850i×10−60.1850i\times 10^{-6}
Table 5. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈4i\in{\mathbf{4}} , for a type I critical point with m=2m=2 and isotropy ΔSd−1\Delta S_{d-1} .
Type II  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy ΔSd−1\Delta S_{d-1}  
10100.3340×10−20.3340\times 10^{-2}−0.7436×10−2-0.7436\times 10^{-2}
100010000.5329×10−60.5329\times 10^{-6}−0.2304×10−3-0.2304\times 10^{-3}
10610^{6}0.5414.×10−120.5414.\times 10^{-12}−0.2371×10−6-0.2371\times 10^{-6}
Table 6. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type II critical point with m=0m=0 and isotropy ΔSd−1\Delta S_{d-1} .
Type II   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy ΔSd−1\Delta S_{d-1}   
10100.2605×10−20.2605\times 10^{-2}−0.6054×10−2-0.6054\times 10^{-2}−0.2055×10−2-0.2055\times 10^{-2}
10001000−0.1011×10−5-0.1011\times 10^{-5}−0.2062×10−3-0.2062\times 10^{-3}−0.7271×10−5-0.7271\times 10^{-5}
10610^{6}−0.5076×10−10-0.5076\times 10^{-10}−0.2131×10−6-0.2131\times 10^{-6}−0.2369×10−9-0.2369\times 10^{-9}
Table 7. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type I critical point with m=1m=1 and isotropy ΔSd−1\Delta S_{d-1} .
Type II    
k=d+2k=d+2∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}∂L/∂β‾3\partial\mathcal{L}/\partial\overline{\beta}_{3}
Isotropy ΔSd−1\Delta S_{d-1}    
10100.2590×10−20.2590\times 10^{-2}- 0.5638×10−20.5638\times 10^{-2}0.1317×10−20.1317\times 10^{-2}0.1147×10−20.1147\times 10^{-2}
10001000−1.018×10−6-1.018\times 10^{-6}−0.2052×10−3-0.2052\times 10^{-3}−5.682×10−6-5.682\times 10^{-6}−2.536×10−6-2.536\times 10^{-6}
10610^{6}−5.099×10−11-5.099\times 10^{-11}−0.2130×10−6-0.2130\times 10^{-6}−1.86×10−10-1.86\times 10^{-10}−8.051×10−11-8.051\times 10^{-11}
Table 8. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈4i\in{\mathbf{4}} , for a type II critical point with m=2m=2 and isotropy ΔSd−1\Delta S_{d-1} .
Type I  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}  
10100.7160×10−20.7160\times 10^{-2}0.6965×10−20.6965\times 10^{-2}
100010000.1374×10−30.1374\times 10^{-3}0.1769×10−30.1769\times 10^{-3}
10610^{6}0.1447×10−60.1447\times 10^{-6}0.1969×10−60.1969\times 10^{-6}
Table 9. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type I critical point with m=0m=0 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type I   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}   
10100.2840×10−20.2840\times 10^{-2}0.4030×10−20.4030\times 10^{-2}1.087×10−21.087\times 10^{-2}
100010000.1304×10−30.1304\times 10^{-3}0.2585×10−30.2585\times 10^{-3}0.4299×10−30.4299\times 10^{-3}
10610^{6}0.1445×10−100.1445\times 10^{-10}0.3027×10−60.3027\times 10^{-6}0.4764×10−90.4764\times 10^{-9}
Table 10. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type I critical point with m=1m=1 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type I    
k=d+2k=d+2∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}∂L/∂β‾3\partial\mathcal{L}/\partial\overline{\beta}_{3}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}    
10100.2186×10−20.2186\times 10^{-2}0.3591×10−20.3591\times 10^{-2}0.5519×10−20.5519\times 10^{-2}0.6705×10−20.6705\times 10^{-2}
100010000.1298×10−30.1298\times 10^{-3}0.2544×10−30.2544\times 10^{-3}0.3176×10−30.3176\times 10^{-3}0.1431×10−30.1431\times 10^{-3}
10610^{6}0.1445×10−60.1445\times 10^{-6}0.2996×10−60.2996\times 10^{-6}0.3636×10−60.3636\times 10^{-6}0.1346×10−60.1346\times 10^{-6}
Table 11. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈4i\in{\mathbf{4}} , for a type I critical point with m=2m=2 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type II  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}  
10100.6278×10−20.6278\times 10^{-2}−0.4145×10−2-0.4145\times 10^{-2}
100010001.065×10−61.065\times 10^{-6}−0.2296×10−3-0.2296\times 10^{-3}
10610^{6}1.083×10−61.083\times 10^{-6}−2.371×10−3-2.371\times 10^{-3}
Table 12. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type II critical point with m=0m=0 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type II   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}   
10100.4799×10−20.4799\times 10^{-2}−0.3701×10−2-0.3701\times 10^{-2}−2.594×10−2-2.594\times 10^{-2}
10001000−1.188×10−6-1.188\times 10^{-6}−0.2042×10−3-0.2042\times 10^{-3}−1.242×10−5-1.242\times 10^{-5}
10610^{6}−7.244×10−11-7.244\times 10^{-11}−0.2092×10−6-0.2092\times 10^{-6}−4.060×10−10-4.060\times 10^{-10}
Table 13. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type II critical point with m=1m=1 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type II  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy Gd−p,pΔG^{\Delta}_{d-p,p}  
p=3p=3  
d=10d=100.8835×10−20.8835\times 10^{-2}−0.1100×10−2-0.1100\times 10^{-2}
d=106d=10^{6}1.624×10−121.624\times 10^{-12}−0.2372×10−6-0.2372\times 10^{-6}
Initial Coeff. of FPSD40=1.625D^{0}_{4}=1.625D20=−2.373D^{0}_{2}=-2.373
Table 14. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for type II critical points with p=3,4p=3,4 and d−p=pd-p=p If d−p=pd-p=p , the diagonal entries of the upper diagonal block converge to +1+1 as d→∞d{\rightarrow}\infty .
Type II, m=0       
 ∂L∂β‾0\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}∂L∂β‾1\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{1}}∂L∂β‾2\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{2}}∂L∂β‾3\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{3}}∂L∂β‾4\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{4}}∂L∂β‾5\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{5}}∂L∂β‾6\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{6}}
Isotropy ΔSd−6\Delta S_{d-6}       
d=10d=103.8e-33.8\text{e-3}3.4e-33.4\text{e-3}3.2e-33.2\text{e-3}4.0e-34.0\text{e-3}2.9e-3e-32.9\text{e-3}\text{e-3}4.4e-34.4\text{e-3}−8.4e-3-8.4\text{e-3}
d=1000d=10006.1e-76.1\text{e-7}−1.6e-6-1.6\text{e-6}−8.0e-6-8.0\text{e-6}1.0e-51.0\text{e-5}−1.4e-5-1.4\text{e-5}9.0e-69.0\text{e-6}−2.8e-4-2.8\text{e-4}
d=106d=10^{6}6.2e-136.2\text{e-13}−2.2e-9-2.2\text{e-9}−8.7e-9-8.7\text{e-9}9.7e-99.7\text{e-9}−1.5e-8-1.5\text{e-8}8.5e-98.5\text{e-9}−2.8e-7-2.8\text{e-7}
Table 15. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type II critical point with m=1m=1 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type IBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy ΔSd\Delta S_{d}L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
443.7653×10−23.7653\times 10^{-2}3.6323×10−23.6323\times 10^{-2}1.3301×10−21.3301\times 10^{-2}
10107.6637×10−27.6637\times 10^{-2}7.4862×10−27.4862\times 10^{-2}1.7751×10−31.7751\times 10^{-3}
1001001.4207×10−11.4207\times 10^{-1}1.4171×10−11.4171\times 10^{-1}3.6221×10−43.6221\times 10^{-4}
10310^{3}1.6853×10−11.6853\times 10^{-1}1.6849×10−11.6849\times 10^{-1}4.2197×10−54.2197\times 10^{-5}
Table 16. Computed values of the loss for type I critical point of isotropy ΔSd\Delta S_{d} . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased. This is always positive. Included are the values obtained when d=4d=4 . However, ξ(d){\boldsymbol{\xi}}(d) is not a spurious minima for d≤7d\leq 7 ; the same is likely true for ξ~(4){\widetilde{\boldsymbol{\xi}}}(4) .
Type IBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy ΔSd−1\Delta S_{d-1}L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10107.2057×10−27.2057\times 10^{-2}7.0390×10−27.0390\times 10^{-2}1.667×10−31.667\times 10^{-3}
1001001.4157×10−11.4157\times 10^{-1}1.4121×10−11.4121\times 10^{-1}3.621×10−43.621\times 10^{-4}
10310^{3}1.6847×10−11.6847\times 10^{-1}1.6843×10−11.6843\times 10^{-1}4.221×10−54.221\times 10^{-5}
10410^{4}1.77465×10−11.77465\times 10^{-1}1.77461×10−11.77461\times 10^{-1}4.4108×10−64.4108\times 10^{-6}
Table 17. Computed values of the loss for type I critical point of isotropy ΔSd−1\Delta S_{d-1} . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased. This is always positive. The table entries are close to the corresponding values for the type I, isotropy ΔSd\Delta S_{d} example of the preceding section.
Type IIBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy ΔSd−1\Delta S_{d-1}L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10101.8702×10−21.8702\times 10^{-2}1.7477×10−21.7477\times 10^{-2}1.226×10−31.226\times 10^{-3}
1001002.7417×10−32.7417\times 10^{-3}2.7123×10−32.7123\times 10^{-3}2.936×10−52.936\times 10^{-5}
10310^{3}2.9255×10−42.9255\times 10^{-4}2.9224×10−42.9224\times 10^{-4}3.159×10−73.159\times 10^{-7}
10410^{4}2.96126×10−52.96126\times 10^{-5}2.96095×10−52.96095\times 10^{-5}3.1280×10−93.1280\times 10^{-9}
Table 18. Computed values of the loss for type II critical point of isotropy ΔSd−1\Delta S_{d-1} . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased. This is always positive.
Type IBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2})L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10106.6993×10−26.6993\times 10^{-2}6.5457×10−26.5457\times 10^{-2}1.5356×10−31.5356\times 10^{-3}
1001001.4107×10−11.4107\times 10^{-1}1.4071×10−11.4071\times 10^{-1}3.6209×10−43.6209\times 10^{-4}
10310^{3}1.6841×10−11.6841\times 10^{-1}1.6837×10−11.6837\times 10^{-1}4.2223×10−54.2223\times 10^{-5}
10410^{4}1.77459×10−11.77459\times 10^{-1}1.77454‘×10−11.77454`\times 10^{-1}4.4110×10−64.4110\times 10^{-6}
Table 19. Computed values of the loss for type I critical point of isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2}) . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased.
Type IIBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2})L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10103.5327×10−23.5327\times 10^{-2}3.3175×10−23.3175\times 10^{-2}2.1518×10−32.1518\times 10^{-3}
1001005.4487×10−35.4487\times 10^{-3}5.3914×10−35.3914\times 10^{-3}5.7244×10−55.7244\times 10^{-5}
10310^{3}5.8473×10−45.8473\times 10^{-4}5.8410×10−15.8410\times 10^{-1}6.298×10−76.298\times 10^{-7}
10410^{4}5.9221×10−55.9221\times 10^{-5}5.9215×10−15.9215\times 10^{-1}6.2546×10−96.2546\times 10^{-9}
Table 20. Computed values of the loss for type II critical point of isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2}) . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased.
Type IIBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy Δ(Sd−3×S3)\Delta(S_{d-3}\times S_{3})L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10105.0240×10−25.0240\times 10^{-2}4.7284×10−24.7284\times 10^{-2}2.9556×10−32.9556\times 10^{-3}
1001008.1326×10−38.1326\times 10^{-3}8.0485×10−38.0485\times 10^{-3}8.4097×10−58.4097\times 10^{-5}
10310^{3}8.7666×10−48.7666\times 10^{-4}8.7572×10−48.7572\times 10^{-4}9.4278×10−79.4278\times 10^{-7}
10410^{4}8.8828×10−58.8828\times 10^{-5}8.8818×10−18.8818\times 10^{-1}9.3801×10−99.3801\times 10^{-9}
Table 21. Computed values of the loss for type II critical point of isotropy Δ(Sd−3×S3)\Delta(S_{d-3}\times S_{3}) . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased.
More in this topic

Shallow Neural Networks

Shallow neural networks use one or a few layers to learn functions, offering a setting to study efficient approximation, training dynamics, and limits of depth.

See all 51 papers

Explore similar work

CardsList
Apr 10, 2026

Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

We study the population loss landscape of two-layer ReLU networks of the form ∑k=1KReLU(wk⊤x)\sum_{k=1}^K \mathrm{ReLU}(w_k^\top x) in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical organisation of minima into discrete families and shows how overparameterisation changes their stability and reachability under gradient-based dynamics. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.

Jie Huang, Bruno Loureiro, Stefano Sarao Mannelli

Physics, Chalmers University of Technology and University of Gothenburg · Ecole Normale Superieure, PSL & CNRS · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg +1

OverparameterizationLoss LandscapeNeural Network OptimizationReLU Neural NetworksLoss Landscape GeometryShallow Neural Networks+1
AskPDF
Sep 30, 2026

Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus

Population loss can remain nearly constant while a neural network learns a substantially more predictive representation. We establish this separation for two-layer ReLU and leaky-ReLU networks trained on Gaussian inputs by simultaneous fixed-step population gradient descent on all parameters. For structured additive teachers whose links are positive mixtures of Gaussian-damped cubics in H1(γ)H^1(γ), we give explicit conditions under which small IID Gaussian initialization yields a high-probability guarantee: at a checkpoint during a high-loss plateau, minimum alignment between the rank-rr teacher subspace and the leading rr-dimensional eigenspace of the predictor's average gradient outer product (AGOP) increases by at least 1/21/2, and the minimum refit MSE under unchanged coefficient budgets decreases by more than 0.3990.399, both relative to initialization. The same trajectory subsequently attains a trained loss below every value in the plateau window. A complementary result treats unequal-weight cubic teachers and small additive Sobolev perturbations using projected-feature refits. For SwiGLU networks with an exactly fitted intercept, we prove leading-AGOP alignment during a loss plateau at fixed width and dimension as Gaussian initialization vanishes, for square-integrable teachers with nonzero Hermite content of degree one, two, or three. A rank-one cubic specialization also gives simultaneous unrestricted-refit gains at a prescribed width. An approximation lower bound further shows that certain interaction targets retain nonzero error when ridge neurons are restricted to shared orthogonal axes within the teacher subspace. Population-moment experiments with ReLU students across 21 teachers and 50 initializations per teacher complement the analysis.

Akash Kumar

Department of Computer Science & Engineering University of California-San Diego

Representation LearningReLU Neural NetworksNeural Network Training Dynamics
AskPDF
Oct 1, 2026

Removing spurious minima for planar features by skip connections

Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.

Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1

Freie Universität Berlin · Technische Universität Berlin

Shallow Neural NetworksReLU Neural NetworksNeural Network OptimizationNonconvex OptimizationTeacher-Student Learning
AskPDF