cs.LGSep 25, 2026

Population loss in shallow ReLU networks: Bias & families of critical points

Authors: Michael Field

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The main result presented is a formula for the population loss in the student-teacher kernel model that is applicable to shallow ReLU networks with bias. This extends previous work of Choo and Saul (2009) and Brutzkus and Globerson (2017). The formula makes essential use of Owen's T-function. The necessary theory of the T-function is given and a high precision coding using MPFR for the T-function, based on an algorithm of Komelj (2023), is available on request. It is shown that various families of spurious minima described in past papers of Arjevani and the author extend to biased networks and that the loss is always strictly decreased when bias is added. The change in landscape geometry caused by adding bias appears to be relatively mild. Only the simplest examples are described in this paper where it is assumed that the number of inputs is equal to the number of neurons (this restriction is for reasons of length). A review of relevant previous results on unbiased networks is included. Aside from Gaussian statistics, the main mathematical tools and ideas come from analytic geometry (analytic and subanalytic sets, the Curve Selection Lemma).

Figures & tables

Figure 1
Figure 1. The region of integration in the x1,x2x_{1},x_{2} -plane with boundary shown in red and bˉ<0<βˉ\bar{b}<0<\bar{\beta} and θ∈(0,π/2]\theta\in(0,\pi/2] . As θ\theta increases, the line LL defined by x1cos⁡(θ)+x2sin⁡(θ)=βˉx_{1}\cos(\theta)+x_{2}\sin(\theta)=\bar{\beta} rotates counter clockwise so that when θ∈(π/2,π)\theta\in(\pi/2,\pi) , LL intersects the negative x1x_{1} -axis.
Figure 2. The region of integration on the strip [γ‾,β‾]×R[\overline{\gamma},\overline{\beta}]\times{\mathbb{R}} , θ\theta near zero and β‾<γ‾\overline{\beta}<\overline{\gamma} .
d=kd=kValue of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0})
660.71830101072557148145×6−20.71830101072557148145\times 6^{-2}
10100.33399917189077800002×10−20.33399917189077800002\times 10^{-2}
20200.10183336077334108245×10−20.10183336077334108245\times 10^{-2}
1001000.49862816276459333028×10−40.49862816276459333028\times 10^{-4}
100010000.53292271608433825762×10−60.53292271608433825762\times 10^{-6}
10610^{6}0.54139203214105478639×10−120.54139203214105478639\times 10^{-12}
Table 1. Numerically computed values of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0}) for a type II critical point with k=dk=d and isotropy ΔSd−1\Delta S_{d-1}
d=kd=kValue of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0})
10100.77142020703795495629×10−20.77142020703795495629\times 10^{-2}
1001000.1220227002721111972×10−20.1220227002721111972\times 10^{-2}
100010000.13745083160523485590×10−30.13745083160523485590\times 10^{-3}
10610^{6}0.14472588806870954343×10−60.14472588806870954343\times 10^{-6}
101210^{12}0.14496749467704066784×10−120.14496749467704066784\times 10^{-12}
Table 2. Numerically computed values of ∂L∂β‾0(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}(\mathfrak{c}(d),\mathbf{0}) for a type I critical point with k=dk=d and isotropy ΔSd\Delta S_{d}
Type I  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy ΔSd−1\Delta S_{d-1}  
10100.7468×10−20.7468\times 10^{-2}0.7283×10−20.7283\times 10^{-2}
100010000.1374×10−30.1374\times 10^{-3}0.1769×10−30.1769\times 10^{-3}
10610^{6}0.1473×10−60.1473\times 10^{-6}0.1969×10−60.1969\times 10^{-6}
Table 3. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type I critical point with m=0m=0 and isotropy ΔSd−1\Delta S_{d-1} .
Type I   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy ΔSd−1\Delta S_{d-1}   
10100.3532×10−20.3532\times 10^{-2}0.4842×10−20.4842\times 10^{-2}0.1070×10−10.1070\times 10^{-1}
100010000.1318×10−30.1318\times 10^{-3}0.4658×10−30.4658\times 10^{-3}0.4885×10−30.4885\times 10^{-3}
10610^{6}0.1446×10−60.1446\times 10^{-6}0.2495×10−50.2495\times 10^{-5}0.2513×10−50.2513\times 10^{-5}
Table 4. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type I critical point with m=1m=1 and isotropy ΔSd−1\Delta S_{d-1} .
Type I    
k=d+2k=d+2∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}∂L/∂β‾3\partial\mathcal{L}/\partial\overline{\beta}_{3}
Isotropy ΔSd−1\Delta S_{d-1}    
10100.2760×10−20.2760\times 10^{-2}0.4500×10−20.4500\times 10^{-2}0.4969×10−10.4969\times 10^{-1}0.7548×10−20.7548\times 10^{-2}
100010000.1295×10−30.1295\times 10^{-3}0.4396×10−30.4396\times 10^{-3}0.4006×10−30.4006\times 10^{-3}0.1937×10−30.1937\times 10^{-3}
10610^{6}0.1449×10−60.1449\times 10^{-6}0.2483×10−50.2483\times 10^{-5}0.2420×10−50.2420\times 10^{-5}0.1850i×10−60.1850i\times 10^{-6}
Table 5. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈4i\in{\mathbf{4}} , for a type I critical point with m=2m=2 and isotropy ΔSd−1\Delta S_{d-1} .
Type II  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy ΔSd−1\Delta S_{d-1}  
10100.3340×10−20.3340\times 10^{-2}−0.7436×10−2-0.7436\times 10^{-2}
100010000.5329×10−60.5329\times 10^{-6}−0.2304×10−3-0.2304\times 10^{-3}
10610^{6}0.5414.×10−120.5414.\times 10^{-12}−0.2371×10−6-0.2371\times 10^{-6}
Table 6. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type II critical point with m=0m=0 and isotropy ΔSd−1\Delta S_{d-1} .
Type II   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy ΔSd−1\Delta S_{d-1}   
10100.2605×10−20.2605\times 10^{-2}−0.6054×10−2-0.6054\times 10^{-2}−0.2055×10−2-0.2055\times 10^{-2}
10001000−0.1011×10−5-0.1011\times 10^{-5}−0.2062×10−3-0.2062\times 10^{-3}−0.7271×10−5-0.7271\times 10^{-5}
10610^{6}−0.5076×10−10-0.5076\times 10^{-10}−0.2131×10−6-0.2131\times 10^{-6}−0.2369×10−9-0.2369\times 10^{-9}
Table 7. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type I critical point with m=1m=1 and isotropy ΔSd−1\Delta S_{d-1} .
Type II    
k=d+2k=d+2∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}∂L/∂β‾3\partial\mathcal{L}/\partial\overline{\beta}_{3}
Isotropy ΔSd−1\Delta S_{d-1}    
10100.2590×10−20.2590\times 10^{-2}- 0.5638×10−20.5638\times 10^{-2}0.1317×10−20.1317\times 10^{-2}0.1147×10−20.1147\times 10^{-2}
10001000−1.018×10−6-1.018\times 10^{-6}−0.2052×10−3-0.2052\times 10^{-3}−5.682×10−6-5.682\times 10^{-6}−2.536×10−6-2.536\times 10^{-6}
10610^{6}−5.099×10−11-5.099\times 10^{-11}−0.2130×10−6-0.2130\times 10^{-6}−1.86×10−10-1.86\times 10^{-10}−8.051×10−11-8.051\times 10^{-11}
Table 8. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈4i\in{\mathbf{4}} , for a type II critical point with m=2m=2 and isotropy ΔSd−1\Delta S_{d-1} .
Type I  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}  
10100.7160×10−20.7160\times 10^{-2}0.6965×10−20.6965\times 10^{-2}
100010000.1374×10−30.1374\times 10^{-3}0.1769×10−30.1769\times 10^{-3}
10610^{6}0.1447×10−60.1447\times 10^{-6}0.1969×10−60.1969\times 10^{-6}
Table 9. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type I critical point with m=0m=0 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type I   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}   
10100.2840×10−20.2840\times 10^{-2}0.4030×10−20.4030\times 10^{-2}1.087×10−21.087\times 10^{-2}
100010000.1304×10−30.1304\times 10^{-3}0.2585×10−30.2585\times 10^{-3}0.4299×10−30.4299\times 10^{-3}
10610^{6}0.1445×10−100.1445\times 10^{-10}0.3027×10−60.3027\times 10^{-6}0.4764×10−90.4764\times 10^{-9}
Table 10. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type I critical point with m=1m=1 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type I    
k=d+2k=d+2∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}∂L/∂β‾3\partial\mathcal{L}/\partial\overline{\beta}_{3}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}    
10100.2186×10−20.2186\times 10^{-2}0.3591×10−20.3591\times 10^{-2}0.5519×10−20.5519\times 10^{-2}0.6705×10−20.6705\times 10^{-2}
100010000.1298×10−30.1298\times 10^{-3}0.2544×10−30.2544\times 10^{-3}0.3176×10−30.3176\times 10^{-3}0.1431×10−30.1431\times 10^{-3}
10610^{6}0.1445×10−60.1445\times 10^{-6}0.2996×10−60.2996\times 10^{-6}0.3636×10−60.3636\times 10^{-6}0.1346×10−60.1346\times 10^{-6}
Table 11. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈4i\in{\mathbf{4}} , for a type I critical point with m=2m=2 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type II  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}  
10100.6278×10−20.6278\times 10^{-2}−0.4145×10−2-0.4145\times 10^{-2}
100010001.065×10−61.065\times 10^{-6}−0.2296×10−3-0.2296\times 10^{-3}
10610^{6}1.083×10−61.083\times 10^{-6}−2.371×10−3-2.371\times 10^{-3}
Table 12. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for a type II critical point with m=0m=0 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type II   
k=d+1k=d+1∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}∂L/∂β‾2\partial\mathcal{L}/\partial\overline{\beta}_{2}
Isotropy Gd−2,2ΔG^{\Delta}_{d-2,2}   
10100.4799×10−20.4799\times 10^{-2}−0.3701×10−2-0.3701\times 10^{-2}−2.594×10−2-2.594\times 10^{-2}
10001000−1.188×10−6-1.188\times 10^{-6}−0.2042×10−3-0.2042\times 10^{-3}−1.242×10−5-1.242\times 10^{-5}
10610^{6}−7.244×10−11-7.244\times 10^{-11}−0.2092×10−6-0.2092\times 10^{-6}−4.060×10−10-4.060\times 10^{-10}
Table 13. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type II critical point with m=1m=1 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type II  
d=kd=k∂L/∂β‾0\partial\mathcal{L}/\partial\overline{\beta}_{0}∂L/∂β‾1\partial\mathcal{L}/\partial\overline{\beta}_{1}
Isotropy Gd−p,pΔG^{\Delta}_{d-p,p}  
p=3p=3  
d=10d=100.8835×10−20.8835\times 10^{-2}−0.1100×10−2-0.1100\times 10^{-2}
d=106d=10^{6}1.624×10−121.624\times 10^{-12}−0.2372×10−6-0.2372\times 10^{-6}
Initial Coeff. of FPSD40=1.625D^{0}_{4}=1.625D20=−2.373D^{0}_{2}=-2.373
Table 14. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈2i\in{\mathbf{2}} , for type II critical points with p=3,4p=3,4 and d−p=pd-p=p If d−p=pd-p=p , the diagonal entries of the upper diagonal block converge to +1+1 as d→∞d{\rightarrow}\infty .
Type II, m=0       
 ∂L∂β‾0\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{0}}∂L∂β‾1\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{1}}∂L∂β‾2\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{2}}∂L∂β‾3\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{3}}∂L∂β‾4\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{4}}∂L∂β‾5\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{5}}∂L∂β‾6\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{6}}
Isotropy ΔSd−6\Delta S_{d-6}       
d=10d=103.8e-33.8\text{e-3}3.4e-33.4\text{e-3}3.2e-33.2\text{e-3}4.0e-34.0\text{e-3}2.9e-3e-32.9\text{e-3}\text{e-3}4.4e-34.4\text{e-3}−8.4e-3-8.4\text{e-3}
d=1000d=10006.1e-76.1\text{e-7}−1.6e-6-1.6\text{e-6}−8.0e-6-8.0\text{e-6}1.0e-51.0\text{e-5}−1.4e-5-1.4\text{e-5}9.0e-69.0\text{e-6}−2.8e-4-2.8\text{e-4}
d=106d=10^{6}6.2e-136.2\text{e-13}−2.2e-9-2.2\text{e-9}−8.7e-9-8.7\text{e-9}9.7e-99.7\text{e-9}−1.5e-8-1.5\text{e-8}8.5e-98.5\text{e-9}−2.8e-7-2.8\text{e-7}
Table 15. Computed values of ∂L∂β‾i(c(d),0)\frac{\partial\mathcal{L}}{\partial\overline{\beta}_{i}}(\mathfrak{c}(d),\mathbf{0}) , i∈3i\in{\mathbf{3}} , for a type II critical point with m=1m=1 and isotropy Gd−2,2ΔG^{\Delta}_{d-2,2} .
Type IBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy ΔSd\Delta S_{d}L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
443.7653×10−23.7653\times 10^{-2}3.6323×10−23.6323\times 10^{-2}1.3301×10−21.3301\times 10^{-2}
10107.6637×10−27.6637\times 10^{-2}7.4862×10−27.4862\times 10^{-2}1.7751×10−31.7751\times 10^{-3}
1001001.4207×10−11.4207\times 10^{-1}1.4171×10−11.4171\times 10^{-1}3.6221×10−43.6221\times 10^{-4}
10310^{3}1.6853×10−11.6853\times 10^{-1}1.6849×10−11.6849\times 10^{-1}4.2197×10−54.2197\times 10^{-5}
Table 16. Computed values of the loss for type I critical point of isotropy ΔSd\Delta S_{d} . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased. This is always positive. Included are the values obtained when d=4d=4 . However, ξ(d){\boldsymbol{\xi}}(d) is not a spurious minima for d≤7d\leq 7 ; the same is likely true for ξ~(4){\widetilde{\boldsymbol{\xi}}}(4) .
Type IBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy ΔSd−1\Delta S_{d-1}L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10107.2057×10−27.2057\times 10^{-2}7.0390×10−27.0390\times 10^{-2}1.667×10−31.667\times 10^{-3}
1001001.4157×10−11.4157\times 10^{-1}1.4121×10−11.4121\times 10^{-1}3.621×10−43.621\times 10^{-4}
10310^{3}1.6847×10−11.6847\times 10^{-1}1.6843×10−11.6843\times 10^{-1}4.221×10−54.221\times 10^{-5}
10410^{4}1.77465×10−11.77465\times 10^{-1}1.77461×10−11.77461\times 10^{-1}4.4108×10−64.4108\times 10^{-6}
Table 17. Computed values of the loss for type I critical point of isotropy ΔSd−1\Delta S_{d-1} . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased. This is always positive. The table entries are close to the corresponding values for the type I, isotropy ΔSd\Delta S_{d} example of the preceding section.
Type IIBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy ΔSd−1\Delta S_{d-1}L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10101.8702×10−21.8702\times 10^{-2}1.7477×10−21.7477\times 10^{-2}1.226×10−31.226\times 10^{-3}
1001002.7417×10−32.7417\times 10^{-3}2.7123×10−32.7123\times 10^{-3}2.936×10−52.936\times 10^{-5}
10310^{3}2.9255×10−42.9255\times 10^{-4}2.9224×10−42.9224\times 10^{-4}3.159×10−73.159\times 10^{-7}
10410^{4}2.96126×10−52.96126\times 10^{-5}2.96095×10−52.96095\times 10^{-5}3.1280×10−93.1280\times 10^{-9}
Table 18. Computed values of the loss for type II critical point of isotropy ΔSd−1\Delta S_{d-1} . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased. This is always positive.
Type IBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2})L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10106.6993×10−26.6993\times 10^{-2}6.5457×10−26.5457\times 10^{-2}1.5356×10−31.5356\times 10^{-3}
1001001.4107×10−11.4107\times 10^{-1}1.4071×10−11.4071\times 10^{-1}3.6209×10−43.6209\times 10^{-4}
10310^{3}1.6841×10−11.6841\times 10^{-1}1.6837×10−11.6837\times 10^{-1}4.2223×10−54.2223\times 10^{-5}
10410^{4}1.77459×10−11.77459\times 10^{-1}1.77454‘×10−11.77454`\times 10^{-1}4.4110×10−64.4110\times 10^{-6}
Table 19. Computed values of the loss for type I critical point of isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2}) . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased.
Type IIBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2})L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10103.5327×10−23.5327\times 10^{-2}3.3175×10−23.3175\times 10^{-2}2.1518×10−32.1518\times 10^{-3}
1001005.4487×10−35.4487\times 10^{-3}5.3914×10−35.3914\times 10^{-3}5.7244×10−55.7244\times 10^{-5}
10310^{3}5.8473×10−45.8473\times 10^{-4}5.8410×10−15.8410\times 10^{-1}6.298×10−76.298\times 10^{-7}
10410^{4}5.9221×10−55.9221\times 10^{-5}5.9215×10−15.9215\times 10^{-1}6.2546×10−96.2546\times 10^{-9}
Table 20. Computed values of the loss for type II critical point of isotropy Δ(Sd−2×S2)\Delta(S_{d-2}\times S_{2}) . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased.
Type IIBias free networkBiased networkLoss decrease
d=kd=k  with bias
Isotropy Δ(Sd−3×S3)\Delta(S_{d-3}\times S_{3})L(ξ(d))\mathcal{L}({\boldsymbol{\xi}}(d))L(ξ~(d))\mathcal{L}({\widetilde{\boldsymbol{\xi}}}(d))Δub(d)\Delta_{ub}(d)
10105.0240×10−25.0240\times 10^{-2}4.7284×10−24.7284\times 10^{-2}2.9556×10−32.9556\times 10^{-3}
1001008.1326×10−38.1326\times 10^{-3}8.0485×10−38.0485\times 10^{-3}8.4097×10−58.4097\times 10^{-5}
10310^{3}8.7666×10−48.7666\times 10^{-4}8.7572×10−48.7572\times 10^{-4}9.4278×10−79.4278\times 10^{-7}
10410^{4}8.8828×10−58.8828\times 10^{-5}8.8818×10−18.8818\times 10^{-1}9.3801×10−99.3801\times 10^{-9}
Table 21. Computed values of the loss for type II critical point of isotropy Δ(Sd−3×S3)\Delta(S_{d-3}\times S_{3}) . Loss is given to 5 or more significant figures and shows how the addition of bias to the student network always reduces the loss. The third column gives the decrease of the loss when the network is biased.
More in this topic

Shallow Neural Networks

Shallow neural networks use one or a few layers to learn functions, offering a setting to study efficient approximation, training dynamics, and limits of depth.

See all 51 papers

Explore similar work

CardsList

  1. Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

    Apr 10, 2026Jie Huang, Bruno Loureiro, Stefano Sarao MannelliOverparameterizationLoss Landscape

    Physics, Chalmers University of Technology and University of Gothenburg · Ecole Normale Superieure, PSL & CNRS · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg +1

  2. Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus

    Sep 30, 2026Akash KumarRepresentation LearningReLU Neural Networks

    Department of Computer Science & Engineering University of California-San Diego

  3. Removing spurious minima for planar features by skip connections

    Oct 1, 2026Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1Shallow Neural NetworksReLU Neural Networks

    Freie Universität Berlin · Technische Universität Berlin