Singular parameters and missing limits in neural PDE solvers
Authors: Daniel Fernández
Organizations: Chair for Dynamics, Control, Machine Learning, and Numerics (Alexander von Humboldt Professorship), Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany.
Neural solvers for partial differential equations (PDEs) can approach an accurate solution while their parameters grow without bound. In such cases, the limiting solution may have no finite representation in the chosen model, leaving the best loss unattained. Our analysis connects missing limits in deep neural tanh- networks to unbounded hidden parameters or increasingly redundant neurons. For a class of models built from translated kernels, we describe the missing functions and recover them by adding kernel derivatives to the model. This completion makes the best approximation attainable under standard assumptions. Numerical studies follow the associated parameter growth and explore how completion affects PDE optimization.
Figures & tables
Figure 1 . Two merging tanh neurons approach a function outside the model. (a) Weights grow with opposite signs; (b) large contributions nearly cancel; (c) their sum approaches u∗ .
Figure 2 . Network outputs can converge while parameters escape or neurons degenerate. The limiting state lies outside the model. Schematic.
Figure 3 . As three Gaussian centres merge, weights grow and the fit improves. A derivative block represents the target exactly. Top: individual contributions; bottom: their sum and the target.
Figure 4 . Two tanh neurons merge during training as their weights grow with opposite signs. Circles mark starts; triangles mark endpoints. (a) Single precision; (b) double precision.
Figure 5 . Large neuron contributions nearly cancel, bringing their sum closer to the target. Top: contributions; bottom: sum (dashed) and target (green). The last two columns show independent refinements.
Figure 6 . Larger coefficient budgets improve the Gaussian approximation. (a) States; (b) errors; (c) rescaled excess energies approach their predicted limits. Dotted line: completed error for ε=0.01 .
Figure 7 . Completion sharply reduces the Poisson error in a representative run. Top: target and final approximations; bottom: error maps. Larger circles indicate larger coefficients.
Standard optimization
Completion
Relative H1 error
9.8×10−6–6.2×10−5
1.7×10−8–2.6×10−7
median
1.8×10−5
8.1×10−8
Coefficient norm
2.3×102–5.0×103
3.53694–3.53698
Smallest center half-gap
2.1×10−4–3.1×10−3
0
Solves (median)
758–1785 (1204.5)
377–525 (431)
Completed pairs
0
4
Table 1 . Final accuracy, coefficient sizes, and work for the Poisson–Neumann problem over ten shared starts.
Figure 8 . Completion improves Poisson accuracy and controls coefficient growth across ten starts. Left: best relative error so far; right: coefficient norm. Crosses mark abnormal termination.
Standard optimization
Completion
Relative H1 error
2.1×10−6–3.5×10−5
3.4×10−8–1.8×10−5
median
1.7×10−5
2.3×10−7
Relative eQ
4.8×10−6–7.4×10−5
2.5×10−8–3.4×10−5
Coefficient norm
7.0×101–1.0×103
2.5×100–1.0×102
Smallest center half-gap
7.5×10−4–1.3×10−2
0
Fits (median)
149–681 (225.5)
108–713 (255.5)
Table 2 . Final accuracy, coefficient sizes, and work for the Matérn problem over ten shared starts.
Figure 9 . Completion improves Matérn accuracy and reduces coefficient growth. Left: best relative error so far; right: coefficient norm. Crosses mark abnormal termination.
Standard optimization
Completion
Problem
τ
Reached
Median
Reached
Median
Poisson
1.0×10−3
10/10
51.5
10/10
69
1.0×10−5
1/10
1283
10/10
291.5
1.0×10−7
0/10
—
7/10
427
Matérn
1.0×10−3
10/10
20
10/10
20
1.0×10−5
3/10
169
9/10
180
Table 3 . Work needed to reach each relative H1 error threshold. Counts are out of ten starts; medians include only starts that reach the threshold.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Standard optimization
Completion
Problem
Order
Median H1
≤10−7
Median H1
≤10−7
Poisson
32
1.8×10−5
0/10
8.1×10−8
7/10
64
1.9×10−5
0/10
7.9×10−8
6/10
Matérn
64
1.7×10−5
0/10
2.3×10−7
5/10
96
1.8×10−5
0/10
1.5×10−8
6/10
Appendix
Table 4. Completion retains lower median errors with finer training quadrature.
National Center for Applied Mathematics Tianjin University Tianjin, 300072, China · School of Mechanical and Aerospace Engineering Jilin University Changchun, 130025, China