We establish empirical scaling laws for Single-Layer Physics-Informed Neural Networks on canonical nonlinear PDEs. We identify a dual optimization failure: (i) a baseline pathology, where the solution error fails to decrease with network width, even at fixed nonlinearity, falling short of theoretical approximation bounds, and (ii) a compounding pathology, where this failure is exacerbated by nonlinearity. We provide quantitative evidence that a simple separable power law is insufficient, and that the scaling behavior is governed by a more complex, non-separable relationship. This failure is consistent with the concept of spectral bias, where networks struggle to learn the high-frequency solution components that intensify with nonlinearity. We show that optimization, not approximation capacity, is the primary bottleneck, and propose a methodology to empirically measure these complex scaling effects.
Figures & tables
Figure 1: Error vs. Network Width ( N ) for the Poisson PDE. Tanh networks find low-error solutions but exhibit high variance and no clear scaling ( α≈0.06±0.4 ). ReLU networks fail to learn ( α≈0.01±0.01 ). The gray and red lines give the theoretical error decay rates of O(N−1/2) and O(N−1) respectively. It should be noted that the confidence intervals of the tanh error often intersect with theoretical estimates, but no consistent trend is observed.
PDE
Activation
Width Exp. ( α±95% CI)
Hardness Exp. ( γ±95% CI)
log(A)
Adj. R2
KdV
ReLU
−0.05 0.03 **
0.17 0.04 ***
−0.66
0.65
Tanh
0.00 0.01
0.18 0.01 ***
−0.27
0.51
Sine-Gordon
ReLU
−0.32 0.07 ***
0.28 0.08 ***
−3.40
0.70
Tanh
−0.14 0.34
1.51 0.39 ***
−7.30
0.52
Allen-Cahn
ReLU
−0.37 0.08 ***
−0.44 0.10 ***
−3.36
0.74
Tanh
−0.02 0.11
−0.03 0.12
−6.78
−0.03
Table 1: Overall scaling exponents from the separable power-law regression error≈A⋅N−α⋅κγ . Here, α represents the average width scaling, while γ quantifies the error’s sensitivity to the hardness parameter κ . Statistical significance based on the p-value of the fitted coefficient is indicated by: p<0.01 (), p<0.001 (*).
Figure 2: Scaling law analysis for the Sine-Gordon equation. (a) Width scaling exponent α vs. hardness κ . Often α<0 , implying increasing network width also increases error. (b) Final error vs. hardness κ for different network widths N . The final error degrades significantly at a certain inflection point of hardness, indicating some kind of regime shift. (c) A representative example of error vs. width N for the median hardness value κ=2.0 demonstrates that increased width fails to reduce error. Error bars denote standard error over 5 seeds.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Scaling law analysis for the Allen-Cahn equation. (a) Width scaling exponent α vs. hardness κ=1/D . For both activations, α is consistently negative, indicating wider networks perform worse. (b) Final error vs. hardness κ . Tanh networks (dashed lines) achieve low error that is remarkably stable against increasing hardness, while ReLU networks (dotted lines) perform orders of magnitude worse. (c) A representative example of error vs. width N for the median hardness value κ=32.0 , clearly showing the negative scaling trend ( α<0 ). Error bars denote standard error over 5 seeds.
Figure 4: Scaling law analysis for the Korteweg-de Vries (KdV) equation. (a) Width scaling exponent α vs. hardness κ=A . For both ReLU and tanh , α is consistently near or below zero, indicating no performance gain from increasing network width. (b) Final error vs. hardness κ . Error increases with hardness for both activations, though tanh generally performs better. (c) A representative example of error vs. width N for κ=2.0 , visualizing the flat ( α≈0 ) or negative scaling. Error bars denote standard error over 5 seeds.
PDE
Hardness ( κ )
ReLU ( α±95% CI)
Tanh ( α±95% CI)
Poisson
N/A
0.01 0.02
0.06 0.43
KdV
0.25
−0.15 0.06
0.00 0.01
0.50
−0.13 0.07
0.00 0.01
1.00
−0.08 0.11
0.00 0.01
2.00
−0.02 0.05
0.00 0.01
4.00
0.00 0.01
0.01 0.01
Appendix
Table 2: Fitted width scaling exponents α for each hardness parameter κ and activation function. The exponent is derived from fitting the mean error over 5 seeds to the power law error≈A⋅N−α .
PDE
Activation
β0 (Intercept)
βN (Width)
βκ (Hardness)
ν (Interaction)
Adj. R2
KdV
ReLU
−0.809 ***
0.083 ***
0.386 ***
-0.042 ***
0.741
Tanh
−0.271 **
−0.001
0.178 **
0.000
0.498
Sine-Gordon
ReLU
−3.716 ***
0.382 ***
0.740 ***
-0.089 ***
0.758
Tanh
−8.094 ***
0.296
2.666 ***
−0.222
0.542
Allen-Cahn
ReLU
−1.202 **
−0.048
−1.061 ***
0.120 ***
0.811
Tanh
−7.800 ***
0.212
0.260
−0.057
−0.008
Appendix
Table 3: Scaling exponents for the interaction model defined in ( 2 ). Statistical significance based on the p-value of the fitted coefficient is indicated by: p<0.01 (), p<0.001 (*).
Physics-Informed Neural Networks (PINNs) are a common class of machine learning-based partial differential equation (PDE) solvers which train a network to represent a solution by minimizing a residual loss that encodes the PDE. Despite their successes, they are known to fail on certain simple equations, converging to an incorrect solution despite low loss. These failure modes have garnered significant attention in the literature over the past several years, motivating both architectural and optimization based solutions. By directly visualizing the residual, we show that failure modes are the result of overfitting: the loss is minimized on the collocation points, but not elsewhere. Applying regularization causes the failure modes to vanish. Finally, we extend double backpropagation over the full set of residuals, and use it to achieve state-of-the-art performance on four standard failure mode equations with up to 23× fewer collocation points and a vanilla architecture.
Nigel T. Andersen, Takashi Matsubara
Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan
Physics-informed neural networks (PINNs) often fail to accurately resolve partial differential equations (PDEs) with high-frequency or multi-scale solutions, as well as strongly nonlinear problems. Two factors underlie this difficulty: spectral bias, the tendency of neural networks to underfit high-frequency features; and representation-coefficient coupling, the entanglement of representation learning and coefficient fitting within a single nonconvex optimization objective. In this work, we propose the Fourier-enhanced alternating Levenberg--Marquardt PINN (FALM-PINN), an optimization framework that decouples representation learning from coefficient fitting. The upper-level problem learns a Fourier-enhanced basis that enriches the latent space with high-frequency components, while the lower-level problem resolves the coupling by fitting the projection coefficients on this basis, solving a nonlinear least-squares problem with the Levenberg--Marquardt algorithm. The framework applies to general nonlinear and coupled PDE systems, and reduces to a single-step convex optimization problem for linear PDEs. We prove global convergence of the alternating training scheme in both cases. Numerical examples on multiple challenging high-frequency and nonlinear PDEs show that FALM-PINN achieves relative L2 errors up to two orders of magnitude lower than state-of-the-art baselines.
Yulun Wu, Matthieu Barreau, Miguel Aguiar +1
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, 100 44, Sweden
Physics-informed neural networks (PINNs) often rely on over-parameterized models to optimize coupled solution and differential-residual objectives, leaving unclear how much capacity is necessary and what pruning should preserve. We study foresight pruning at initialization for sparse PirateNet PDE solvers. Standard neural tangent kernel spectrum-aware pruning (NTK-SAP) aims to preserve output-side training dynamics but may overlook parameters whose main influence arises through derivatives in the governing equations. We introduce physics-informed spectrum-aware pruning (PI-SAP), which assigns saliency using sensitivity of the PDE residual. Experiments on the Gray-Scott equations, complex Ginzburg-Landau equation, Burgers' equation, and linear convection equation show that PI-SAP more consistently preserves Gray-Scott residual fidelity and is competitive under aggressive sparsity. However, no criterion is uniformly optimal across equations or sparsity levels. Small-batch PINN-NTK diagnostics further show that residual fidelity, solution accuracy, and kernel conditioning are distinct objectives, motivating pruning methods that explicitly balance solution-side and residual-side training dynamics during optimization.
Ahmad Ishaque Karimi, Uvini Balasuriya Mudiyanselage, Kookjin Lee
School of Computing and Augmented Intelligence Arizona State University Tempe, USA