Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity

Faris Chaudhry

cs.LGMar 13, 2026

Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity

Authors: Faris Chaudhry

Organizations: Department of Computer Science, Imperial College London

Abstract

We establish empirical scaling laws for Single-Layer Physics-Informed Neural Networks on canonical nonlinear PDEs. We identify a dual optimization failure: (i) a baseline pathology, where the solution error fails to decrease with network width, even at fixed nonlinearity, falling short of theoretical approximation bounds, and (ii) a compounding pathology, where this failure is exacerbated by nonlinearity. We provide quantitative evidence that a simple separable power law is insufficient, and that the scaling behavior is governed by a more complex, non-separable relationship. This failure is consistent with the concept of spectral bias, where networks struggle to learn the high-frequency solution components that intensify with nonlinearity. We show that optimization, not approximation capacity, is the primary bottleneck, and propose a methodology to empirically measure these complex scaling effects.

Figures & tables

Figure 1: Error vs. Network Width ( $N$ ) for the Poisson PDE. Tanh networks find low-error solutions but exhibit high variance and no clear scaling ( $\alpha\approx 0.06\pm 0.4$ ). ReLU networks fail to learn ( $\alpha\approx 0.01\pm 0.01$ ). The gray and red lines give the theoretical error decay rates of $\mathcal{O}(N^{-1/2})$ and $\mathcal{O}(N^{-1})$ respectively. It should be noted that the confidence intervals of the tanh error often intersect with theoretical estimates, but no consistent trend is observed.

PDE	Activation	Width Exp. ( $\alpha\pm 95\%$ CI)	Hardness Exp. ( $\gamma\pm 95\%$ CI)	log(A)	Adj. $R^{2}$
KdV	ReLU	$-0.05$ 0.03 **	$0.17$ 0.04 ***	$-0.66$	$0.65$
KdV	Tanh	$0.00$ 0.01	$0.18$ 0.01 ***	$-0.27$	$0.51$
Sine-Gordon	ReLU	$-0.32$ 0.07 ***	$0.28$ 0.08 ***	$-3.40$	$0.70$
Sine-Gordon	Tanh	$-0.14$ 0.34	$1.51$ 0.39 ***	$-7.30$	$0.52$
Allen-Cahn	ReLU	$-0.37$ 0.08 ***	$-0.44$ 0.10 ***	$-3.36$	$0.74$
Allen-Cahn	Tanh	$-0.02$ 0.11	$-0.03$ 0.12	$-6.78$	$-0.03$

Table 1: Overall scaling exponents from the separable power-law regression

\text{error}\approx A\cdot N^{-\alpha}\cdot\kappa^{\gamma}

. Here,

\alpha

represents the average width scaling, while

\gamma

quantifies the error’s sensitivity to the hardness parameter

\kappa

. Statistical significance based on the p-value of the fitted coefficient is indicated by:

p<0.01

(), $p<0.001$ (*).

Figure 2: Scaling law analysis for the Sine-Gordon equation. (a) Width scaling exponent $\alpha$ vs. hardness $\kappa$ . Often $\alpha<0$ , implying increasing network width also increases error. (b) Final error vs. hardness $\kappa$ for different network widths $N$ . The final error degrades significantly at a certain inflection point of hardness, indicating some kind of regime shift. (c) A representative example of error vs. width $N$ for the median hardness value $\kappa=2.0$ demonstrates that increased width fails to reduce error. Error bars denote standard error over 5 seeds.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Scaling law analysis for the Allen-Cahn equation. (a) Width scaling exponent $\alpha$ vs. hardness $\kappa=1/D$ . For both activations, $\alpha$ is consistently negative, indicating wider networks perform worse. (b) Final error vs. hardness $\kappa$ . Tanh networks (dashed lines) achieve low error that is remarkably stable against increasing hardness, while ReLU networks (dotted lines) perform orders of magnitude worse. (c) A representative example of error vs. width $N$ for the median hardness value $\kappa=32.0$ , clearly showing the negative scaling trend ( $\alpha<0$ ). Error bars denote standard error over 5 seeds.

Figure 4: Scaling law analysis for the Korteweg-de Vries (KdV) equation. (a) Width scaling exponent $\alpha$ vs. hardness $\kappa=A$ . For both ReLU and tanh , $\alpha$ is consistently near or below zero, indicating no performance gain from increasing network width. (b) Final error vs. hardness $\kappa$ . Error increases with hardness for both activations, though tanh generally performs better. (c) A representative example of error vs. width $N$ for $\kappa=2.0$ , visualizing the flat ( $\alpha\approx 0$ ) or negative scaling. Error bars denote standard error over 5 seeds.

PDE	Hardness ( $\kappa$ )	ReLU ( $\alpha\pm 95\%$ CI)	Tanh ( $\alpha\pm 95\%$ CI)
Poisson	N/A	$0.01$ 0.02	$0.06$ 0.43
KdV	$0.25$	$-0.15$ 0.06	$0.00$ 0.01
	$0.50$	$-0.13$ 0.07	$0.00$ 0.01
	$1.00$	$-0.08$ 0.11	$0.00$ 0.01
	$2.00$	$-0.02$ 0.05	$0.00$ 0.01
	$4.00$	$0.00$ 0.01	$0.01$ 0.01

Appendix

Table 2: Fitted width scaling exponents

\alpha

for each hardness parameter

\kappa

and activation function. The exponent is derived from fitting the mean error over 5 seeds to the power law

\text{error}\approx A\cdot N^{-\alpha}

PDE	Activation	$\beta_{0}$ (Intercept)	$\beta_{N}$ (Width)	$\beta_{\kappa}$ (Hardness)	$\nu$ (Interaction)	Adj. $R^{2}$
KdV	ReLU	$-0.809$ ***	$0.083$ ***	$0.386$ ***	-0.042 ***	0.741
KdV	Tanh	$-0.271$ **	$-0.001$	$0.178$ **	$0.000$	$0.498$
Sine-Gordon	ReLU	$-3.716$ ***	$0.382$ ***	$0.740$ ***	-0.089 ***	0.758
Sine-Gordon	Tanh	$-8.094$ ***	$0.296$	$2.666$ ***	$-0.222$	$0.542$
Allen-Cahn	ReLU	$-1.202$ **	$-0.048$	$-1.061$ ***	0.120 ***	0.811
Allen-Cahn	Tanh	$-7.800$ ***	$0.212$	$0.260$	$-0.057$	$-0.008$

Appendix

Table 3: Scaling exponents for the interaction model defined in ( 2 ). Statistical significance based on the p-value of the fitted coefficient is indicated by:

p<0.01

(), $p<0.001$ (*).

Physics-Informed ML

Physics-informed machine learning combines data-driven models with physical laws to solve, predict, and infer scientific systems.

See all 833 papers

PINNs Failure Modes are Overfitting
May 29, 2026Nigel T. Andersen, Takashi MatsubaraNeural Network Generalization Neural PDE Solvers
Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan
Alternating Levenberg-Marquardt Training of Physics-Informed Neural Networks with Fourier-Enhanced Features
Aug 6, 2026Yulun Wu, Matthieu Barreau, Miguel Aguiar +1Neural PDE Solvers Spectral Bias
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, 100 44, Sweden
Physics-Informed Foresight Pruning for Sparse PINN Solvers of Nonlinear PDEs
Aug 26, 2026Ahmad Ishaque Karimi, Uvini Balasuriya Mudiyanselage, Kookjin LeeNeural PDE Solvers Neural Network Pruning
School of Computing and Augmented Intelligence Arizona State University Tempe, USA

May 29, 2026J/K move · Enter open · S save

PINNs Failure Modes are Overfitting

Nigel T. Andersen, Takashi Matsubara

Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan

Neural Network Generalization Neural PDE Solvers Backpropagation Physics-Informed ML

Open paper AskPDF arXiv

Physics-Informed Neural Networks (PINNs) are a common class of machine learning-based partial differential equation (PDE) solvers which train a network to represent a solution by minimizing a residual loss that encodes the PDE. Despite their successes, they are known to fail on certain simple equations, converging to an incorrect solution despite low loss. These failure modes have garnered significant attention in the literature over the past several years, motivating both architectural and optimization based solutions. By directly visualizing the residual, we show that failure modes are the result of overfitting: the loss is minimized on the collocation points, but not elsewhere. Applying regularization causes the failure modes to vanish. Finally, we extend double backpropagation over the full set of residuals, and use it to achieve state-of-the-art performance on four standard failure mode equations with up to

23\times

fewer collocation points and a vanilla architecture.

Abstract

Physics-Informed ML

Explore similar work

PINNs Failure Modes are Overfitting

Alternating Levenberg-Marquardt Training of Physics-Informed Neural Networks with Fourier-Enhanced Features

Physics-Informed Foresight Pruning for Sparse PINN Solvers of Nonlinear PDEs