Neither Precision Nor Architecture Alone: Controlled Tests of Failure Remedies for Physics-Informed Neural Networks

Jinyuan Zhang; Peng He; He Hu; Yin Yuan; ShengShuo Jiao

cs.LGAug 26, 2026

Neither Precision Nor Architecture Alone: Controlled Tests of Failure Remedies for Physics-Informed Neural Networks

Authors: Jinyuan Zhang, Peng He, He Hu, Yin Yuan, ShengShuo Jiao

Organizations: Hubei University, Wuhan, China

Abstract

Physics-Informed Neural Networks (PINNs) frequently fail on stiff or advection-dominated PDEs, and two recent accounts offer competing remedies: switching from FP32 to FP64 to repair an L-BFGS stopping artifact, or replacing the MLP with a state-space-model (SSM) backbone plus sub-sequence alignment to counter architectural simplicity bias. We test both under matched, seed-paired controls in a pre-registered 144-run study spanning convection, reaction, and wave, plus an independent 85-run convection/wave study; success is relative $\ell_2$ error below $0.05$ . The two remedies act on disjoint regime-and-seed slices: neither substitutes for the other. On hard convection ( $β{=}50$ ), alignment recovers 2/5 seeds in FP32 and 3/5 in FP64, where the unaligned SSM succeeds on 0/5 seeds at either precision and the vanilla MLP moves only from 0/5 to 1/5 across the precision switch---the recoveries trace to the alignment objective, not the backbone. On reaction the backbone alone already succeeds on 3/5--4/5 seeds, so each remedy covers a regime the other does not. Responses are also seed-specific: the same precision switch flips individual seeds in opposite directions and, on wave, lowers median error with no statistically significant success gain. Tightening the inner L-BFGS tolerance in an independent repeated-step runner likewise lowers median error at a large runtime cost, with success counts unchanged. Precision, stopping, backbone, and alignment must therefore be evaluated jointly and reported per seed.

Figures & tables

Figure 1: Why the two remedies require a joint test. Prior studies isolate numerical precision/stopping from architecture/alignment, whereas our matched comparisons vary precision, inner L-BFGS tolerance, backbone, and alignment weight while controlling the PDE regime and seed. rL2 and archived diagnostic proxies appear as descriptive summaries.

Figure 2: Overview of the controlled study workflow.

Slice	RQ	$\boldsymbol{n}$
Local study: five-seed conv/react slice	1–2,4	60
Local study: instrumented conv/react subset	5	$87^{*}$
Local study: model-protocol comparison	2	36
Independent study: convection $\alpha$ sweep	2	25
Independent study: convection tolerance sweep	3	30
Independent study: wave MLP precision slice	4	20

Table 1: Evidence coverage. Local controlled-study slices overlap; independent-study slices partition the 85 runs, with one non-comparable slice excluded from validation claims. Asterisked proxy counts have unequal observability.

Figure 3: Local controlled-study five-seed slice (success defined as rL2 $<0.05$ ), pooling registered and matched follow-up runs; FP64 no-alignment seeds 3–4 are drawn from the follow-ups. Precision and architecture/alignment produce regime- and seed-specific gains, with no uniform monotone improvement.

PDE	Regime	s=0	s=1	s=2	s=3	s=4	Rate
Conv- $\beta$ 50	mlp FP32	1.005	0.896	0.806	0.914	0.800	0/5
	mlp FP64	0.080	0.738	0.077	0.045	0.052	1/5
	ssm_noalign FP32	2.027	1.361	1.188	1.327	1.372	0/5
	ssm_noalign FP64	1.312	1.400	1.298	1.238	1.326	0/5
	ssm_align FP32	0.011	0.0498	0.549	0.386	0.904	2/5
	ssm_align FP64	0.006	1.287	0.005	0.168	0.026	3/5

Table 2: Per-seed rL2 in the local controlled-study slice (five seeds per cell; bold is rL2

<0.05

on unrounded values).

Neural PDE Solvers

Neural PDE solvers use trainable models to approximate solutions or solution operators for partial differential equations, often with physical constraints.

See all 268 papers

PINNs Failure Modes are Overfitting
May 29, 2026Nigel T. Andersen, Takashi MatsubaraNeural Network Generalization Neural PDE Solvers
Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan
LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networks
Jul 15, 2026Nilay Anurag, Shital Adhikari, Taniya Kapoor +1Neural PDE Solvers Neural Network Optimization
Department of Computer Science, Stevens Institute of Technology · Artificial Intelligence Group, Wageningen University & Research
Silent Failures in Physics-Informed Neural Networks: Parameter Poisoning and the Limits of Loss-Based Validation
Jun 23, 2026David McShannon, Nicholas DietrichModel Misspecification Physics-Informed ML

May 29, 2026J/K move · Enter open · S save

PINNs Failure Modes are Overfitting

Nigel T. Andersen, Takashi Matsubara

Graduate School of Information Science and Technology Hokkaido University Sapporo, Japan · RIKEN Center for Advanced Intelligence Project (AIP) Tokyo, Japan

Neural Network Generalization Neural PDE Solvers Backpropagation Physics-Informed ML

Open paper AskPDF arXiv

Physics-Informed Neural Networks (PINNs) are a common class of machine learning-based partial differential equation (PDE) solvers which train a network to represent a solution by minimizing a residual loss that encodes the PDE. Despite their successes, they are known to fail on certain simple equations, converging to an incorrect solution despite low loss. These failure modes have garnered significant attention in the literature over the past several years, motivating both architectural and optimization based solutions. By directly visualizing the residual, we show that failure modes are the result of overfitting: the loss is minimized on the collocation points, but not elsewhere. Applying regularization causes the failure modes to vanish. Finally, we extend double backpropagation over the full set of residuals, and use it to achieve state-of-the-art performance on four standard failure mode equations with up to

23\times

fewer collocation points and a vanilla architecture.

Abstract

Neural PDE Solvers

Explore similar work

PINNs Failure Modes are Overfitting

LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networks

Silent Failures in Physics-Informed Neural Networks: Parameter Poisoning and the Limits of Loss-Based Validation