Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models

Serge Kotchourko; Amin Totounferoush; Michael W. Mahoney; Steffen Staab

cs.LGMar 24, 2025

Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models

Authors: Serge Kotchourko, Amin Totounferoush, Michael W. Mahoney, Steffen Staab

Organizations: University of Stuttgart, Germany · ICSI, LBNL, and University of California, Berkeley, USA

Abstract

Scientific foundation models (SciFMs) aim to learn generalizable representations of physical systems governed by partial differential equations (PDEs), enabling transfer across tasks and domains. While physics-informed methods, which leverage PDE residuals as supervisory signals, have shown promise in scientific machine learning (SciML) for improving accuracy and reducing data requirements, their potential in the context of SciFMs remains relatively unexplored. In this evaluation study, we investigate whether (and how) physics-informed pre-training improves the generalization, robustness, and data efficiency of SciFMs. We conduct systematic experiments across a diverse set of PDEs, ranging from simple problems with periodic boundary conditions to more challenging systems such as the Navier-Stokes equations and non-periodic geometries. Our results show that physics-informed pre-training provides clear benefits in nice,'' e.g., structured, well-aligned settings: it enhances generalization and reduces data dependence, compared to data-only pre-training. However, these advantages diminish significantly as the downstream tasks become harder,'' e.g., as they involve discontinuities or deviate from the pre-training distribution. In complex or structurally different problems, such as those involving new boundary conditions or PDE operators, physics-informed models may perform only on par with---or even worse---than data-driven baselines. While residual-based pre-training helps in idealized regimes, realizing broadly transferable SciFMs will likely require subtler spatiotemporal inductive biases and more principled integration of physical knowledge into model architectures.

Figures & tables

Figure 1 : We evaluate physics-informed pre-trained models on their ability to generalize to shifted parameter regimes, unseen PDE operators, and noisy fine-tuning data—capturing challenges in new physics, new operators, and robustness.

Training-Stage	Benchmark	System	Coefficient Range
			ID	Slight-OOD	Medium-OOD	High-OOD
Pre-Training	-	Poisson ( $D$ )	$(1,5)$	-
		Advection-Diffusion ( $\Psi$ )	$(0.2,1)$	-
		Helmholtz ( $\omega$ )	$(1,10)$	-
Fine-Tuning	New Physics (zero- & n-shot)	Poisson ( $D$ )	$(1,2.5)$	$(2.5,7.5)$	$(7.5,12.5)$	$(15,20)$
		Advection-Diffusion ( $\Psi$ )	$(0.2,0.4)$	$(0.4,1.6)$	$(2,3)$	$(4,5)$

Table 1 : Scalar PDE coefficient ranges

(\min,\max)

for pre-training and fine-tuning: diffusion

D

, advection-diffusion ratio

\Psi

, wave number

\omega

, reaction term

r

, velocity field

v

, kinematic viscosity

\nu

Metric	Model	Poisson				Advection-Diffusion				Helmholtz
Metric	Model	ID	S-OOD	M-OOD	H-OOD	ID	S-OOD	M-OOD	H-OOD	ID	S-OOD	M-OOD	H-OOD
$\mu_{\ell_{2}}$	Scratch	1.0080	1.1736	1.0061	1.8586	1.0005	1.1850	1.2476	1.3294	1.0137	1.1039	1.5014	1.3249
	Data-Loss	0.0025	0.0102	0.1225	0.5750	0.0068	0.0434	0.8813	1.8948	0.0040	0.2448	1.2165	1.2121
	Physics-Informed	0.0005	0.0067	0.1326	0.7091	0.0025	0.0260	1.2705	4.1350	0.0007	0.2363	1.2177	1.1825

Table 2 : Zero-shot evaluation on downstream tasks governed by the pre-training PDEs, under in-distribution and slight, medium, and high out-of-distribution (S/M/H-OOD) parameterizations. Best per metric in bold.

Panel of Figure 2 — Figure 2 : $\mu_{\ell_{2}}$ error of fine-tuned models on the Advection-Diffusion (top) and Helmholtz (bottom) tasks using up to $2^{15}$ downstream examples, across ID and OOD parameter regimes, comparing pre-training strategies.

Figure 3 : $\mu_{\ell_{2}}$ error for varying noise levels in the solution domain on the OOD Helmholtz datasets. From left to right, $\sigma=0.01,0.05,0.1,0.2$ .

Figure 4 : $\mu_{\ell_{2}}$ metrics with increasing numbers of examples used during fine-tuning for the downstream fine-tuning tasks.

Panel of Figure 5 — Figure 5 : The fRMSE for low frequencies (a), medium frequencies (b), high frequencies (c), and $\mu_{\ell_{2}}$ loss (d) for the downstream task of turbulent Navier–Stokes.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Architecture	Model	Poisson				Advection-Diffusion				Helmholtz
Metric	Architecture	Model	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD
$\mu_{\ell_{2}}$	FNO	Physics-informed	0.0005	0.0067	0.1326	0.7091	0.0025	0.0260	1.2705	4.1350	0.0007	0.2363	1.2177	1.1825
	AViT	Physics-informed	0.2852	0.3476	0.4591	0.5693	0.3166	0.3899	0.5949	0.7527	0.7792	0.9276	0.9834	0.9809
$L_{\infty}$	FNO	Physics-informed	0.0007	0.0124	0.0836	0.3155	0.0023	0.1325	1.6630	2.7702	0.0036	1.5212	1.6534	0.6385
	AViT	Physics-informed	0.2635	0.2098	0.2266	0.2487	0.2432	0.2487	0.6253	1.4406	6.2318	5.9284	1.5243	0.5796

Appendix

Table 3 : Comparison of FNO and AViT architectures (approximately 67M parameters) evaluated on coefficient ranges progressively pushed OOD from the pre-training distribution. Bold entries indicate better performance. Predictions with error

>

0.2 are considered failures and omitted from analysis.

Metric	Model	Poisson				Advection-Diffusion				Helmholtz
Metric	Model	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD
$\mu_{\ell_{2}}$	Scratch	1.0080	1.1736	1.0061	1.8586	1.0005	1.1850	1.2476	1.3294	1.0137	1.1039	1.5014	1.3249
	Data-Loss	0.0025	0.0102	0.1225	0.5750	0.0068	0.0434	0.8813	1.8948	0.0040	0.2448	1.2165	1.2121
	Physics-Informed	0.0005	0.0067	0.1326	0.7091	0.0025	0.0260	1.2705	4.1350	0.0007	0.2363	1.2177	1.1825
$L_{\infty}$	Scratch	0.8171	0.4959	0.4243	0.4702	0.6644	0.6286	0.9239	1.6784	7.8970	7.5244	1.5851	0.6475
	Data-Loss	0.0035	0.0182	0.1661	0.7197	0.0060	0.1172	1.0009	2.0415	0.0557	1.5207	1.6423	0.6822

Appendix

Table 4 : Zero-shot evaluation on downstream tasks governed by the pre-training PDEs, under in-distribution and slight, medium, and high out-of-distribution (S/M/H-OOD) parameterizations, using the

\mu_{\ell_{2}}

and

L_{\infty}

metrics. Best per metric in bold.

Figure 6 : The $\mu_{\ell_{2}}$ metric for the downstream tasks of Poisson, Advection-Diffusion, and Helmholtz, respectively, where the coefficient ranges are gradually pushed OOD.

Figure 7 : The $L_{\infty}$ metric for the downstream tasks of Poisson, Advection-Diffusion, and Helmholtz, respectively, where the coefficient ranges are gradually pushed OOD.

Figure 8 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Advection-Diffusion task, using the $\mu_{\ell_{2}}$ metric. The coefficient ranges for the ID, Slight-, Medium-, and High-OOD settings follow Table 1 .

Figure 9 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Advection-Diffusion task, using the $L_{\infty}$ metric.

Figure 10 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Helmholtz task, using the $\mu_{\ell_{2}}$ metric.

Figure 11 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Helmholtz task, using the $L_{\infty}$ metric.

Figure 12 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Advection-Diffusion task, using the fRMSE metric for low-, medium-, and high-frequency bands.

Figure 13 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Helmholtz task, using the fRMSE metric for low-, medium-, and high-frequency bands.

Figure 14 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Advection-Diffusion task, using the $\mu_{\ell_{2}}$ metric.

Figure 15 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Advection-Diffusion task, using the $L_{\infty}$ metric.

Figure 16 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Helmholtz task, using the $\mu_{\ell_{2}}$ metric.

Figure 17 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Helmholtz task, using the $L_{\infty}$ metric.

Figure 18 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Advection-Diffusion task, using the fRMSE metric.

Figure 19 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Helmholtz task, using the fRMSE metric.

Figure 20 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Reaction-Diffusion task.

Figure 21 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Reaction-Diffusion problem.

Figure 22 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Reaction-Advection-Diffusion task.

Figure 23 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Reaction-Advection-Diffusion problem.

Figure 24 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Navier-Stokes task.

Figure 25 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Navier-Stokes (Laminar) problem.

Figure 26 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Darcy task.

Figure 27 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Darcy flow problem.

Explore similar work

Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts
May 28, 2026Mengdi Chu, Yang Liu, Ayan Biswas +1Wireless Foundation Models Distribution Shifts
The Ohio State University · Los Alamos National Laboratory
PIMSM: Physics-Informed Multi-Scale Mamba for Stable Neural Representations under Distribution Shift
May 8, 2026Sangyoon Bae, Shinjae Yoo, Jiook ChaTime Series Foundation Models Wireless Foundation Models
Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, South Korea, 08826 · Department of Psychology Seoul National University Seoul, South Korea, 08826
A PAC-Bayesian View of Generalisation for Physics-Informed Machine Learning
May 25, 2026Thien V. Nguyen, Amaury Habrard, Benjamin GuedjPhysics-Informed Learning Pac-Bayesian Theory
Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Laboratoire Hubert Curien UMR 5516, F-42023, Saint-Étienne, France · Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Laboratoire Hubert Curien UMR 5516, Inria, F-42023, Saint-Étienne, France · Institut Universitaire de France +1

May 28, 2026, cs.LGJ/K move · Enter open · S save

Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts

Mengdi Chu, Yang Liu, Ayan Biswas, Han-Wei Shen

The Ohio State University · Los Alamos National Laboratory

Wireless Foundation Models Distribution Shifts Spatio-Temporal Forecasting Cross-Task Knowledge Transfer

Open paper AskPDF arXiv

Recent physics foundation models claim general spatiotemporal forecasting ability, yet their evaluations often collapse performance into a single average score under a fixed training distribution. This makes it difficult to determine whether a model has learned generalizable physical dynamics or only performs well under particular settings. We construct a benchmark with 8 physical dynamics, 3 training-data mixtures, and 25 test regimes induced by dynamic-scale and initial-condition complexity shifts, covering in-distribution, distribution-shift, and out-of-distribution settings. We evaluate five physics foundation model architectures and four model variants per architecture (scratch and three pretrained sizes), resulting in 60,000 measurements. Our results show that current physics foundation models behave as conditional rather than universal generalists: their generality depends on the physical regime, temporal scale, initial-condition setting, pretraining, model size, and architecture. Improving the training data distribution only partially mitigates this limitation. Pretraining and scaling are also unable to reliably remove their ability biases. We argue that improving physics foundation models requires moving beyond scaling models or expanding data, toward learning mechanisms that better capture transferable physical knowledge across regimes, temporal scales, and distribution shifts.

Abstract

Explore similar work

Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts

PIMSM: Physics-Informed Multi-Scale Mamba for Stable Neural Representations under Distribution Shift

A PAC-Bayesian View of Generalisation for Physics-Informed Machine Learning