Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models

Serge Kotchourko; Amin Totounferoush; Michael W. Mahoney; Steffen Staab

cs.LGMar 24, 2025

Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models

Authors: Serge Kotchourko, Amin Totounferoush, Michael W. Mahoney, Steffen Staab

Organizations: University of Stuttgart, Germany · ICSI, LBNL, and University of California, Berkeley, USA

Abstract

Scientific foundation models (SciFMs) aim to learn generalizable representations of physical systems governed by partial differential equations (PDEs), enabling transfer across tasks and domains. While physics-informed methods, which leverage PDE residuals as supervisory signals, have shown promise in scientific machine learning (SciML) for improving accuracy and reducing data requirements, their potential in the context of SciFMs remains relatively unexplored. In this evaluation study, we investigate whether (and how) physics-informed pre-training improves the generalization, robustness, and data efficiency of SciFMs. We conduct systematic experiments across a diverse set of PDEs, ranging from simple problems with periodic boundary conditions to more challenging systems such as the Navier-Stokes equations and non-periodic geometries. Our results show that physics-informed pre-training provides clear benefits in nice,'' e.g., structured, well-aligned settings: it enhances generalization and reduces data dependence, compared to data-only pre-training. However, these advantages diminish significantly as the downstream tasks become harder,'' e.g., as they involve discontinuities or deviate from the pre-training distribution. In complex or structurally different problems, such as those involving new boundary conditions or PDE operators, physics-informed models may perform only on par with---or even worse---than data-driven baselines. While residual-based pre-training helps in idealized regimes, realizing broadly transferable SciFMs will likely require subtler spatiotemporal inductive biases and more principled integration of physical knowledge into model architectures.

Figures & tables

Figure 1 : We evaluate physics-informed pre-trained models on their ability to generalize to shifted parameter regimes, unseen PDE operators, and noisy fine-tuning data—capturing challenges in new physics, new operators, and robustness.

Training-Stage	Benchmark	System	Coefficient Range
			ID	Slight-OOD	Medium-OOD	High-OOD
Pre-Training	-	Poisson ( $D$ )	$(1,5)$	-
		Advection-Diffusion ( $\Psi$ )	$(0.2,1)$	-
		Helmholtz ( $\omega$ )	$(1,10)$	-
Fine-Tuning	New Physics (zero- & n-shot)	Poisson ( $D$ )	$(1,2.5)$	$(2.5,7.5)$	$(7.5,12.5)$	$(15,20)$
		Advection-Diffusion ( $\Psi$ )	$(0.2,0.4)$	$(0.4,1.6)$	$(2,3)$	$(4,5)$

Table 1 : Scalar PDE coefficient ranges

(\min,\max)

for pre-training and fine-tuning: diffusion

D

, advection-diffusion ratio

\Psi

, wave number

\omega

, reaction term

r

, velocity field

v

, kinematic viscosity

\nu

Metric	Model	Poisson				Advection-Diffusion				Helmholtz
Metric	Model	ID	S-OOD	M-OOD	H-OOD	ID	S-OOD	M-OOD	H-OOD	ID	S-OOD	M-OOD	H-OOD
$\mu_{\ell_{2}}$	Scratch	1.0080	1.1736	1.0061	1.8586	1.0005	1.1850	1.2476	1.3294	1.0137	1.1039	1.5014	1.3249
	Data-Loss	0.0025	0.0102	0.1225	0.5750	0.0068	0.0434	0.8813	1.8948	0.0040	0.2448	1.2165	1.2121
	Physics-Informed	0.0005	0.0067	0.1326	0.7091	0.0025	0.0260	1.2705	4.1350	0.0007	0.2363	1.2177	1.1825

Table 2 : Zero-shot evaluation on downstream tasks governed by the pre-training PDEs, under in-distribution and slight, medium, and high out-of-distribution (S/M/H-OOD) parameterizations. Best per metric in bold.

Panel of Figure 2 — Figure 2 : $\mu_{\ell_{2}}$ error of fine-tuned models on the Advection-Diffusion (top) and Helmholtz (bottom) tasks using up to $2^{15}$ downstream examples, across ID and OOD parameter regimes, comparing pre-training strategies.

Figure 3 : $\mu_{\ell_{2}}$ error for varying noise levels in the solution domain on the OOD Helmholtz datasets. From left to right, $\sigma=0.01,0.05,0.1,0.2$ .

Figure 4 : $\mu_{\ell_{2}}$ metrics with increasing numbers of examples used during fine-tuning for the downstream fine-tuning tasks.

Panel of Figure 5 — Figure 5 : The fRMSE for low frequencies (a), medium frequencies (b), high frequencies (c), and $\mu_{\ell_{2}}$ loss (d) for the downstream task of turbulent Navier–Stokes.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Architecture	Model	Poisson				Advection-Diffusion				Helmholtz
Metric	Architecture	Model	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD
$\mu_{\ell_{2}}$	FNO	Physics-informed	0.0005	0.0067	0.1326	0.7091	0.0025	0.0260	1.2705	4.1350	0.0007	0.2363	1.2177	1.1825
	AViT	Physics-informed	0.2852	0.3476	0.4591	0.5693	0.3166	0.3899	0.5949	0.7527	0.7792	0.9276	0.9834	0.9809
$L_{\infty}$	FNO	Physics-informed	0.0007	0.0124	0.0836	0.3155	0.0023	0.1325	1.6630	2.7702	0.0036	1.5212	1.6534	0.6385
	AViT	Physics-informed	0.2635	0.2098	0.2266	0.2487	0.2432	0.2487	0.6253	1.4406	6.2318	5.9284	1.5243	0.5796

Appendix

Table 3 : Comparison of FNO and AViT architectures (approximately 67M parameters) evaluated on coefficient ranges progressively pushed OOD from the pre-training distribution. Bold entries indicate better performance. Predictions with error

>

0.2 are considered failures and omitted from analysis.

Metric	Model	Poisson				Advection-Diffusion				Helmholtz
Metric	Model	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD	ID	Slight-OOD	Medium-OOD	High-OOD
$\mu_{\ell_{2}}$	Scratch	1.0080	1.1736	1.0061	1.8586	1.0005	1.1850	1.2476	1.3294	1.0137	1.1039	1.5014	1.3249
	Data-Loss	0.0025	0.0102	0.1225	0.5750	0.0068	0.0434	0.8813	1.8948	0.0040	0.2448	1.2165	1.2121
	Physics-Informed	0.0005	0.0067	0.1326	0.7091	0.0025	0.0260	1.2705	4.1350	0.0007	0.2363	1.2177	1.1825
$L_{\infty}$	Scratch	0.8171	0.4959	0.4243	0.4702	0.6644	0.6286	0.9239	1.6784	7.8970	7.5244	1.5851	0.6475
	Data-Loss	0.0035	0.0182	0.1661	0.7197	0.0060	0.1172	1.0009	2.0415	0.0557	1.5207	1.6423	0.6822

Appendix

Table 4 : Zero-shot evaluation on downstream tasks governed by the pre-training PDEs, under in-distribution and slight, medium, and high out-of-distribution (S/M/H-OOD) parameterizations, using the

\mu_{\ell_{2}}

and

L_{\infty}

metrics. Best per metric in bold.

Figure 6 : The $\mu_{\ell_{2}}$ metric for the downstream tasks of Poisson, Advection-Diffusion, and Helmholtz, respectively, where the coefficient ranges are gradually pushed OOD.

Figure 7 : The $L_{\infty}$ metric for the downstream tasks of Poisson, Advection-Diffusion, and Helmholtz, respectively, where the coefficient ranges are gradually pushed OOD.

Figure 8 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Advection-Diffusion task, using the $\mu_{\ell_{2}}$ metric. The coefficient ranges for the ID, Slight-, Medium-, and High-OOD settings follow Table 1 .

Figure 9 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Advection-Diffusion task, using the $L_{\infty}$ metric.

Figure 10 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Helmholtz task, using the $\mu_{\ell_{2}}$ metric.

Figure 11 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Helmholtz task, using the $L_{\infty}$ metric.

Figure 12 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Advection-Diffusion task, using the fRMSE metric for low-, medium-, and high-frequency bands.

Figure 13 : Evaluation of models fine-tuned with varying numbers of downstream samples for the n-shot Helmholtz task, using the fRMSE metric for low-, medium-, and high-frequency bands.

Figure 14 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Advection-Diffusion task, using the $\mu_{\ell_{2}}$ metric.

Figure 15 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Advection-Diffusion task, using the $L_{\infty}$ metric.

Figure 16 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Helmholtz task, using the $\mu_{\ell_{2}}$ metric.

Figure 17 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Helmholtz task, using the $L_{\infty}$ metric.

Figure 18 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Advection-Diffusion task, using the fRMSE metric.

Figure 19 : Evaluation of models fine-tuned with varying numbers of noisy downstream samples for the n-shot Helmholtz task, using the fRMSE metric.

Figure 20 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Reaction-Diffusion task.

Figure 21 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Reaction-Diffusion problem.

Figure 22 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Reaction-Advection-Diffusion task.

Figure 23 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Reaction-Advection-Diffusion problem.

Figure 24 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Navier-Stokes task.

Figure 25 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Navier-Stokes (Laminar) problem.

Figure 26 : Evaluation of models fine-tuned with increasing numbers of downstream samples for the n-shot Darcy task.

Figure 27 : The fRMSE for low frequencies, medium frequencies, and high frequencies for the downstream tasks of the Darcy flow problem.

Explore similar work

May 28, 2026cs.LG

Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts

Recent physics foundation models claim general spatiotemporal forecasting ability, yet their evaluations often collapse performance into a single average score under a fixed training distribution. This makes it difficult to determine whether a model has learned generalizable physical dynamics or only performs well under particular settings. We construct a benchmark with 8 physical dynamics, 3 training-data mixtures, and 25 test regimes induced by dynamic-scale and initial-condition complexity shifts, covering in-distribution, distribution-shift, and out-of-distribution settings. We evaluate five physics foundation model architectures and four model variants per architecture (scratch and three pretrained sizes), resulting in 60,000 measurements. Our results show that current physics foundation models behave as conditional rather than universal generalists: their generality depends on the physical regime, temporal scale, initial-condition setting, pretraining, model size, and architecture. Improving the training data distribution only partially mitigates this limitation. Pretraining and scaling are also unable to reliably remove their ability biases. We argue that improving physics foundation models requires moving beyond scaling models or expanding data, toward learning mechanisms that better capture transferable physical knowledge across regimes, temporal scales, and distribution shifts.

Mengdi Chu, Yang Liu, Ayan Biswas +1

The Ohio State University · Los Alamos National Laboratory

Wireless Foundation Models Distribution Shifts Spatio-Temporal Forecasting Cross-Task Knowledge Transfer

Ask PDF

May 8, 2026cs.LG

PIMSM: Physics-Informed Multi-Scale Mamba for Stable Neural Representations under Distribution Shift

Scientific foundation models are expected to reuse representations under changes in dataset, acquisition protocol, and deployment domain, yet many sequence backbones treat scientific temporal structure as an unconstrained pattern to be fitted. We argue that this misses a central property of natural dynamical systems: neural and atmospheric time series are organized by interacting processes across multiple physical timescales, and failure to preserve this multiscale structure contributes to brittleness under distribution shift. We formalize this failure mode as temporal kernel mismatch, where a model fits in-distribution dynamics with an effective memory policy that is not anchored to the signal's physical timescales, leading to representation drift and degraded transfer. We propose Physics-Informed Multi-Scale Mamba (PIMSM), a state-space architecture that maps spectrum-estimated transition points between frequency regimes (knee frequencies) to scale-specific discretization parameters and anchors them to acquisition time units. On Human Connectome Project fMRI, PIMSM improves robustness and representation stability under severe temporal-context truncation, extreme low-resource transfer, and resting-state-to-task-state generalization. Without modality-specific adaptation, the same architecture also attains the lowest variable-wise MAE across all reported horizons and variables on Weather-5K held-out-station spatial out-of-distribution forecasting. These results support temporal-scale alignment as a practical inductive bias for scientific foundation models that must preserve structure, not only fit correlations, under deployment shift.

Sangyoon Bae, Shinjae Yoo, Jiook Cha

Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, South Korea, 08826 · Department of Psychology Seoul National University Seoul, South Korea, 08826

Time Series Foundation Models Wireless Foundation Models Neural Representations Spatio-Temporal Forecasting Distribution Shifts

Ask PDF

May 25, 2026cs.LG

A PAC-Bayesian View of Generalisation for Physics-Informed Machine Learning

Physics-informed machine learning (PIML) integrates mechanistic knowledge, typically in the form of partial differential equations (PDE), into data-driven models. Despite strong empirical performance, its statistical generalisation properties remain poorly understood, particularly in the regression setting with unbounded losses. Existing analyses rely on approximation or stability arguments and do not fully capture how physical structure influences generalisation from finite data. In this work, we develop a PAC-Bayesian framework for PIML that provides high-probability generalisation guarantees in the presence of unbounded losses. We adopt a multi-task perspective that jointly treats data fidelity, PDE residuals, initial and boundary conditions, avoiding the looseness induced by standard union-bound approaches. Our analysis leverages the structure of physics-informed objectives to derive novel bounds where the complexity scales with input-gradient norms of the losses, revealing a direct link between physical regularity and generalisation. We instantiate this framework under Sobolev and Poincaré-type assumptions, yielding two classes of bounds that trade off statistical complexity and smoothness in different regimes. Building on these results, we propose a self-bounding-aware learning algorithm that directly optimises tractable surrogates of the derived bounds, along with a practical procedure to estimate the associated constants in realistic settings. Empirical evaluations on standard PDE benchmarks demonstrate that our bounds are non-vacuous, significantly tighter than union-bound baselines, and can be effectively minimised during training. Overall, our results provide a principled statistical foundation for the generalisation of physics-informed models.

Thien V. Nguyen, Amaury Habrard, Benjamin Guedj

Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Laboratoire Hubert Curien UMR 5516, F-42023, Saint-Étienne, France · Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Laboratoire Hubert Curien UMR 5516, Inria, F-42023, Saint-Étienne, France · Institut Universitaire de France +1

Physics-Informed Learning Pac-Bayesian Theory Partial Differential Equations Generalisation

Ask PDF