Out-of-distribution transfer of PDE foundation models to material dynamics under extreme loading
Authors: Mahindra Rautela, Alexander Most, Siddharth Mansingh, Aleksandra Pachalieva, Bradley Love, Daniel O Malley, Alexander Scheinker, Kyle Hickmann, Diane Oyen, Nathan Debardeleben, Earl Lawrence, Ayan Biswas
Organizations: AOT-IC Group, Los Alamos National Laboratory, Los Alamos, New Mexico, US, 87545 · CAI Division, Los Alamos National Laboratory, Los Alamos, New Mexico, US, 87545 · EES Divison, Los Alamos National Laboratory, Los Alamos, New Mexico, US, 87545 · XCP Division, Los Alamos National Laboratory, Los Alamos, New Mexico, US, 87545 · HPC Division, Los Alamos National Laboratory, Los Alamos, New Mexico, US, 87545
Most PDE foundation models are pretrained and fine-tuned on fluid-centric benchmarks. Their utility under extreme-loading material dynamics remains unclear. We benchmark out-of-distribution transfer on two discontinuity-dominated regimes in which shocks, evolving interfaces, and fracture produce highly non-smooth fields: shock-driven multi-material interface dynamics (perturbed layered interface or PLI) and dynamic fracture/failure evolution (FRAC). We formulate the downstream task as terminal-state prediction, i.e., learning a long-horizon map that predicts the final state directly from the first snapshot without intermediate supervision. Using a unified training and evaluation protocol, we evaluate two open-source pretrained PDE foundation models, POSEIDON and MORPH, and compare fine-tuning from pretrained weights against training from scratch across training-set sizes to quantify sample efficiency under distribution shift.
Figures & tables
Dataset/Models
MORPH
POSEIDON
PLI
0.05445
0.13892
FRAC
0.04103
0.03704
Table 1: MSE on the test set
Figure 1: Terminal-state prediction on PLI comparing MORPH-Ti (a) and POSEIDON-T (b). Columns show the input, target terminal state, prediction, and squared error (T−P)2 .
Figure 2: Terminal-state prediction on FRAC comparing MORPH-Ti (a) and POSEIDON-T (b). For each test example, columns show the input (initial state), target terminal state, model prediction, and squared error map (T−P)2 .
Figure 3: Test MSE as a function of training-set size for PLI (a) and FRAC (b), comparing fine-tuning from pretrained weights versus training from scratch for MORPH and POSEIDON. Curves quantify sample efficiency for first-frame → final-frame operator learning.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Representative PLI spatiotemporal evolution over uniformly spaced time-steps. Rows visualize selected per-material density channels.
Figure 5: Representative FRAC trajectories (tungsten subset) showing the evolution of fracture/damage patterns from the initial condition to the terminal state. Each row corresponds to a distinct simulation, with frames annotated by characteristic time.
Figure 6: Additional FRAC terminal-state prediction results comparing MORPH-Ti (a) and POSEIDON-T (b) using the same visualization as Fig. 2 (input, target, prediction, and squared error (T−P)2 ).
Figure 7: Additional PLI terminal-state prediction results comparing MORPH-Ti (a) and POSEIDON-T (b). Columns show input, target, prediction, and squared error (T−P)2 .
Figure 8: Further PLI terminal-state prediction results comparing MORPH-Ti (a) and POSEIDON-T (b). Columns show input, target, prediction, and squared error (T−P)2 .
Foundation models for partial differential equations (PDEs) have emerged as powerful surrogates pre-trained on diverse physical systems, but adapting them to new downstream tasks remains challenging due to limited task-specific data and distribution shifts. While fine-tuning has proven transformative in natural language processing, best practices for adapting PDE foundation models remain underexplored. Although physics-informed training has successfully trained accurate solvers across a wide range of PDE problems, its potential for fine-tuning data-based foundation models has not been systematically studied. In this work, we introduce a physics-informed fine-tuning framework that adapts pre-trained PDE foundation models by incorporating physical constraints (PDE residuals and boundary conditions) directly into the fine-tuning objective. This enables effective adaptation in data-scarce regimes while promoting physical consistency. We evaluate our method on a downstream task composed of an unseen PDE class and compare it with data-driven finetuning counterparts. Our results demonstrate that physics-informed fine-tuning achieves competitive accuracy without requiring PDE solutions for training. Furthermore, a hybrid fine-tuning strategy yields superior generalization to out-of-distribution scenarios when only minimal training data is available. These findings establish physics-informed fine-tuning as a scalable and data-efficient paradigm, providing a physically interpretable pathway for adapting foundation models in scientific machine learning.
Vlad Medvedev, Leon Armbruster, Christopher Straub +2
Department Modeling and Artificial Intelligence Fraunhofer Institute for Integrated Systems and Device Technology IISB Schottkystrasse 10, 91058 Erlangen, Germany
Pretrained partial differential equation (PDE) foundation models can generalize across different equations, but adapting them to unseen PDE systems typically requires dense solution data, which is often expensive or unavailable. To address this limitation, we propose an unsupervised PDE-based finetuning framework that eliminates the need for ground-truth solutions. We first pretrain a neighborhood attention Transformer on diverse time-dependent PDEs spanning varying spatial scales, yielding transferable representations across heterogeneous equations. In the adaptation stage, we construct a physics-based objective using the PDE residual and boundary conditions, and finetune the model on unseen equations via low-rank adaptation (LoRA). To address the uneven learning across physical quantities in standard LoRA, we introduce NSLoRA, a Newton-Schulz orthogonalized variant that rebalances adaptation. Our method achieves performance comparable to supervised LoRA finetuning without requiring any ground-truth solutions, while consistently outperforming competitive neural operator baselines and recent PDE foundation models across heterogeneous PDE benchmarks spanning multiple spatial dimensions.
Ziye Song, Zhao Wei, Xin Yu +2
Nanyang Technological University · CFAR, A*STAR · Adelaide University
We introduce Tadpole, a novel foundation model for three-dimensional partial differential equations (PDEs) that addresses key challenges in transferability, scalability to high dimensionality, and multi-functionality. Tadpole is pre-trained as an autoencoder on synthetic 3D PDE data generated by an efficient online data-generation framework. This enables large-scale, diverse training without storage or I/O overhead, demonstrated by scaling to an equivalent of hundreds of terabytes of training data. By autoencoding single-channel spatial crops, Tadpole learns rich and transferable representations across heterogeneous physical systems with varying numbers of state variables and spatial resolutions. Although pre-trained solely as an autoencoder, Tadpole can be efficiently applied for multiple downstream tasks beyond reconstruction, including dynamics learning and generative modeling. For dynamics learning, we propose a novel parameter-efficient fine-tuning strategy that integrates low-rank adaptation, latent-space transformations, and reintroduced skip connections, achieving accurate temporal modeling with a minimal number of trainable parameters. Tadpole demonstrates strong fine-tuning performance across various downstream tasks, highlighting its versatility and effectiveness as a foundation model for 3D PDE learning. Source code and pre-trained weights of Tadpole are available at https://github.com/tum-pbs/tadpole
Qiang Liu, Felix Koehler, Benjamin Holzschuh +1
TUM School of Computation, Information and Technology, Technical University of Munich, Garching, Germany · MCML, Munich Center for Machine Learning, Munich, Germany