cs.LGOct 29, 2026

Frozen in Time: How a Physics Foundation Model Forgets Under Sim-to-Real Adaptation

Authors: Haaris Mian, Miles Cranmer, Alessandro Favero, Payel Mukhopadhyay

Organizations: University of Cambridge · Polymathic AI

Abstract

Scientific foundation models (SciFMs) have achieved remarkable performance on a wide range of PDEs, including the ability to efficiently adapt to unseen data distributions with additional finetuning. As with all neural architectures, finetuning such models comes with the risk of catastrophic forgetting, whereby adapting to new tasks degrades prior knowledge of the model. We study catastrophic forgetting in Walrus-1.3B, a SciFM for continuum dynamics, over the course of a two-stage sim-to-real adaptation: first finetuning on simulated flow around a cylinder and then adapting to experimental data of the same system. We find that finetuning on the simulations rapidly degrades rollout performance on the original pretraining datasets, with one-step errors increasing up to 35x after just 20 epochs. Subsequent adaptation to real data further erodes performance on the simulated task. We characterize the potential drivers of forgetting, such as dataset dependence and task representation similarity. We then show that replaying as little as 1% simulated data during real-data finetuning recovers most of the lost simulation performance while retaining effective adaptation to the experimental dynamics. Our results identify catastrophic forgetting as a practical obstacle to sim-to-real adaptation of SciFMs and show that lightweight replay strategies significantly improve retention.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Nov 19, 2025

Walrus: A Cross-Domain Foundation Model for Continuum Dynamics

Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis-based stabilization method, load-balanced distributed 2D and 3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short and long term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches. Code and weights are released for community use.
Oct 1, 2026

Sim+Real: Joint Simulation-Experiment Training Improves Balanced Prediction in Physical Systems

Simulation and experimental measurements provide complementary data for learning spatiotemporal physical systems, but standard simulation-to-experiment fine-tuning optimizes only the experimental objective after transfer and can degrade simulation performance. We formulate simulation--experiment prediction as a multi-objective learning problem with domain-specific simulation and experimental risks. On four fluid systems from RealPDEBench and two model capacities, we compare Simulation only, Experiment only, Sim→\rightarrowExp, and Joint training, evaluating every final model on both held-out domains. Sim→\rightarrowExp tends to specialize more strongly to experimental data at the cost of simulation-domain forgetting. Joint training consistently achieves the best balanced performance over a broad range of simulation--experiment evaluation weightings, while substantially improving simulation retention over Sim→\rightarrowExp. Joint also better preserves simulation-only fields absent from experimental measurements. Project page: https://mahindrautela.github.io/morph.
Jan 9, 2025

Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives

Have we made mitigating catastrophic forgetting more complicated than it needs to be? Continual adaptation often degrades performance on earlier tasks, motivating rehearsal buffers, auxiliary objectives, and adaptive merging procedures. We investigate how far a simpler alternative-periodic parameter interpolation during fine-tuning-can go on its own. We introduce Sequential Fine-tuning with Averaging (SFA) and evaluate it alongside replay-free Lookahead, originally designed for optimization stability to find compelling results. In Qwen3-14B adapter continual learning, fixed configurations of both methods outperform the tested replay and adaptive-merging configurations in mean final accuracy using up to 36.3% less task training time. Furthermore, across language and vision tasks, our results compete with more complex continual-learning methods. Together, these findings suggest that simple in-training interpolation deserves consideration before introducing more elaborate forgetting-mitigation mechanisms.