Foundation models trained on text and images have been shown to develop abstract internal features that align with human concepts and that can be directly manipulated via activation steering to alter model behaviour. Whether scientific foundation models learn similarly abstract and domain-general representations has remained an open question. Inspired by recent work identifying single directions in activation space that control complex behaviours in large language models, we show that Walrus, a large physics foundation model, learns linearly steerable representations of physical phenomena. By computing the difference between activations representing contrasting physical regimes, we identify single directions in activation space that correspond to vorticity, diffusion, and even temporal progression. We find that injecting these concept directions back into the model during inference enables fine-grained causal control: vortices can be induced or removed, diffusion enhanced or suppressed, and simulations sped up or slowed down. Moreover, directions derived from one physical system also steer the behaviour of other physical systems, suggesting that they are domain-general. These results provide early evidence that scientific foundation models can learn general representations of physical principles, and further support for the Linear Representation Hypothesis.
Figures & tables
Figure 1: Schematic illustration of methodology. Activations are first extracted from the physics model during forward passes over input segments that exhibit physical feature f , yielding activations μf,i , and from segments lacking the feature, ¬f , yielding νf,i . The difference between these activations, Δf , is then injected back into Walrus during inference to steer future results.
Figure 2: Negative Δvortex injection into shear flow vortex regime, for α values of 0, 0.3, 0.5 and 1.0. Frame: 64.
Figure 3: Positive Δvortex injection into shear flow laminar regime, for α values of 0, 0.2, 0.3 and 0.4. Frame: 64.
Figure 4: On the left tracer fields for Δdiffusion injection into Shear Flow vortex regime with (top) α=0.1 and (bottom) α=−0.1 . On the right tracer fields for Δspeed injection into Shear Flow vortex regime with (top) α=0.1 and (bottom) α=−0.1 . Frame(left): 30, Frame(right): 24.
Figure 5: Transfer of Δvortex concept injection to Rayleigh-Bénard simulations. Pressure and buoyancy fields for (top) averaging over spatial dimensions: (left) α=−0.1 , (centre) α=0.0 , (right) α=0.1 ; (bottom) including spatial dimensions (no averaging): (left) α=−0.1 , (centre) α=0.0 , (right) α=0.1 . Frame(top): 40, Frame(bottom): 50.
Figure 6: (Left) Density field for Δvortex injection into Euler quadrants. (Middle) Density field for Δspeed injection into Euler quadrants. (Right) Chemical species B for Δvortex injection into Gray-Scott reaction diffusion. All computed by averaging over spatial dimensions, with α=0.1 (top), α=0.0 (middle), and α=−0.1 (bottom). Frames: 50 (left), 28 (middle), 48 (right).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Trace, pressure, x velocity and y velocity for Δvortex injection into shear flow laminar regime with (top) α=0.2 , (middle) α=0.0 and (bottom) α=−0.2 . Frame: 64.
Figure 8: Trace, pressure, x velocity and y velocity for Δvortex injection into shear flow laminar regime with (top) α=0.7 , (middle) α=0.0 and (bottom) α=−0.5 . Frame: 64.
Figure 9: Trace, pressure, x velocity and y velocity for Δdiffusion injection into shear flow vortex regime with (top) α=0.2 , (middle) α=0.0 and (bottom) α=−0.2 . Frame: 28.
Video world models learn representations of physical dynamics, but controlling their physical expectations at inference time remains an open problem. Recent interpretability work identified a Physics Emergence Zone (PEZ), a group of middle transformer layers in VideoMAE where physical plausibility is represented separately from other visual features. However, it remained unclear whether this structure could be used to directly control the model's physics reasoning. We present physics steering, a training-free method that uses the weight vector of a linear probe at a PEZ layer as a Concept Activation Vector (CAV) and injects it into hidden states during inference. This shifts the model's physical expectations without changing any model weights. On the IntPhys benchmark, this intervention reliably shifts the model's plausibility judgment in either direction, depending on the steering sign. The effect appears only when the intervention is applied within the Physics Emergence Zone, suggesting that the relevant physics representation is localized there. We further find that physics is encoded separately from motion direction, and that different intuitive physics principles occupy distinct directions within this representation space. Together, these results show that physical reasoning in VideoMAE is not only readable, but also directly steerable.
Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis-based stabilization method, load-balanced distributed 2D and 3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short and long term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches. Code and weights are released for community use.
Michael McCabe, Payel Mukhopadhyay, Tanya Marwah +22
Polymathic AI · Flatiron Institute · New York University +9
Recent physics foundation models claim general spatiotemporal forecasting ability, yet their evaluations often collapse performance into a single average score under a fixed training distribution. This makes it difficult to determine whether a model has learned generalizable physical dynamics or only performs well under particular settings. We construct a benchmark with 8 physical dynamics, 3 training-data mixtures, and 25 test regimes induced by dynamic-scale and initial-condition complexity shifts, covering in-distribution, distribution-shift, and out-of-distribution settings. We evaluate five physics foundation model architectures and four model variants per architecture (scratch and three pretrained sizes), resulting in 60,000 measurements. Our results show that current physics foundation models behave as conditional rather than universal generalists: their generality depends on the physical regime, temporal scale, initial-condition setting, pretraining, model size, and architecture. Improving the training data distribution only partially mitigates this limitation. Pretraining and scaling are also unable to reliably remove their ability biases. We argue that improving physics foundation models requires moving beyond scaling models or expanding data, toward learning mechanisms that better capture transferable physical knowledge across regimes, temporal scales, and distribution shifts.
Mengdi Chu, Yang Liu, Ayan Biswas +1
The Ohio State University · Los Alamos National Laboratory