Foundation models trained on text and images have been shown to develop abstract internal features that align with human concepts and that can be directly manipulated via activation steering to alter model behaviour. Whether scientific foundation models learn similarly abstract and domain-general representations has remained an open question. Inspired by recent work identifying single directions in activation space that control complex behaviours in large language models, we show that Walrus, a large physics foundation model, learns linearly steerable representations of physical phenomena. By computing the difference between activations representing contrasting physical regimes, we identify single directions in activation space that correspond to vorticity, diffusion, and even temporal progression. We find that injecting these concept directions back into the model during inference enables fine-grained causal control: vortices can be induced or removed, diffusion enhanced or suppressed, and simulations sped up or slowed down. Moreover, directions derived from one physical system also steer the behaviour of other physical systems, suggesting that they are domain-general. These results provide early evidence that scientific foundation models can learn general representations of physical principles, and further support for the Linear Representation Hypothesis.
Figures & tables
Figure 1: Schematic illustration of methodology. Activations are first extracted from the physics model during forward passes over input segments that exhibit physical feature f , yielding activations μf,i , and from segments lacking the feature, ¬f , yielding νf,i . The difference between these activations, Δf , is then injected back into Walrus during inference to steer future results.
Figure 2: Negative Δvortex injection into shear flow vortex regime, for α values of 0, 0.3, 0.5 and 1.0. Frame: 64.
Figure 3: Positive Δvortex injection into shear flow laminar regime, for α values of 0, 0.2, 0.3 and 0.4. Frame: 64.
Figure 4: On the left tracer fields for Δdiffusion injection into Shear Flow vortex regime with (top) α=0.1 and (bottom) α=−0.1 . On the right tracer fields for Δspeed injection into Shear Flow vortex regime with (top) α=0.1 and (bottom) α=−0.1 . Frame(left): 30, Frame(right): 24.
Figure 5: Transfer of Δvortex concept injection to Rayleigh-Bénard simulations. Pressure and buoyancy fields for (top) averaging over spatial dimensions: (left) α=−0.1 , (centre) α=0.0 , (right) α=0.1 ; (bottom) including spatial dimensions (no averaging): (left) α=−0.1 , (centre) α=0.0 , (right) α=0.1 . Frame(top): 40, Frame(bottom): 50.
Figure 6: (Left) Density field for Δvortex injection into Euler quadrants. (Middle) Density field for Δspeed injection into Euler quadrants. (Right) Chemical species B for Δvortex injection into Gray-Scott reaction diffusion. All computed by averaging over spatial dimensions, with α=0.1 (top), α=0.0 (middle), and α=−0.1 (bottom). Frames: 50 (left), 28 (middle), 48 (right).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Trace, pressure, x velocity and y velocity for Δvortex injection into shear flow laminar regime with (top) α=0.2 , (middle) α=0.0 and (bottom) α=−0.2 . Frame: 64.
Figure 8: Trace, pressure, x velocity and y velocity for Δvortex injection into shear flow laminar regime with (top) α=0.7 , (middle) α=0.0 and (bottom) α=−0.5 . Frame: 64.
Figure 9: Trace, pressure, x velocity and y velocity for Δdiffusion injection into shear flow vortex regime with (top) α=0.2 , (middle) α=0.0 and (bottom) α=−0.2 . Frame: 28.