We investigate the impact of tokeniser pretraining on the accuracy and efficiency of physics emulation. Modern high-resolution simulations produce vast volumes of data spanning diverse physical regimes and scales. Training foundation models to learn the dynamics underlying such data enables the modelling of complex multiphysics phenomena, especially in data-limited settings. The emerging class of physics foundation models typically aims to learn two tasks jointly: (i) extracting compact representations of high-resolution spatiotemporal data, and (ii) capturing governing physical dynamics. However, learning both tasks from scratch simultaneously can impede the effectiveness of either process. We show that pretraining the tokeniser with an autoencoding objective prior to training the dynamics model enhances computational efficiency for physics emulation. Notably, the magnitude of this benefit depends on domain alignment: pretraining on the same physical system as the emulation task yields the largest improvements, while pretraining on other systems provides moderate gains. In-domain pretraining reduces VRMSE by 64% after 10,500 training steps compared to training from scratch. To our knowledge, this is the first systematic investigation of tokeniser pretraining for physics foundation models. We further introduce flexible spatiotemporal compression operations that extend causal convolutions to support runtime-adjustable compression ratios, enabling efficient adaptation to diverse downstream tasks. Our findings provide practical guidance for training efficient physics emulators and highlight the importance of strategic pretraining data selection.
Figures & tables
Dataset
Training size
736k
267.4k
171.1k
12.6k
Dimensions
(512,512)
(512,128)
(256,512)
(256,256)
Field Type
Field Name
Euler
Rayleigh-Bénard
Shear Flow
Active Matter
Scalar
Energy
✓
Density
✓
Pressure
✓
✓
✓
Table 1: Dataset sizes, dimensions, and available fields.
Figure 1: Overview of the experimental setup.
Tokeniser
Processor & Projection
Total
5 M
100 M
105 M
Table 2: Parameter counts for the tokeniser and downstream components.
Strategy
Tokeniser
# Trainable
Trainable Fraction
Fully trainable ( )
5 M
5 M
100 %
Mostly frozen ( )
85 k
2 %
Table 3: Parameter training strategies and corresponding trainable parameter budgets.
Table 4: Wall-clock time for 2100 training steps (33,600 examples across eight GPUs).
VRMSE
Spectral–Low
Spectral–Mid
Spectral–High
No pretraining
0.439
0.021
1.039
1.408
In-domain
0.158
0.001
0.167
1.381
In-domain
0.162
0.002
0.190
1.545
Out-of-domain
0.355
0.012
0.642
1.659
Out-of-domain
0.643
0.544
0.801
1.229
Table 5: Next-frame prediction performance on the validation set after 10,500 training steps for different pretraining and freezing configurations. Metrics are averaged over 10 random subsets of 4096 validation examples. The and symbols indicate fully trainable and mostly frozen tokenisers, respectively.
Figure 2: Next-frame prediction learning curves on the validation set, shown over 29,400 training steps for different pretraining and freezing configurations. Each panel shows a different evaluation metric: VRMSE (leftmost), and spectral error at low-, mid-, and high-frequency ranges.
Figure 3: Autoregressive rollout learning curves over 210,000 training steps. Each panel shows VRMSE at a different rollout horizon. The shaded region indicates the training range in Figure 2 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: MAGVIT-Simple architecture.
Hyperparameter
Tokeniser Pretraining
Rollout Training
Optimiser
Optimiser
SOAP
AdamW
[Peak] learning rate
5×10−4
5×10−5
Betas
(0.95,0.95)
(0.9,0.999)
Weight decay
0.01
10−4
ϵ
10−8
10−10
Appendix
Table 6: Training hyperparameters for tokeniser pretraining and autoregressive rollout training.
Depth
Temporal Compression
Spatial Compression
#1
1
2
#2
{1, 2}
{2, 4}
#3
{1, 2}
{2, 4}
Appendix
Table 7: Temporal and spatial compression factors at each tokeniser depth. Values in braces indicate the set from which a factor is uniformly sampled during training.