Self-supervised world models are often said to organize their latents around physical quantities. We test this on 2D decaying Navier-Stokes flow in three pre-registered studies, each tested once on held-out data. In Study 1, a joint-embedding predictive architecture (JEPA) trained to predict the flow fails the pre-registered criteria for organizing its latent around fluid quantities better than an untrained network of the same architecture (not supported); a JEPA trained on non-physical linear dynamics and a static invariance model that never sees time do as well or better. In Study 2, plain autoencoders that render the field make enstrophy prominent in their slowest unsupervised latent directions (best r 0.80-0.86 vs 0.38 untrained; fresh seeds, sealed data). Adding VICReg lowers this prominence in every comparison, though not by the pre-registered margin for vorticity, while raising the autoencoders' linear decodability (partially supported). In Study 3, we build steerable directions ("dials") by routing: the decoder gets overall scale only through a small conditioning code, while its other pathway carries a degraded copy of spectral shape. Label-free rules find a loudness dial in all 8 trained models and a shape dial in 3 of 5 fresh seeds on sealed data, against a pre-registered bar of 4; neither control shows a shape dial (partially supported). Across the studies, what a latent contains and what can be controlled through it come apart. In exploration, a supervised direction reading spectral shape almost perfectly moved nothing when steered. During the build, the loudness dial had to be found by its effect on the output (closest to a pure rescaling), not by data variance; for shape, a variance rule and an effect-constrained rule each worked in 3 of 5 seeds. The program used about 70 logged hours on one consumer GPU; all pre-registrations, deviations, code and outputs are public.
Figures & tables
Property
Measure
Used in
Decodability
Ridge-regression probe from the z -scored latent to a target; R2 on held-out trajectories
Studies 1–3
Prominence
Correlation between a target and the latent’s slowest unsupervised directions
Studies 1–2
Steerability
Change in the decoded field when the latent is moved along a direction
Study 3
Table 1
Model
Training signal
Role
JEPA
Predicts the target encoder’s 8×8 feature map of frame t+k from frames (t−1,t) , k∈{1,4} ; EMA target; VICReg variance and covariance terms; no pixel loss
Main model
Pixel predictor
Decodes the velocity field of frame t+k ; mean squared error
Comparison
Untrained
None (the Kaiming initialization trained models start from)
Control
Pooled JEPA
As the JEPA, but predicts the pooled 128-d vector
Ablation
Static VICReg
Invariance between two copies of one frame under random shifts and square symmetries; never sees time
Control
Non-physical JEPA
As the JEPA, trained on the same initial fields evolved by random damped, phase-rotating linear dynamics on each Fourier mode, with a hidden rate per trajectory
Control
Table 2
Criterion
JEPA
Untrained
Needed
Outcome
S1 log enstrophy
0.930
0.727
0.864
counts: +0.204 [ +0.175 , +0.235 ]
S1 log palinstrophy
0.746
0.437
0.637
counts: +0.310 [ +0.254 , +0.366 ]
S1 log(1 + vortex count)
0.752
0.531
0.731
counts: +0.221 [ +0.178 , +0.275 ]
S1 Okubo–Weiss fraction (mandatory)
0.507
0.456
0.656
does not count: +0.051 [ +0.002 , +0.110 ]
S1 log dissipation
0.176
0.134
0.334
does not count: +0.042 [ −0.005 , +0.085 ]
S1
3 of 5, incl. Okubo–Weiss
fails (3 of 5, Okubo–Weiss not among them)
Table 1: Study 1 verdict, in-distribution test set, pooled latent, mean of 3 seeds, 95% paired trajectory-bootstrap intervals. The pre-registered verdict is not supported (C, primary).
Figure 1: Each model’s gain over the untrained control on every pre-registered S1–S3 quantity (test set, pooled latent): seeds, seed mean and stored 95% paired trajectory-bootstrap interval, against the gain each rule needed. The JEPA fails S1 (Okubo–Weiss), S2 and S3; static VICReg and the non-physical JEPA match or beat it (C).
Quantity
JEPA − non-physical JEPA
JEPA − static VICReg
JEPA − pooled JEPA
log enstrophy
−0.026 [ −0.034 , −0.020 ]
−0.043 [ −0.056 , −0.034 ]
+0.068 [ +0.056 , +0.082 ]
log palinstrophy
−0.055 [ −0.075 , −0.038 ]
−0.154 [ −0.192 , −0.124 ]
+0.081 [ +0.062 , +0.104 ]
log(1 + vortex count)
−0.037 [ −0.052 , −0.023 ]
−0.090 [ −0.113 , −0.068 ]
+0.115 [ +0.092 , +0.141 ]
Okubo–Weiss fraction
+0.003 [ −0.042 , +0.051 ]
−0.166 [ −0.233 , −0.109 ]
+0.084 [ +0.042 , +0.135 ]
log dissipation
−0.002 [ −0.031 , +0.029 ]
−0.015 [ −0.072 , +0.047 ]
+0.028 [ −0.006 , +0.061 ]
logν , frame-pair R2 (no CI)
−0.066
−0.242
+0.139
Table 2: Paired differences in probe R2 (and S3 ∣r∣ ), JEPA minus each comparison model; negative means the comparison model is better (C, descriptive).
Figure 2: Plain autoencoders that render the present match the imaginer’s enstrophy prominence; adding VICReg drops it to control level while raising linear decodability (E). The E4 panel is Figure B1 (Appendix B ).
Hypothesis
Rule
Result
Outcome
H1, velocity autoencoder
E1 ≥ 0.70; ≥ control + 0.25; CI > 0
E1 0.860; +0.480 [ +0.390 , +0.565 ]
pass
H1, vorticity autoencoder
same
E1 0.799; +0.418 [ +0.360 , +0.486 ]
pass
H1
both pass
holds
H2, velocity: plain − VICReg
≥ 0.20; CI > 0
+0.338 [ +0.289 , +0.390 ]
pass
H2, vorticity: plain − VICReg
same
+0.149 [ +0.105 , +0.193 ]
fail
H2
both pass
fails
Table 3: Study 2 confirmation on the sealed set; untrained control E1 0.380 [0.30, 0.46]. The pre-registered verdict is partially supported : H1 holds and H2 fails (C, primary; H3 and H4 secondary).
Figure 3: Enstrophy prominence E1 on the sealed Study 2 set (seeds, mean, 95% trajectory-bootstrap interval). Fresh seeds 3–5 decide H1 and H2; exploration seeds 0–2 are reported and carry H4 (C).
Figure 4: Of 84 candidate latent directions in 21 existing models, none moves the decoded field by the dial threshold; even the supervised shape direction (hollow markers), which predicts s almost perfectly, barely moves it. Dashed line: the 0.1 floor used at the time; dotted: the 0.2 floor committed afterwards. Effects below 10−4 are drawn in the shaded strip (E).
τ
Content shape R2
Decoder
Shape-knob σshape (median)
Both dials, all criteria
ω error
0
0.97
sub-pixel
0.07
0 of 3
0.015
0.2
0.61
resize
0.88
3 of 3
0.022
0.5
0.29
sub-pixel
0.93
0 of 3 (artefacts)
0.041
0.5
0.29
resize
0.96
3 of 3
0.039
1
0.18
sub-pixel
0.88
0 of 3
0.125
2
0.07
sub-pixel
0.93
0 of 3
0.374
Table 4: Shape-noise dose (exploration data, 3 seeds per row). “Content shape R2 ” is how well s can be read from the content branch’s input; “shape knob” is the loudness-null direction (E).
Seed
Loudness dial
Shape dial
Both
σshape loudness / shape, general
σshape loudness / shape, matched energy
What failed
100
yes
yes
yes
0.00 / 0.94
0.03 / 0.94
—
101
yes
yes
yes
0.00 / 0.98
0.04 / 0.78
—
102
yes
no
no
0.00 / 0.41
0.05 / 0.46
shape, both sets: effect, monotonicity, cycle correlation and gain, grid artefacts (3.5% / 4.7% of frames); envelope on matched energy
103
yes
yes
yes
0.00 / 1.00
0.05 / 0.81
—
104
yes
no
no
0.00 / 0.96
0.03 / 0.96
shape, matched energy only: envelope −0.09 (tolerance −0.05 )
Count
5 of 5
3 of 5
3 of 5
pass ≥ 4 → partially supported
Table 5: Study 3 confirmation, per fresh seed (C, primary). The exact 95% interval for a rate observed as 3 of 5 is 15–95%.
Figure 5: Both dials of the five fresh main models and one seed of each control, decoded on one exploration frame (the sealed sets are spent and are not reopened for plotting); ✓/✗ give each direction’s verdict on both sealed sets. Loudness scales the field; shape moves energy between scales; seed 102’s shape direction fails. All 11 models: Figure D1 (Appendix D ) (C).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity (in-distribution test, raw pooled latent)
Untrained
JEPA
Pixel
Pooled JEPA
Static VICReg
Non-physical JEPA
log enstrophy, R2
0.727
0.930
0.890
0.862
0.973
0.956
log palinstrophy, R2
0.437
0.746
0.733
0.665
0.900
0.802
log(1 + vortex count), R2
0.531
0.752
0.521
0.637
0.842
0.789
Okubo–Weiss fraction, R2
0.456
0.507
0.270
0.423
0.673
0.504
log dissipation, R2
0.134
0.176
0.156
0.148
0.191
0.178
logν , frame pair, R2
0.481
0.457
0.710
0.317
0.699
0.522
Appendix
Table A1: Means over 3 seeds. Probe rows are R2 after partialling out time × initial-condition family, except the ν row, which is not partialled; S3 rows are ∣r∣ . Seed standard deviations, the shift-averaged readout and MLP probes are in REPORT.md , appendix A1–A2. Applying the full pre-registered rules to every model against the control, only static VICReg meets S1 (4 of 5, including Okubo–Weiss), and only the pixel predictor meets S3; no model meets S2.
Figure B1: E4 =a/b , the energy–enstrophy mix of the enstrophy-tracking direction (Section 5.1 ), for the plain and VICReg autoencoders of Figure 2 on the exploration set: seeds and seed mean, log axis. E4 =1 means the direction tracks logZ0 alone; the dashed line is the untrained control (E).
Item
Value
Exploration data
1,006 trajectories (378 inviscid); matched-energy set of 300 inviscid trajectories at E0 = 0.2893
JEPA enstrophy ∣r∣ by readout: pooled / map mean ⊕ std / map PCA-512
0.353 / 0.303 / 0.295
Kill gate (a), exploration set: pixel vs untrained
0.829 vs 0.418, gap +0.411 [ +0.326 , +0.514 ]: pass
Kill gate (b), matched energy: pixel vs untrained
0.979 vs 0.177, gap +0.802 [ +0.718 , +0.827 ]: pass
Table B1: Exploratory (E). Horizon is confounded with training length, and the imaginer’s k=1 and k=16 cells rest on two seeds and one seed.
Figure C1: Steered fields at τ=0.5 from a sub-pixel decoder and from the resize-convolution decoder: same seed and frame, each at its worst steering amplitude, with the out-of-band power share above each panel (E).
Figure C2: As the content’s shape signal is degraded, the loudness-null direction becomes a shape knob; both dials pass only with the resize decoder; rendering error rises with dose. The curve mixes decoders: the τ=0.2 point and the second τ=0.5 point use the resize-convolution decoder, the rest the sub-pixel decoder (E).
Stage
What changed
What it showed
0
Measure 21 existing models
0 of 84 directions are dials (best effect 0.086); the supervised s -direction is readable ( ∣r∣ 0.97–0.99) but moves nothing
1
Channel-split latent, two render heads, FiLM variant
No dials; normalized content still carries loudness ( R2 0.99)
2
Separate branches, scale-free content
Loudness dial in 3 of 3 seeds (later withdrawn by the artefact criterion); the dial code holds s at R2 0.95 but steering it does not move s
2b
Whitened content
PCA shape dial in 2 of 3 seeds (1 of 3 after the artefact criterion); no clean loudness dial; rendering 3.8 times the VICReg autoencoder
A
No training: loudness by symmetry; grid-artefact criterion
Pure loudness dial in 2 of 3 seeds; earlier loudness dials fail on tails (checkerboards in 5–14% of frames at α=+2 )
A ′
No training: shape = top PC after projecting out loudness in the latent
50/50 mixtures ( σshape 0.50–0.54)
Appendix
Table C1: Exploratory (E). Several design and rule choices were made after seeing results; each stage’s predictions were committed before it ran.
Seed
Shape dial: Δs per unit α , exploration / matched
σshape
Loudness dial: Δa per unit α
σshape
Artefact frames
0
+0.43 / +0.46
0.99 / 0.95
+0.66 / +0.69
0.00 / 0.03
0%
1
+0.33 / +0.29
0.87 / 0.82
+0.53 / +0.50
0.00 / 0.02
0%
2
+0.35 / +0.33
0.88 / 0.86
−0.62 / −0.66
0.00 / 0.05
0%
Appendix
Table C2: Final exploration model ( τ=0.2 , resize decoder), per seed (E). The sign of a direction is arbitrary.
Figure D1: As Figure 5 , for all 11 scored models: main seeds 100–104, τ=0 seeds 200–202 and untrained seeds 300–302, decoded on the same exploration frame; ✓/✗ give each direction’s verdict on both sealed sets (C).
Item
Value
PC2 shape rule: seeds with both dials
3 of 5 (100, 102, 104)
PC2 failures
101: σshape 0.63 on the general set; 103: σshape 0.36 / 0.30 (mostly loudness)
∣cos∣ between the loudness-null and PC2 directions, seeds 100–104
0.99, 0.95, 0.35, 0.87, 0.96 (computed after the run)
τ=0 control: loudness dial; loudness-null and PC2 effects
3 of 3 (effect 0.78–0.85, σshape≤ 0.01); 0.02–0.10 and 0.12–0.23, all failing the cycle check
Untrained control
no dials; artefacts in 100% of frames; rendering error 1.9–16.2
Appendix
Table D1: Study 3, pre-registered secondary and descriptive results (C), except where marked.
Seed
cos2(g,PC2)
Fraction of PC2’s shape effect removed by projecting out g (linear / steered)
Variance after projection: top direction / along PC2
σd , loudness-null / PC2
100
0.009
+0.07 / +0.08
2.91 / 2.90
5.12 / 5.12
101
0.057
+0.17 / +0.18
2.44 / 2.43
5.22 / 5.16
102
0.000
−0.00 / −0.00
2.39 / 1.48
2.11 / 4.47
103
0.117
−0.66 / −0.70
2.32 / 2.17
5.34 / 5.05
104
0.001
+0.02 / +0.01
3.08 / 2.94
3.70 / 3.84
Appendix
Table E1: Post hoc (PH), exploration data only. g is the mean gradient of output loudness with respect to the z -scored dial code. In seed 103, projecting out g increases PC2’s shape effect, because there PC2 mixes in loudness that works against shape.
When
Event
Before Study 1
About 70 CPU planning pilots, summarized in the log, not reproducible from the repository
Study 1
Pre-registration committed before training; amendment A1 seven minutes later (no-collapse rule limited to collapse-prone objectives)
Study 1
Validation run crashed at its final console print after writing complete outputs; output-only fix
Study 1
Validation results seen before the test; afterwards only the encoding fix and an unscored traversal readout changed
Exploration
Stage 2 launch aborted by a line-ending bug before any training; restarted from scratch
Exploration
Budget raised four times (from 14 to 30 GPU-hours), each before the work it covered
Appendix
Table F1: Every entry is dated and tied to a commit in the released timeline ( results/paper/TIMELINE.md ).
Line of work
Logged hours
Study 1
11.5
Exploration and Study 2
29.6
Dial build (exploration)
≈ 17.8
Study 3 confirmation
≈ 8.8
Post hoc analysis
2.27
Total
≈ 70
Appendix
Table F2: Logged wall-clock hours on one NVIDIA RTX 3060 (12 GB), including CPU-bound phases within each line. About 70 CPU-only planning pilots before Study 1 are summarized in the log and not included.
Joint-Embedding Predictive Architectures (JEPAs) provide a powerful framework for latent world modeling and planning in a reconstruction-free manner. Although numerous JEPA-based approaches have been proposed to mitigate representation collapse, our experiments on localized, out-of-distribution visual noise reveal that performance degradation remains pronounced and unresolved. We propose Flow-JEPA (F-JEPA), a flow-based latent dynamics model that jointly generates a sequence of future latent states conditioned on the current observation and actions. A Gaussian distribution serves as the flow source, exposing the vector field to perturbed latent trajectories as it learns to transport them toward clean future representations. This formulation retains the reconstruction-free JEPA framework while switching from pointwise transition regression to stochastic trajectory-level prediction. F-JEPA raises mean success from 86% to 92% under clean observations and from 67% to 86% under noisy conditions. Further evaluations over varying perturbation severity and inference settings show that the performance advantage persists across a broad range of conditions. These results suggest that conditional flow matching provides a promising alternative to deterministic autoregressive prediction as a dynamics formulation in JEPA world models.
Yanchen Huo, Ziying Song, Yadan Luo
Nanyang Technological University · The University of Queensland
Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system's evolution in time. However, these emulators are computationally expensive to train and are subject to performance pitfalls, such as compounding errors during autoregressive rollout. In this work, we take a different perspective and look at scientific tasks further downstream of predicting the next frame, such as estimation of a system's governing physical parameters. Accuracy on these tasks offers a uniquely quantifiable glimpse into the physical relevance of the representations of these models. We evaluate the effectiveness of general-purpose self-supervised methods in learning physics-grounded representations that are useful for downstream scientific tasks. Surprisingly, we find that not all methods designed for physical modeling outperform generic self-supervised learning methods on these tasks, and methods that learn in the latent space (e.g., joint embedding predictive architectures, or JEPAs) outperform those optimizing pixel-level prediction objectives. Code is available at https://github.com/helenqu/physical-representation-learning.
Helen Qu, Rudy Morel, Michael McCabe +4
Flatiron Institute · Princeton University · Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM +1
Latent world models that integrate a flow in a frozen self supervised latent space train stably and cheaply, yet silently lose the property manipulation depends on most: motion. The pretrained flow never moves the manipulated object; retraining it with latent-only losses only trades stillness for teleport-like motion. We trace the failure to the training signal, not the representation: anchor-sparse, latent-only supervision never says where along the horizon change belongs. Decode-augmented rollout training (DART) repairs this while keeping the representation frozen, retraining only the flow with decode-path supervision. DART outperforms its latent only parent on the full protocol, restores the temporal structure of motion, and re-couples predicted motion to the scene; at larger scale it further improves prediction quality, closing nearly half the remaining gap to an oracle-informed interpolation reference. Finally, we report an unexpected finding about evaluation: pixel error alone rewards frozen predictions.
Xiwen Chen, Rigaudiere Z. Li, Zhiruo Zhou +2
Shenzhen International Graduate School, Tsinghua University · Shanghai Jiaotong University · Wuhan University of Technology