Embodied systems experience the world as 'a symphony of flows': a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks. More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/
Figures & tables
Figure 1: Flow Equivariant World Models (FloWM) maintain structured dynamic memory in partially observed environments. When an agent observes dynamics, turns away, then turns back to the original viewpoint, flow equivariance asserts dynamics continue even when unobserved; existing work loses track of objects and eventually hallucinates new objects in their place.
Figure 2: Existing world model memory is inherently limited in partially observed dynamic environments. a) Standard autoregressive video diffusion evicts frames beyond the sliding window. b) Information dependencies between past observations and generated frames cause inconsistency without memory. c) Existing memory solutions are view-dependent (unstructured), and thus cannot predict dynamic scenes consistently. d) FloWM remembers past observations in a structured memory that is continually updated via internal dynamics.
Figure 3: Visualization of the Simple Recurrent FloWM on MNIST World. The world state (bottom) is windowed to produce an observation ft . The RNN processes this observation and embeds it in its hidden state composed of the ‘velocity channels’ (stacked vertically, of which there are three for visualization purposes). Self-motion and external motion flow equivariance are obtained by acting on the hidden state with the combination of the action flow ψ1(at) and each channel’s corresponding flow ψ1(ν) .
Figure 4: Transformer-Based FloWM . a) Image observation ft at time t and memory tokens ht within the current field of view ( FoV ) are passed through a ViT encoder Eθ . The latent map ht is fully learned, visualized as a map here for clarity. b) The update writes to the memory tokens at the correct FoV locations, then transforms the latent map according to known action at and internal flow ψ1(ν) , producing ht+1 . c) We decode using cross attention over the new FoV tokens of ht+1 with a ViT decoder Dθ to predict next image f^t+1 .
Figure 5: FloWM generalizes further and trains faster. a) Timesteps 0 to 49 are given as observations. Models are trained to predict up to t=69 . We see that FloWM rollouts do not diverge even at timestep 199, while baselines slowly degrade in image quality or lose track of the digits. b) MSE vs. rollout timestep shows length generalization. c) Learning efficiency of the FloWM vs. ablations.
Figure 6: Flow Equivariant World Models accurately predict moving objects in a 3D environment over long time-spans. a) Visualization of rollout (after 50 observation frames). FloWM stays consistent until the final frame, while the baselines and ablations hallucinate object position and color. b) Corresponding averaged MSE per timestep increases over time for baselines.
Figure 7: Downstream Planning on Block World. Distance to the red block using a simple MPC planning algorithm for different learned world models. The optimal distance for the reward function is at a distance of 2 away.
Figure 8: Ground truth block positions and trained probe predictions visualized for two of six blocks in a Dynamic Block World rollout (top down view). a) Probes on FloWM’s learned latent space easily recover the ground truth locations. b) Baseline (DFoT) predictions result in hallucinated block locations.
Model
MSE ↓
PSNR ↑
SSIM ↑
20
150
20
150
20
150
FloWM (Ours)
0.0005
0.0018
32.99
27.56
0.9900
0.9813
(no VC)
0.0041
0.0334
23.83
14.77
0.9576
0.7729
(no SME)
0.1234
0.1317
9.088
8.805
0.0366
0.0127
(no SME, no VC)
0.1233
0.1333
9.091
8.751
0.0374
0.0146
(+ action-concat)
0.1125
0.1359
9.491
8.669
0.0623
0.0149
Table 1: Rollout metrics on 2D MNIST World. Models are evaluated by conditioning on 50 context frames, then generating 20 (training length) and 150 (length generalization) future frames.
Model
MSE ↓
PSNR ↑
SSIM ↑
70
210
70
210
70
210
FloWM (Ours)
0.000603
0.001539
32.19
28.13
0.9673
0.9525
(no VC)
0.007615
0.009614
21.18
20.17
0.9045
0.8935
(no SME, no VC)
0.009579
0.012625
20.19
18.99
0.8782
0.8631
DFoT
0.011759
0.021684
19.30
16.64
0.9377
0.8885
DFoT-SSM
0.022616
0.022570
16.46
16.46
0.8877
0.8879
Table 2: Rollout metrics on 3D Dynamic Block World. Given 70 context frames, models are evaluated on generation of 70 (training objective) and 210 (length extrapolation) future prediction frames.
Model
Mean ∣ Dist - 2 ∣
Final ∣ Dist - 2 ∣
FloWM (Ours)
0.727
0.250
DreamerV3 RSSM
6.449
6.220
DFoT
5.571
4.138
Table 3: Planning Metrics on Block World. Agent distance from the optimal location for the “Find the red block” task.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Partially observable dynamic world modeling in 2D environments. Grayed out areas are not visible to the agent at time t . The agent moves its view each timestep via action at and must predict future states after an observation phase, conditioned on its own future action sequence.
Figure 10: Top Down Probe Visualization for All Colors on a test set rollout on Dynamic Block World. a) Here we visualize the FloWM Probe prediction of the absolute block position. The agent position is ground truth. The predicted block locations remain accurate as the agent moves around, demonstrating the learned flow equivariance of the model. b) DFoT predicts sporadic block positions, demonstrating it doesn’t maintain a principled representation of the environment and the objects within it.
Figure 11: Probe Prediction Through Time on a test set rollout on Dynamic Block World. a) Here we visualize the position prediction for the spatial axes through time. The FloWM model only has a slight error on some of the coordinates, and is able to correct itself. b) DFoT’s predictions through time do not appear to follow the ground positions closely.
Figure 12: Textured 3D Block World Rollouts. Here we visualize additional qualitative rollout results on the Textured 3D Block World split. Note the hallucinated objects in both DFoT and DFoT-SSM as time goes on, whereas FloWM remains consistent.
Data Subset
World Size
Observation Resolution
# Blocks
Block Velocity Range, x and y
Static Block World
15x15
128x128
6-10
0
Dynamic Block World
15x15
128x128
6-10
-1 to +1 (no diagonals)
Textured Dynamic Block World
15x15
128x128
6-10
-1 to +1 (no diagonals)
Appendix
Table 4: Generation parameters for Block World dataset subsets.
Model
MSE ↓
PSNR ↑
SSIM ↑
70
210
70
210
70
210
FloWM (Ours)
0.000826
0.000926
30.83
30.33
0.9388
0.9376
DFoT
0.005581
0.009050
22.53
20.43
0.8960
0.8577
DFoT-SSM
0.009658
0.012169
20.15
19.15
0.8581
0.8384
Appendix
Table 5: Rollout results on Textured 3D Dynamic Block World. Models are evaluated by generating 70 and 210 future frames conditioned on 70 context frames.
Model
MSE ↓
PSNR ↑
SSIM ↑
70
210
70
210
70
210
FloWM (Ours)
0.000860
0.000937
30.65
30.28
0.9675
0.9629
(no VC)
0.000789
0.000641
31.03
31.93
0.9800
0.9803
(no SME, no VC)
0.015230
0.015816
18.17
18.01
0.8586
0.8510
DFoT
0.011686
0.021069
19.32
16.76
0.9378
0.8910
DFoT-SSM
0.004763
0.007985
23.22
20.98
0.9743
0.9576
Appendix
Table 6: Rollout performance on 3D Static Block World with partial observability. Models are evaluated by generating 70 or 210 future frames, conditioned on 70 context frames.
Data Subset
Self-Motion
Dynamics
Partially Observable
dynamic_fo_no_sm
No
Yes
No
dynamic_fo
Yes
Yes
No
static_po
Yes
No
Yes
dynamic_po
Yes
Yes
Yes
Appendix
Table 7: MNIST world data subsets demonstrating scaling difficulty in self-motion, dynamics, and partial observability.
Data Subset
World Size
Window Size
# Digits
Self-motion Range
Digit Velocity Range, x and y
dynamic_fo_no_sm
32
32
3
0
-2 to +2
dynamic_fo
32
32
3
10
-2 to +2
static_po
50
32
5
10
0
dynamic_po
50
32
5
10
-2 to +2
Appendix
Table 8: Generation parameters for MNIST World dataset subsets.
Figure 13: Rollout Error (MSE) vs. Forward Prediction Steps for all data subsets of MNIST World. The dynamic subset is replicated from the main text for ease of comparison.
Model
MSE ↓
PSNR ↑
SSIM ↑
20
150
20
150
20
150
FloWM (Ours)
0.00009
0.00104
40.61
29.83
0.9982
0.9752
(no SME)
0.00006
0.00173
41.89
27.61
0.9981
0.9797
(no SME, no VC)
0.00031
0.02845
35.05
15.46
0.9950
0.7746
(no VC)
0.00026
0.00925
35.87
20.34
0.9877
0.9061
(action concat)
0.00015
0.01349
38.36
18.70
0.9973
0.8099
Appendix
Table 9: Rollout performance on 2D Dynamic MNIST World with full observability and without self-motion, dynamic_fo_no_sm . Models are trained to generate 20 future frames given 50 context frames, and evaluated by generating 20 (training length) and 150 (length generalization) future frames respectively, conditioned on 50 context frames.
Model
MSE ↓
PSNR ↑
SSIM ↑
20
150
20
150
20
150
FloWM (Ours)
0.00013
0.00136
39.01
28.65
0.9977
0.9763
(no SME)
0.15595
0.16167
8.07
7.91
0.0355
0.0105
(no SME, no VC)
0.15597
0.18251
8.07
7.39
0.0351
0.0141
(no VC)
0.00010
0.26278
40.21
5.80
0.9982
0.1718
(action concat)
0.13425
0.15148
8.72
8.20
0.0796
0.0294
Appendix
Table 10: Rollout performance on 2D Dynamic MNIST World with full observability, dynamic_fo . Models are trained to generate 20 future frames given 50 context frames, and evaluated by generating 20 (training length) and 150 (length generalization) future frames respectively, conditioned on 50 context frames.
Model
MSE ↓
PSNR ↑
SSIM ↑
20
150
20
150
20
150
FloWM (Ours)
0.00013
0.00467
38.78
23.31
0.9970
0.8053
(no SME)
0.12097
0.12790
9.17
8.93
0.0460
0.0116
(no SME, no VC)
0.12086
0.13046
9.18
8.85
0.0520
0.0144
(no VC)
0.00002
0.00004
47.63
44.49
0.9994
0.9993
(action concat)
0.09473
0.11860
10.24
9.26
0.1595
0.1354
Appendix
Table 11: Rollout performance on 2D Static MNIST World with partial observability, static_po . Models are trained to generate 20 future frames given 50 context frames, and evaluated by generating 20 (training length) and 150 (length generalization) future frames respectively, conditioned on 50 context frames.
Component
Option
Value
Training
Learning rate
1e-4
Effective batch size
16
Training steps
150k
GPU usage
2 × H100
Teacher Forcing
0.125
Model
Hidden channels
256
Appendix
Table 12: Block World ViT FloWM Configurations.
Model
Forward Pass GFLOPs
Backward Pass GFLOPs
Batch Size
Steps
Total compute (EFLOPs)
FloWM (Ours)
31.18
99.50
16
150k
27.5
DFoT
4.94
9.91
32
300k
15.7
DFoT-SSM
5.00
10.03
32
300k
10.5
Appendix
Table 13: Training compute on 3D Dynamic Block World. FLOPs are measured for the forward pass and backward pass with batch size 1 and with 2 frames as reported in the first two columns. To compute the total training compute, we measure FLOPs at batch size 1 and with 140 frames, then scale by the actual batch size and number of training steps.
Model
Model size (M params)
Frames
Context
Rollout time (s)
Rollout throughput (generated frames/s)
Resident alloc (GiB)
Peak Delta (GiB)
FloWM
12.8
2
1
0.015
522.6
0.190
0.314
DFoT
95.3
2
1
0.423
18.9
0.802
0.024
DFoT-SSM
97.8
2
1
0.801
10.0
0.810
0.393
FloWM
12.8
140
70
0.967
579.0
0.394
0.444
DFoT
95.3
140
70
3.353
167.0
1.024
1.698
DFoT-SSM
97.8
140
70
2.402
233.2
1.016
1.697
Appendix
Table 14: Inference scaling of rollout time and memory usage based on number of generated and context frames. We measure the following on 1 H200 GPU with a batch size of 8. FloWM has the highest throughput and sublinear memory scaling with the number of generated and context frames.
FloWM setting
Map grid
FoV
FoV fraction
Decoder KV tokens
Map-update sequence tokens
Rollout time (s)
Rollout throughput (generated frames/s)
Resident alloc (GiB)
Peak Delta (GiB)
Default
33×33
60∘
0.163
177
241
0.967
579.0
0.394
0.444
Half map size
17×17
60∘
0.183
53
117
0.606
923.9
0.394
0.317
Larger FoV
33×33
90∘
0.265
289
353
1.514
369.8
0.395
0.467
Appendix
Table 15: Scaling of compute time and memory usage with FloWM map parameter choices. With different latent map sizes and Field of View (FoV) choices for FloWM, we can see reasonable scaling of throughput and memory usage.
Model
MSE ↓
PSNR ↑
SSIM ↑
70
210
70
210
70
210
FloWM (Ours)
0.000603
0.001539
32.19
28.13
0.9673
0.9525
DreamerV3 RSSM (50M)
0.01844
0.01881
17.34
17.26
0.8696
0.8673
DreamerV3 RSSM (400M)
0.01636
0.01647
17.86
17.83
0.8799
0.8782
Appendix
Table 16: Additional model parameter size ablation on the RSSM baseline, on the 3D Dynamic Block World dataset. Given 70 context frames, models are evaluated on generation of 70 (training objective) and 210 (length extrapolation) future prediction frames.
Table 30
Config
MnistWorld
Block World
Training
Effective batch size
128
32
Learning rate
2e-4 (linear warmup)
2e-4 (linear warmup)
Warmup steps
2,000
2,000
Weight decay
1e-3
1e-3
Training steps
245k
300k
Appendix
Table 19: DFoT configurations for different datasets. Section and key are organized hierarchically in the first column.
Config
MnistWorld
Block World
Training
Effective batch size
128
32
Learning rate
2e-4 (linear warmup)
2e-4 (linear warmup)
Warmup steps
2,000
2,000
Weight decay
1e-3
1e-3
Training steps
200k
300k
Appendix
Table 20: DFoT-SSM configurations. Classifier-free guidance ( Ho and Salimans, 2022 ) for conditioning is not used during inference; though the models have been trained to allow for it, we find their instruction following ability not to be a limiting factor. Loss weighting uses sigmoid reweighting proposed by Kingma and Gao (2023) and adopted by Hoogeboom et al. (2025) . History guidance follows the stabilized conditional method (level = 0.02) from Song et al. (2025) ; please refer to their codebase for details.
Component
Option
Value
Training
Learning rate
4e-4
Effective batch size
256
Precision
Float16 mixed precision
Strategy
Distributed Data Parallel
Warmup steps
10,000
Training epochs
172
Appendix
Table 21: VAE configurations for MNIST World. The input size from the dataset is 32 × 32.
Component
Option
Value
Training
Learning rate
4e-4
Effective batch size
256
Precision
Float16 mixed precision
Strategy
Distributed Data Parallel
Warmup steps
10,000
Training epochs
40
Appendix
Table 22: VAE configurations for Block World. The input size from the dataset is 128 × 128.
Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.
World modeling requires forecasting uncertain futures while preserving information useful for downstream perception. Existing visual world models often struggle to satisfy both goals: VAE-based stochastic models operate in low-dimensional reconstruction latents, which can limit perception performance, while deterministic predictors using strong pretrained features collapse multimodal futures into a single blurry mean. In this work, we propose FlowWM, a stochastic world model that performs flow matching directly within pretrained feature space (e.g., DINOv3). This is challenging because pretrained features are substantially high-dimensional, making standard diffusion recipes suboptimal. To address this, we investigate the design choices needed for feature-space flow matching and introduce a differentiable one-step projection mechanism that enables efficient training with temporal consistency and task-driven objectives. We evaluate FlowWM on two benchmarks: a synthetic benchmark for systematic evaluation of accuracy and diversity, and a real-world benchmark FuturePerception. FlowWM improves perception performance, mode coverage, and horizon robustness, validating our proposed design for stochastic world modeling in high-dimensional feature spaces.
Francois Porcher, Nicolas Carion, Karteek Alahari +1
World models learn to predict future states of an environment, enabling planning and mental simulation. Current approaches default to Transformer-based predictors operating in learned latent spaces. This comes at a cost: O(N^2) computation and no explicit spatial inductive bias. This paper asks a foundational question: is self-attention necessary for predictive world modeling, or can alternative computational substrates achieve comparable or superior results? I introduce FluidWorld, a proof-of-concept world model whose predictive dynamics are governed by partial differential equations (PDEs) of reaction-diffusion type. Instead of using a separate neural network predictor, the PDE integration itself produces the future state prediction. In a strictly parameter-matched three-way ablation on unconditional UCF-101 video prediction (64x64, ~800K parameters, identical encoder, decoder, losses, and data), FluidWorld is compared against both a Transformer baseline (self-attention) and a ConvLSTM baseline (convolutional recurrence). While all three models converge to comparable single-step prediction loss, FluidWorld achieves 2x lower reconstruction error, produces representations with 10-15% higher spatial structure preservation and 18-25% more effective dimensionality, and critically maintains coherent multi-step rollouts where both baselines degrade rapidly. All experiments were conducted on a single consumer-grade PC (Intel Core i5, NVIDIA RTX 4070 Ti), without any large-scale compute. These results establish that PDE-based dynamics, which natively provide O(N) spatial complexity, adaptive computation, and global spatial coherence through diffusion, are a viable and parameter-efficient alternative to both attention and convolutional recurrence for world modeling.