Flow Equivariant World Models: Structured Memory for Dynamic Environments
Organizations: Kempner Institute, Harvard University · CSE, UC San Diego · ML, Carnegie Mellon University · SEAS, Harvard University.
Abstract
Embodied systems experience the world as 'a symphony of flows': a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks. More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/
Figures & tables
| Model | MSE | PSNR | SSIM | |||
| 20 | 150 | 20 | 150 | 20 | 150 | |
| FloWM (Ours) | 0.0005 | 0.0018 | 32.99 | 27.56 | 0.9900 | 0.9813 |
| (no VC) | 0.0041 | 0.0334 | 23.83 | 14.77 | 0.9576 | 0.7729 |
| (no SME) | 0.1234 | 0.1317 | 9.088 | 8.805 | 0.0366 | 0.0127 |
| (no SME, no VC) | 0.1233 | 0.1333 | 9.091 | 8.751 | 0.0374 | 0.0146 |
| (+ action-concat) | 0.1125 | 0.1359 | 9.491 | 8.669 | 0.0623 | 0.0149 |
| Model | MSE | PSNR | SSIM | |||
| 70 | 210 | 70 | 210 | 70 | 210 | |
| FloWM (Ours) | 0.000603 | 0.001539 | 32.19 | 28.13 | 0.9673 | 0.9525 |
| (no VC) | 0.007615 | 0.009614 | 21.18 | 20.17 | 0.9045 | 0.8935 |
| (no SME, no VC) | 0.009579 | 0.012625 | 20.19 | 18.99 | 0.8782 | 0.8631 |
| DFoT | 0.011759 | 0.021684 | 19.30 | 16.64 | 0.9377 | 0.8885 |
| DFoT-SSM | 0.022616 | 0.022570 | 16.46 | 16.46 | 0.8877 | 0.8879 |
| Model | Mean Dist - 2 | Final Dist - 2 |
| FloWM (Ours) | 0.727 | 0.250 |
| DreamerV3 RSSM | 6.449 | 6.220 |
| DFoT | 5.571 | 4.138 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Data Subset | World Size | Observation Resolution | # Blocks | Block Velocity Range, x and y |
| Static Block World | 15x15 | 128x128 | 6-10 | 0 |
| Dynamic Block World | 15x15 | 128x128 | 6-10 | -1 to +1 (no diagonals) |
| Textured Dynamic Block World | 15x15 | 128x128 | 6-10 | -1 to +1 (no diagonals) |
| Model | MSE | PSNR | SSIM | |||
| 70 | 210 | 70 | 210 | 70 | 210 | |
| FloWM (Ours) | 0.000826 | 0.000926 | 30.83 | 30.33 | 0.9388 | 0.9376 |
| DFoT | 0.005581 | 0.009050 | 22.53 | 20.43 | 0.8960 | 0.8577 |
| DFoT-SSM | 0.009658 | 0.012169 | 20.15 | 19.15 | 0.8581 | 0.8384 |
| Model | MSE | PSNR | SSIM | |||
| 70 | 210 | 70 | 210 | 70 | 210 | |
| FloWM (Ours) | 0.000860 | 0.000937 | 30.65 | 30.28 | 0.9675 | 0.9629 |
| (no VC) | 0.000789 | 0.000641 | 31.03 | 31.93 | 0.9800 | 0.9803 |
| (no SME, no VC) | 0.015230 | 0.015816 | 18.17 | 18.01 | 0.8586 | 0.8510 |
| DFoT | 0.011686 | 0.021069 | 19.32 | 16.76 | 0.9378 | 0.8910 |
| DFoT-SSM | 0.004763 | 0.007985 | 23.22 | 20.98 | 0.9743 | 0.9576 |
| Data Subset | Self-Motion | Dynamics | Partially Observable |
| dynamic_fo_no_sm | No | Yes | No |
| dynamic_fo | Yes | Yes | No |
| static_po | Yes | No | Yes |
| dynamic_po | Yes | Yes | Yes |
| Data Subset | World Size | Window Size | # Digits | Self-motion Range | Digit Velocity Range, x and y |
| dynamic_fo_no_sm | 32 | 32 | 3 | 0 | -2 to +2 |
| dynamic_fo | 32 | 32 | 3 | 10 | -2 to +2 |
| static_po | 50 | 32 | 5 | 10 | 0 |
| dynamic_po | 50 | 32 | 5 | 10 | -2 to +2 |
| Model | MSE | PSNR | SSIM | |||
| 20 | 150 | 20 | 150 | 20 | 150 | |
| FloWM (Ours) | 0.00009 | 0.00104 | 40.61 | 29.83 | 0.9982 | 0.9752 |
| (no SME) | 0.00006 | 0.00173 | 41.89 | 27.61 | 0.9981 | 0.9797 |
| (no SME, no VC) | 0.00031 | 0.02845 | 35.05 | 15.46 | 0.9950 | 0.7746 |
| (no VC) | 0.00026 | 0.00925 | 35.87 | 20.34 | 0.9877 | 0.9061 |
| (action concat) | 0.00015 | 0.01349 | 38.36 | 18.70 | 0.9973 | 0.8099 |
| Model | MSE | PSNR | SSIM | |||
| 20 | 150 | 20 | 150 | 20 | 150 | |
| FloWM (Ours) | 0.00013 | 0.00136 | 39.01 | 28.65 | 0.9977 | 0.9763 |
| (no SME) | 0.15595 | 0.16167 | 8.07 | 7.91 | 0.0355 | 0.0105 |
| (no SME, no VC) | 0.15597 | 0.18251 | 8.07 | 7.39 | 0.0351 | 0.0141 |
| (no VC) | 0.00010 | 0.26278 | 40.21 | 5.80 | 0.9982 | 0.1718 |
| (action concat) | 0.13425 | 0.15148 | 8.72 | 8.20 | 0.0796 | 0.0294 |
| Model | MSE | PSNR | SSIM | |||
| 20 | 150 | 20 | 150 | 20 | 150 | |
| FloWM (Ours) | 0.00013 | 0.00467 | 38.78 | 23.31 | 0.9970 | 0.8053 |
| (no SME) | 0.12097 | 0.12790 | 9.17 | 8.93 | 0.0460 | 0.0116 |
| (no SME, no VC) | 0.12086 | 0.13046 | 9.18 | 8.85 | 0.0520 | 0.0144 |
| (no VC) | 0.00002 | 0.00004 | 47.63 | 44.49 | 0.9994 | 0.9993 |
| (action concat) | 0.09473 | 0.11860 | 10.24 | 9.26 | 0.1595 | 0.1354 |
| Component | Option | Value |
| Training | Learning rate | 1e-4 |
| Effective batch size | 16 | |
| Training steps | 150k | |
| GPU usage | 2 H100 | |
| Teacher Forcing | 0.125 | |
| Model | Hidden channels | 256 |
| Model | Forward Pass GFLOPs | Backward Pass GFLOPs | Batch Size | Steps | Total compute (EFLOPs) |
| FloWM (Ours) | 31.18 | 99.50 | 16 | 150k | 27.5 |
| DFoT | 4.94 | 9.91 | 32 | 300k | 15.7 |
| DFoT-SSM | 5.00 | 10.03 | 32 | 300k | 10.5 |
| Model | Model size (M params) | Frames | Context | Rollout time (s) | Rollout throughput (generated frames/s) | Resident alloc (GiB) | Peak Delta (GiB) |
| FloWM | 12.8 | 2 | 1 | 0.015 | 522.6 | 0.190 | 0.314 |
| DFoT | 95.3 | 2 | 1 | 0.423 | 18.9 | 0.802 | 0.024 |
| DFoT-SSM | 97.8 | 2 | 1 | 0.801 | 10.0 | 0.810 | 0.393 |
| FloWM | 12.8 | 140 | 70 | 0.967 | 579.0 | 0.394 | 0.444 |
| DFoT | 95.3 | 140 | 70 | 3.353 | 167.0 | 1.024 | 1.698 |
| DFoT-SSM | 97.8 | 140 | 70 | 2.402 | 233.2 | 1.016 | 1.697 |
| FloWM setting | Map grid | FoV | FoV fraction | Decoder KV tokens | Map-update sequence tokens | Rollout time (s) | Rollout throughput (generated frames/s) | Resident alloc (GiB) | Peak Delta (GiB) |
| Default | 0.163 | 177 | 241 | 0.967 | 579.0 | 0.394 | 0.444 | ||
| Half map size | 0.183 | 53 | 117 | 0.606 | 923.9 | 0.394 | 0.317 | ||
| Larger FoV | 0.265 | 289 | 353 | 1.514 | 369.8 | 0.395 | 0.467 |
| Model | MSE | PSNR | SSIM | |||
| 70 | 210 | 70 | 210 | 70 | 210 | |
| FloWM (Ours) | 0.000603 | 0.001539 | 32.19 | 28.13 | 0.9673 | 0.9525 |
| DreamerV3 RSSM (50M) | 0.01844 | 0.01881 | 17.34 | 17.26 | 0.8696 | 0.8673 |
| DreamerV3 RSSM (400M) | 0.01636 | 0.01647 | 17.86 | 17.83 | 0.8799 | 0.8782 |
| Config | MnistWorld | Block World |
| Training | ||
| Effective batch size | 128 | 32 |
| Learning rate | 2e-4 (linear warmup) | 2e-4 (linear warmup) |
| Warmup steps | 2,000 | 2,000 |
| Weight decay | 1e-3 | 1e-3 |
| Training steps | 245k | 300k |
| Config | MnistWorld | Block World |
| Training | ||
| Effective batch size | 128 | 32 |
| Learning rate | 2e-4 (linear warmup) | 2e-4 (linear warmup) |
| Warmup steps | 2,000 | 2,000 |
| Weight decay | 1e-3 | 1e-3 |
| Training steps | 200k | 300k |
| Component | Option | Value |
| Training | Learning rate | 4e-4 |
| Effective batch size | 256 | |
| Precision | Float16 mixed precision | |
| Strategy | Distributed Data Parallel | |
| Warmup steps | 10,000 | |
| Training epochs | 172 |
| Component | Option | Value |
| Training | Learning rate | 4e-4 |
| Effective batch size | 256 | |
| Precision | Float16 mixed precision | |
| Strategy | Distributed Data Parallel | |
| Warmup steps | 10,000 | |
| Training epochs | 40 |