-LeWM: Learning World Models via S-Information for Robot Control
Organizations: Vrije Universiteit Amsterdam, The Netherlands
Abstract
World models have increasingly been applied as a model-based control method for robotics tasks such as manipulation and locomotion. While early world model architectures relied on decoders to prevent mode collapse and learn a predictable latent space, recent works on world models, such as Joint Embedding Predictive Architecture or LeWorldModel, increasingly aim to learn embeddings without a grounding decoder, utilising only a latent forward dynamics predictor and a stabilising entropy term. In this paper, we set out to demonstrate that the further introduction of two additional latent losses to LeWorldModel, a backward dynamics predictor and an inverse dynamics action predictor, enforces further structure on the latent embedding, leading to further gains in both performance and planning efficiency. These are critical properties for improving the applicability of such methods to real-time model-based control of real-world robots. We demonstrate these gains on standard world model benchmarks and two simulated robot manipulation tasks.
Figures & tables
| Push-T | TwoRoom | Reacher | Cube | Scene | |
|---|---|---|---|---|---|
| Random policy (floor) | 2.0 2.0 | 23.6 3.6 | 13.6 3.8 | 43.6 8.9 | 43.2 6.7 |
| LeWM [ 9 ] | 90.0 5.7 | 90.9 4.1 | 86.0 6.3 | 66.5 6.3 | 57.7 8.0 |
| LeWM + multi-step † | 88.9 3.3 | 92.8 4.2 | 83.6 4.4 | 71.0 5.3 | 53.6 5.9 |
| Sub-JEPA [ 16 ] | 84.0 8.7 | 93.6 2.8 | 80.7 5.9 | 70.3 6.6 | 55.5 8.1 |
| RC-aux [ 17 ] | 87.9 5.2 | 91.6 3.4 | 84.8 5.4 | 70.3 5.4 | 58.0 5.5 |
| SMWM [ 18 ] | 88.5 5.4 | 73.1 ‡ 38.9 | 84.5 5.8 | 81.9 4.9 | 60.0 3.1 |
| loss weight multiplier | |||
|---|---|---|---|
| inverse | 82 | 90 | 87 |
| backward | 86 | 90 | 82 |
| SIGReg | 88 | 90 | 76 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Block height | Gripper closed AUROC | Gripper closed Acc. | Gripper opening | Effector | Block | Effector block | |
|---|---|---|---|---|---|---|---|
| LeWM | 0.958 0.006 | 0.988 0.005 | 0.950 0.012 | 0.837 0.035 | 0.837 0.023 | 0.929 0.030 | 0.775 0.055 |
| LeWM + multi-step | 0.951 0.012 | 0.986 0.004 | 0.951 0.011 | 0.838 0.026 | 0.980 0.000 | 0.647 0.037 | 0.728 0.025 |
| Sub-JEPA | 0.755 0.022 | 0.980 0.005 | 0.944 0.005 | 0.761 0.026 | 0.929 0.005 | 0.898 0.013 | 0.887 0.006 |
| RC-aux | 0.958 0.005 | 0.988 0.006 | 0.954 0.009 | 0.840 0.035 | 0.843 0.026 | 0.933 0.026 | 0.783 0.062 |
| -LeWM | 0.978 0.005 | 0.988 0.007 | 0.963 0.014 | 0.888 0.027 | 0.997 0.000 | 0.846 0.016 | 0.900 0.014 |
| CEM30 | CEM8 | CEM4 | CEM2 | |
| Nominal budget reduction | 1 | 3.75 | 7.5 | 15 |
| NVIDIA GeForce RTX 3090 | ||||
| Solve latency (ms) | 692.2 7.9 | 191.0 1.9 | 99.5 0.9 | 53.8 0.7 |
| Replanning rate (Hz) | 1.44 0.02 | 5.21 0.05 | 9.97 0.09 | 18.31 0.22 |
| Measured speed-up | 1.00 | 3.61 0.02 | 6.91 0.05 | 12.69 0.10 |
| NVIDIA H100 | ||||
World Model Learning
World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.
See all 463 papers