JointGV: Structure-Aware Correction of Visual World Models with Parallel Graph Dynamics for Long-Horizon Modeling
Organizations: University of Oxford · University of Electronic Science and Technology of China · Monash University · University College London, University of London ·
Abstract
Visual world models predict future visual states conditioned on actions, but recursive errors in entity poses and spatial relations limit their long-horizon prediction for robot planning and control. We ask whether a separately predicted relational state can correct this drift without restructuring the visual model. This is challenging because graph nodes and visual states share neither elements nor a feature space. We propose Joint Graph--Visual Dynamics Correction (JointGV), a plug-in based on a cross-granularity dynamics hypothesis: despite their different representations, graph-node and visual-state changes describe the same physical transition. JointGV rolls out a task-conditioned scene graph alongside the visual world model, couples their predicted changes, and combines the resulting graph-motion context with graph-predicted node positions to compute a bounded residual. Graph-learning and correction gradients do not propagate into the original visual dynamics. We attach JointGV to DINO-WM (spatial tokens) and DreamerV3 (a global latent state), evaluating both on DMC Cheetah Run and Walker Walk across three seeds and extending the evaluation to contact-rich manipulation with DMC Finger Turn Hard. Across 32-step rollouts, JointGV improves structural and visual-prediction accuracy, reducing joint-position error by 11--28%. For spatial tokens, learned graph--visual change coupling gives lower error than broadcasting the same mean graph-motion context to every token, particularly when non-adjacent parts appear close together in the image. On real-robot inverted-pendulum sequences, JointGV reduces DINO-WM's 32-step pole-angle error from to and improves DreamerV3 and LeWorldModel as well. In simulated inverted-pendulum control, an actor trained with JointGV-corrected rollouts completes 46 of 50 episodes, compared with none using the original rollouts.
Figures & tables
| Joint-position error (px) | Token MSE | RGB MSE | SSIM | LPIPS | ||||
| Method | Yr. | |||||||
| Longer visual training | ||||||||
| DINO-WM ( ) [ 34 ] | 2025 | 0.830 | 0.931 | 1.036 | 0.456 | 2.31 | 0.841 | 0.0435 |
| DINO-WM + JointGV ( ) | Ours | 0.730 | 0.796 | 0.852 | 0.341 | 1.84 | 0.861 | 0.0351 |
| DINO-WM ( ) | 2025 | 0.882 | 0.940 | 0.957 | 0.339 | 2.13 | 0.853 | 0.0420 |
| DINO-WM + JointGV ( ) | Ours | 0.750 | 0.815 | 0.863 | 0.314 | 1.80 | 0.863 | 0.0352 |
| Visual world model | Visual state | Task | Original | + JointGV | Reduction (%) |
|---|---|---|---|---|---|
| DINO-WM | Spatial tokens | Cheetah Run | |||
| DreamerV3 | Global latent | Cheetah Run | |||
| DINO-WM | Spatial tokens | Walker Walk | |||
| DreamerV3 | Global latent | Walker Walk |
| Joint-position error (px) | |||
|---|---|---|---|
| Method | |||
| DINO-WM backbone [ 34 ] | 10.813 | 12.476 | 14.861 |
| DINO-WM + JointGV | 9.444 | 9.911 | 10.754 |
| Measure (50 unseen initial states) | Original | + JointGV |
|---|---|---|
| Six-second completion | 0/50 | 46/50 |
| Mean time before termination (s) | 2.28 | 5.86 |
| Shared reward-head MSE | 0.0851 | 0.0375 |
| Joint-position error (px) | DINOv2 token MSE | RGB MSE | SSIM | LPIPS | |||
| Output | |||||||
| Detached plug-in: corrected output ( ) | 0.730 | 0.796 | 0.852 | 0.341 | 1.84 | 0.861 | 0.0351 |
| Full gradient coupling: original visual output ( ) | 0.992 | 1.184 | 1.340 | 0.903 | 2.96 | 0.786 | 0.0592 |
| Full gradient coupling: corrected output ( ) | 0.738 | 0.812 | 0.882 | 0.381 | 1.97 | 0.853 | 0.0374 |
| Partial gradient coupling: original visual output ( ) | 0.925 | 1.039 | 1.153 | 0.785 | 2.58 | 0.813 | 0.0532 |
| Partial gradient coupling: corrected output ( ) | 0.771 | 0.838 | 0.890 | 0.449 | 1.99 | 0.846 | 0.0364 |
| Correction variant | Own backbone | Corrected | Reduction |
|---|---|---|---|
| Residual controls | |||
| Visual-only residual | 0.8978 | 0.8935 | 0.0043 |
| Visual-only residual + node supervision | 0.8978 | 0.8830 | 0.0148 |
| Graph-conditioned correction | |||
| Mean graph-motion broadcast | 0.8978 | 0.8146 | 0.0832 |
| Cosine change coupling | 0.9021 | 0.7934 | 0.1087 |
| Method | Seed | Backbone | Corrected | Reduction | 95% CI |
|---|---|---|---|---|---|
| JointGV | 0 | 0.8978 | 0.7906 | 11.95% | [11.08, 12.72]% |
| JointGV | 1 | 0.9377 | 0.7790 | 16.92% | [16.17, 17.70]% |
| JointGV | 2 | 0.9899 | 0.8001 | 19.18% | [18.20, 20.10]% |
| JointGV | Mean std | 0.9418 0.0462 | 0.7899 0.0106 | 16.01 3.70% | – |
| Visual-only residual | 0 | 0.8978 | 0.8935 | 0.49% | [-0.27, 1.23]% |
| Visual-only residual | 1 | 0.9516 | 0.9194 | 3.38% | [2.46, 4.35]% |
| Model / task | Output | Joint-position error (px) | RGB MSE | SSIM | LPIPS | ||
|---|---|---|---|---|---|---|---|
| DINO-WM / Cheetah Run | Original | 0.830 | 0.931 | 1.036 | 2.31 | 0.841 | 0.0435 |
| + JointGV | 0.730 | 0.796 | 0.852 | 1.84 | 0.861 | 0.0351 | |
| DreamerV3 / Cheetah Run | Original | 0.378 | 0.427 | 0.470 | 0.978 | 0.929 | 0.0178 |
| + JointGV | 0.355 | 0.394 | 0.425 | 0.798 | 0.939 | 0.0153 | |
| DINO-WM / Walker Walk | Original | 7.807 | 9.626 | 10.953 | 13.52 | 0.568 | 0.2705 |
| Paired difference (px) | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| Minimum distance between non-adjacent nodes (Q1: closest) | ||||
| Cheetah Run: mean broadcast OT | ||||
| Walker Walk: JointGV original | ||||
| Graph-predicted position error (Q1: most accurate) | ||||
| Cheetah Run: mean broadcast OT | ||||
| Walker Walk: JointGV original | ||||
| Joint-position error (px) | Token MSE | RGB MSE | SSIM | LPIPS | |||
|---|---|---|---|---|---|---|---|
| Method | |||||||
| DINO-WM backbone [ 34 ] | 10.813 | 12.476 | 14.861 | 2.204 | 11.29 | 0.633 | 0.3673 |
| DINO-WM + JointGV | 9.444 | 9.911 | 10.754 | 0.823 | 5.44 | 0.761 | 0.2444 |
| Model | Output | Joint (px) | Pole (deg) | RGB MSE |
|---|---|---|---|---|
| DINO-WM | Original | 2.590 | 12.564 | 10.659 |
| + JointGV | 2.124 | 6.689 | 8.067 | |
| DreamerV3 | Original | 1.137 | 3.805 | 4.342 |
| + JointGV | 0.894 | 2.825 | 2.217 | |
| LeWorldModel [ 13 ] | Original | 1.576 | 4.823 | 5.805 |
| + JointGV | 1.442 | 4.242 | 5.055 |
| Hardware: seven paired trials | Original | + JointGV |
|---|---|---|
| Angle-triggered stops | 7/7 | 7/7 |
| Mean recorded control window (s) | 0.85 | 0.98 |
World Models for Robotics
World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.
See all 426 papers