Latent world models enable visual control by planning action sequences in compact representation spaces, but a gap remains between how latent dynamics are learned and how they are used for planning. First, teacher-forced training does not expose the dynamics model to the recursively predicted latent states encountered during planning. Second, sampling-based planners typically initialize action search from goal-agnostic proposals, without exploiting the action-conditioned dynamics learned by the world model. We introduce Dynamics-Informed Action Learning (DIAL) to bridge these two gaps. During training, rollout alignment exposes the forward model to its own autoregressive latent predictions, while multi-step inverse supervision trains a lightweight inverse dynamics head to infer actions from latent-state transitions. We further apply the inverse head to model-generated transitions with its parameters held fixed, encouraging predicted rollouts to retain action-relevant information. At test time, DIAL autoregressively couples the inverse and forward dynamics to construct a dynamics-informed action sequence toward the goal, which initializes the mean of the Cross-Entropy Method (CEM) proposal distribution. The subsequent CEM search then refines this initialization using the standard sampling and updating procedure. Experiments across four visual control benchmarks demonstrate that DIAL outperforms existing latent world models and planning baselines in both standard and long-horizon scenarios.
Figures & tables
Figure 1: DIAL aligns latent dynamics learning with goal-directed planning. (a) During training, teacher forcing performs one-step prediction from observed latent states. (b) At planning time, autoregressive rollouts can accumulate prediction drift, while generic action proposals lack goal-directed initialization. (c) DIAL addresses both gaps by training on model-generated rollouts and using learned inverse dynamics to construct a dynamics-informed CEM proposal.
Figure 2: Overview of DIAL. (1) Rollout alignment trains the latent dynamics predictor Fθ on autoregressively generated latent contexts to reduce the mismatch between training and planning. (2) Multi-step inverse supervision trains the inverse-dynamics head Gψ on observed latent state pairs and uses a frozen inverse head Gψˉ to encourage model-generated states to preserve action-relevant information. (3) Dynamics-informed planning autoregressively couples Gψ and Fθ to generate a goal-directed action sequence, which initializes only the mean of the CEM proposal.
Method
Offset 25
Offset 50
PushT
Cube
Reacher
TwoRoom
Macro
PushT
Cube
Reacher
TwoRoom
Macro
PLDM [ 19 , 20 ]
78.0
65.0
78.0
97.0
79.5
20.0
46.0
66.0
78.0
52.5
DINO-WM [ 29 ]
74.0
86.0
79.0
100.0
84.8
14.0
46.0
20.0
94.0
43.5
LeWM [ 12 ]
96.0
74.0
86.0
87.0
85.8
60.0
64.0
72.0
56.0
63.0
RC-aux [ 11 ]
90.8
76.0
87.2
98.0
88.0
64.0
54.0
82.0
54.0
63.5
Fast-LeWM [ 3 ]
96.0
80.0
88.0
98.0
90.5
46.0
58.0
80.0
50.0
58.5
Table 1: Success rates (%) at the standard 25-step and long-horizon 50-step goal offsets. Offset-25 results are quoted from the corresponding papers unless otherwise noted. Offset-50 results are evaluated using a unified harness with identical episodes ( n=50 ) and the same CEM candidate budget across methods. Bold and underline indicate the best and second-best results.
Figure 3: Visualization of planning trajectories on Cube and PushT. Compared with LeWM and INTACT, DIAL maintains object interaction and makes greater progress toward the goal, while other methods either lose object interaction or terminate far from target.
Figure 4: Effect of goal consistency in proposal initialization. Success rates (%) for uninitialized, mismatched-goal, and matched-goal proposals across environments and goal offsets under the constrained search budget ( 30×10 , n=50 episodes per setting).
Figure 5: Dynamics-informed initialization under constrained budgets. Success rates (%) across environments for uninitialized and dynamics-informed proposals under different goal offsets and search budgets. Full and low budgets correspond to 300×30 and 30×10 candidate rollouts.
LWM
LRA
Lobs
Lroll
PushT
Cube
Reacher
TwoRoom
Macro
✓
96.0
72.0
80.0
84.0
83.0
✓
✓
94.0
70.0
88.0
82.0
83.5
✓
✓
90.0
80.0
90.0
80.0
85.0
✓
✓
✓
94.0
84.0
82.0
86.0
86.5
✓
✓
✓
✓
96.0
84.0
92.0
86.0
89.5
Table 2: Loss ablation at the 25-step goal offset under unseeded CEM. Each variant is trained from scratch with identical architecture and optimization schedule, and evaluated with the full 300×30 budget. Success rates (%) are reported for different combinations of training objectives.
component
hyperparameter
value
input
image resolution / patch size
224×224 / 14
representation
latent dimension D
192
context
history h / rollout depth L
3 / 5
predictor
layers / heads / MLP width
6 / 16 / 2048
residual branch
layers / bottleneck rank
3 / 32
inverse head
depth / hidden width
2 / 256
Table 3: DIAL training hyperparameters. Values are shared across PushT, Cube, Reacher, and TwoRoom unless noted otherwise.
r
warm start
TwoRoom
PushT
Cube
Reacher
macro
5
on
42→92
62→58
54→58
88→84
61.5→73.0
1
on
44→86
30→28
50→58
38→46
40.5→54.5
1
off
22→90
30→46
54→60
30→82
34.0→69.5
Table 4: Sensitivity to replanning interval and warm starting. Macro success at the 50-step goal offset for standard CEM and dynamics-informed initialization under different replanning intervals. Entries are reported as standard CEM → dynamics-informed initialization.
Figure 10
k=5
k=10
env
nl2
R2
∥a^∥∞
nl2
R2
∥a^∥∞
TwoRoom
0.98
−0.02
1.3
1.07
−0.24
2.3
Reacher
1.09
−0.21
0.8
1.14
−0.32
0.8
Cube
1.94
−2.13
4.3
2.36
−4.55
5.2
PushT
4.61
−13.4
4.1
5.62
−16.8
4.5
Table 5: Inverse-head fidelity at different temporal gaps. Normalized ℓ2 error, R2 , and maximum predicted action magnitude of Gψ at five- and ten-step gaps across four environments.
Latent world models learn action-conditioned state transitions in a compact representation space, providing a predictive model for visual planning and control. However, planning with these models typically requires costly test-time optimization over action sequences. We propose to replace this online search with a goal-conditioned inverse model that directly predicts actions from current and goal observations. A key challenge is that action prediction alone neither ensures that goal representations encode action-relevant information nor that predicted actions are consistent with the learned latent prediction model. We introduce Hindsight Inverse JEPA Planning (HIJP), a search-free framework that addresses both aspects during training. HIJP relabels future observations along offline trajectories as goals, providing action supervision across multiple temporal horizons. By retaining gradients through the goal representations, this supervision directly shapes the visual representation. HIJP further feeds predicted actions into the latent predictor model and aligns the resulting next-state representations with encoded successor observations, enforcing action-transition consistency. At inference time, the inverse dynamics model and latent predictor models are applied alternately to generate multi-step action sequences without online search. Across four visual control tasks, HIJP achieves 86.8% overall success and a 1.9-16.5x planning speedup over search-based planning.
Guanchen Du, Jinhao Du, Yanpeng Sun +3
Nanjing University of Science and Technology · University of Oxford · Singapore University of Technology and Design +1
Learning intrinsic dynamics---the relationship between actions and their consequences---is essential for world models. However, visual variations unrelated to the underlying control process can obscure this relationship in learned representations. We investigate the connection between action predictability, defined as the ability to reconstruct an action from consecutive latent states, and the preservation of intrinsic dynamics. Our central hypothesis is that encouraging action predictability discourages representations from discarding the distinctions between different actions and their consequences. We provide two complementary forms of evidence for this connection. Theoretically, we derive a positive lower bound on squared inverse prediction error when the encoder merges distinguishable outcomes of different actions from the same state and appearance, and show that forward prediction can still be exact on those transitions. Empirically, controlled appearance-diversity comparisons, supervision interventions, and latent probes support the role of action prediction in preserving control-relevant transition information. We instantiate action predictability with Concat IDM, which predicts actions from concatenated latent states and updates the shared encoder using action labels already available in offline trajectories, without changing the test-time planner. Across three independent training runs on Push-T with 24 appearances, Concat IDM increases planning success from 9.1% to 77.2%; its paired improvement is 52.9 percentage points larger than that observed with a single appearance. Together, these findings support action predictability as an inductive bias toward intrinsic dynamics and demonstrate the practical benefits of Concat IDM for latent planning under visual diversity.
Jiamin Shi, Yuanhong Zhang, Zhiyang Liu +2
Xi’an Jiaotong University · National University of Singapore · Beijing Zhongguancun Academy
Modern vision-based world models can represent observations as compact yet expressive latent manifolds, but fast goal-oriented planning in these spaces remains challenging. This raises a central question: when does a learned representation simplify control, rather than merely enabling prediction? We study this question in a pretrained LeWorldModel, whose latent geometry is regularized for smoothness and uniformity. Our key insight is that, under such geometry, planning can be amortized into a latent inverse-dynamics mapping instead of requiring online search. We therefore replace iterative planning with a lightweight Goal-Conditioned Inverse Dynamics Model (GC-IDM) that maps the current latent state, goal latent state, and remaining horizon directly to the next action. Empirically, across four benchmark environments spanning navigation, contact-rich manipulation, and continuous control, our controller matches or exceeds CEM in seven of eight environment-protocol settings while reducing per-decision cost by 100-130x. A broader sweep over test-time planners (CEM, MPPI, iCEM, and gradient-based methods) shows that this result is not specific to a particular optimizer. These findings suggest that much of the structure recovered by test-time planning is already locally encoded in the latent representation. More broadly, our results indicate that sufficiently structured latent spaces can shift part of the planning burden from online optimization to learned inference. Our code is publicly available at https://github.com/hdnndh/Latent-Geometry-Beyond-Search-Amortizing-Planning-in-World-Models .
Hoang Nguyen, Xiaohao Xu, Xiaonan Huang
Department of Robotics, University of Michigan, Ann Arbor