cs.CVNov 7, 2026

Dynamics-Informed Action Learning for Goal-Conditioned Latent Planning

Authors: Yinuo Jiang, Guangzhi Wang, Yiran Wang, Junfu Pu, Cheng Cheng, Honghao Chen, Cheng Cheng, Ying Shan

Organizations: Huazhong University of Science and Technology · Tencent ARC Lab

Abstract

Latent world models enable visual control by planning action sequences in compact representation spaces, but a gap remains between how latent dynamics are learned and how they are used for planning. First, teacher-forced training does not expose the dynamics model to the recursively predicted latent states encountered during planning. Second, sampling-based planners typically initialize action search from goal-agnostic proposals, without exploiting the action-conditioned dynamics learned by the world model. We introduce Dynamics-Informed Action Learning (DIAL) to bridge these two gaps. During training, rollout alignment exposes the forward model to its own autoregressive latent predictions, while multi-step inverse supervision trains a lightweight inverse dynamics head to infer actions from latent-state transitions. We further apply the inverse head to model-generated transitions with its parameters held fixed, encouraging predicted rollouts to retain action-relevant information. At test time, DIAL autoregressively couples the inverse and forward dynamics to construct a dynamics-informed action sequence toward the goal, which initializes the mean of the Cross-Entropy Method (CEM) proposal distribution. The subsequent CEM search then refines this initialization using the standard sampling and updating procedure. Experiments across four visual control benchmarks demonstrate that DIAL outperforms existing latent world models and planning baselines in both standard and long-horizon scenarios.

Figures & tables

Explore similar work

CardsList
  1. Beyond Action Prediction: Learning Search-Free Planning in Latent Space

    Oct 16, 2026Guanchen Du, Jinhao Du, Yanpeng Sun +3Latent-Space PlanningModel-Based Planning

  2. Learning Intrinsic Dynamics in Latent World Models

    Oct 15, 2026Jiamin Shi, Yuanhong Zhang, Zhiyang Liu +2Latent Dynamics ModelingLatent World Models

  3. Latent Geometry Beyond Search: Amortizing Planning in World Models

    May 9, 2026Hoang Nguyen, Xiaohao Xu, Xiaonan HuangWorld Model-Based PlanningModel-Based Planning