Vision-Language Agents Can Learn in Their Own Latent Worlds

Haoxi Li; Sikai Bai; Jianfei Ma; Qinglin Hou; Xiaocheng Lu; Tao Han; Jie Zhang; Song Guo

cs.AIOct 27, 2026

Vision-Language Agents Can Learn in Their Own Latent Worlds

Authors: Haoxi Li, Sikai Bai, Jianfei Ma, Qinglin Hou, Xiaocheng Lu, Tao Han, Jie Zhang, Song Guo

Organizations: Department of Computer Science and Engineering, HKUST, Hong Kong, China · School of Computing, National University of Singapore, Singapore · Department of Computer Science, University of Southern California, Los Angeles, USA

Abstract

Learning from experience is essential for building the next generation of agents that continually expand their capabilities. Pretrained vision-language models (VLMs) already internalize rich world knowledge, yet this knowledge is rarely leveraged to drive policy improvement through online interaction. Realizing this potential efficiently remains challenging, as policy updates shift the agent's interaction distribution, requiring the world model to adapt accordingly, while continually updating large pretrained models is computationally expensive. To address this challenge, we introduce Dwell, a model-based reinforcement learning (RL) framework that reframes a VLM as a recurrent latent transition operator, endowing it with the ability to predict, adjust, and act in latent space. Leveraging the VLM's own representations and its dual role as both world model and policy, the agent continually learns from real interactions to anticipate and evaluate action outcomes, and from latent imagination to improve its behavior. Across five interactive visual environments, Dwell achieves the strongest overall performance among trainable methods while substantially reducing the compute required for imagination, showing that effective world models for VLM agents need not reconstruct the future; they need to predict futures that are useful for action. Our code is publicly available at https://github.com/Tom111-lhx/Dwell

Figures & tables

Figure 1: Overview of Dwell . Left: The VLM learns a recurrent latent world by matching action-conditioned priors to observation-conditioned posteriors, while grounding the latent states with a frozen V-JEPA2 encoder. Right: The same VLM backbone serves as both world model and policy, alternating online interaction with latent planning for policy improvement.

Figure 2: Learning Efficiency. Success rate throughout training on Sokoban and FrozenLake . Left : comparison with model-based approaches. Right : comparison with model-free approaches.

Figure 3: Computational Efficiency. End-to-end GPU-hours required to reach the reported training step on FrozenLake , Sokoban , Navigation , and EB-ALFRED . GPU-hours include all computation incurred during online training, including world-model updates, imagined rollouts, and policy/value optimization. All methods are evaluated under the same environment-specific training schedule and hardware configuration.

Figure 4: Ablation Studies. Left: effect of removing the representation-grounding objective

\mathcal{L}_{\mathrm{rep}}

or the predictive dynamics objective

\mathcal{L}_{\mathrm{dyn}}

. Right: effect of freezing the world model after initialization instead of continually updating it with newly collected experience. Predictive dynamics provide the primary supervision for useful latent imagination, while representation grounding and continual world-model adaptation further improve policy learning.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{J}_{\rm WM}=\mathcal{L}_{\rm dyn}+\alpha\mathcal{L}_{\rm rep}+\beta\mathcal{L}_{\rm rew}+\eta\mathcal{L}_{\rm reg}.

Appendix

Algorithm 1 Dwell

Variant	Sokoban	FrozenLake	Navigation
Dwell (Shared)	$0.53_{\scriptscriptstyle\pm 0.012}$	$0.51_{\scriptscriptstyle\pm 0.021}$	$0.39$
Bridged Latent Interface	$0.38_{\scriptscriptstyle\pm 0.138}$	$0.42_{\scriptscriptstyle\pm 0.097}$	$0.31_{\scriptscriptstyle\pm 0.017}$

Appendix

Table 2: Effect of the shared latent interface. We compare the default Dwell , where predicted latent states are consumed directly by the policy, with a bridged variant that maps world-model states into a separate policy latent space through a learned adapter. Results are success rates averaged over three training seeds.

Environment	Observation	Actions	Horizon	Eval. episodes
Sokoban	top-down RGB, $6\!\times\!6$ , one box	4 moves	25/25	256
FrozenLake	top-down RGB, $4\!\times\!4$	4 moves	25/25	128
Navigation	first-person RGB, instruction	8 motions	30/10	120
PrimitiveSkill	RGB, instruction	3 parameterized skills	3/3 turns	64
EB-ALFRED	first-person RGB, instruction	162 skills	30/30	20

Appendix

Table 3: Environment settings. Horizon is the maximum number of training / evaluation decisions. PrimitiveSkill observations also include task-state text; each episode has three language turns with at most two primitive commands per turn (at most six in total), and its 64 evaluation episodes cover four tasks (16 each).

	Real reward		Dwell in imagination
Environment	Success	Step	Label	Reward	Episode end
Sokoban	$+10$	$-0.1$	binary	$1.1\,\mathbf{1}[p\geq\tau]-0.01$	$p\geq\tau$
FrozenLake	$+10$	$0$	3-class	$0.1\,\mathbf{1}[p_{\mathrm{goal}}\geq 0.5]$	goal or hole
Navigation	$+10$	$0$	binary	$1.1\,p-0.1$	cutoff at $H$
PrimitiveSkill	$+1$	$0.1+\Delta s$	binary, stage	$1.1\,p-0.1$	success or $H$
EB-ALFRED	$+1$	$-0.1$	binary	$1.1\,\widetilde{p}\,\mathbf{1}[t=H]-0.1$	cutoff at $H$

Appendix

Table 4: Reward assignment.

p

is the predicted success probability,

\tau

a calibrated threshold,

\widetilde{p}

a damped version of

p

, and

t

the imagination step. For PrimitiveSkill, the step reward is a

0.1

format reward plus

\Delta s

, the increase in the highest completed task stage, and the feedback head also predicts the stage.

Setting	Sokoban	FrozenLake	Navigation	EB-ALFRED
Backbone and latent state
VLM backbone	Qwen2.5-VL-3B-Instruct (frozen)
LoRA rank / $\alpha$ / dropout	$32$ / $64$ / $0.05$
Latent slots $K$ / width $D$	$36$ / $2048$
Visual teacher $\mathrm{E}$	vjepa2-vitg-fpc64-384-ssv2 (frozen)
World-model fitting

Appendix

Table 5: Key hyperparameters of Dwell .

RL for VLMs

Reinforcement learning for vision-language models aligns multimodal reasoning, tool use, and action with task-specific feedback.

See all 472 papers

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning
Jul 3, 2026Hanchen Cui, Sergio Arnaud, Arjun Majumdar +5World Model-Based Planning Robotic Manipulation
1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities
WISE: World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models
Sep 3, 2026Chenhao Zhang, Hanyu Zhao, Hang Cheng +2Robustness of VLA Models World Models for Robotics
1Tsinghua University · 2Beijing Academy of Artificial Intelligence (BAAI)
DRILL: Training World Models to Improve Policies, Not Predict Pixels
Oct 22, 2026Zhaolu Kang, Tailong Luo, Lei Wei +9World Models for VLA Models World Model Learning
Hunyuan, Tencent · Peking University

Jul 3, 2026J/K move · Enter open · S save

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning

Hanchen Cui, Sergio Arnaud, Arjun Majumdar, Daniel Dugas+4

1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities

World Model-Based Planning Robotic Manipulation Vision-Language-Action Models Test-Time Adaptation of VLA Models Robustness of VLA Models

Open paper AskPDF arXiv

Pretrained vision-language-action (VLA) policies show promising zero-shot generalization, but often fail under deployment-time distribution shift, leading to decreased robustness and inconsistent instruction following. While prior work commonly tackles this by finetuning on in-distribution data, it assumes demonstrations collected on tasks in the target environment. In this work, we propose DREAMSTEER, a deployment-time steering framework for pretrained VLAs without any finetuning or parameter modifications. The key insight in DREAMSTEER is to leverage a latent world model and a value model to steer pretrained VLA policies. During deployment, DREAMSTEER samples candidate action chunks from a VLA policy and predefined motion primitives, imagines their outcomes using an action-conditioned latent world model, and ranks the imagined trajectories with a language-conditioned value model. Across four real-world manipulation benchmarks with unseen objects, DREAMSTEER improves task success rate from 23.75% to 66.25% and instruction-following accuracy from 38.75% to 56.25% over the base VLA policy.

Abstract

RL for VLMs

Explore similar work

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning

WISE: World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models

DRILL: Training World Models to Improve Policies, Not Predict Pixels