Recent work shows that pre-trained, instruction-tuned vision-language models (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world-model controllers excel at fast observation-to-action control, but lack open-ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act autonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train controllers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling objectives. We evaluate our proposed approach across seven embodied environments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actuators. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks.
Figures & tables
Figure 1: Illustration of inference with a VLM planner and trained controller.
Figure 2: Per-setting performance (left) and efficiency (right) results for (a) agent scaling and (b) planning modes. Model-size and reasoning scaling curves are shown in Figure 3 (Appendix A.6 ).
Table 3
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Scale (tag)
rssm.deter
rssm.hidden
rssm.classes
depth
units
50M ( size50m )
4096
512
32
32
512
200M ( size200m )
8192
1024
64
64
1024
800M ( size800m )
24576
3072
192
192
3072
Appendix
Table 4: Model-scale configurations.
Method
Throughput
RT-2 ( Zitkovich et al., 2023 )
16.2
DEPS ( Wang et al., 2023 )
20.5
LS-Imagine ( Li et al., 2025 )
23.7
JARVIS-1 ( Wang et al., 2025 )
26.0
Instruct-to-Act (Qwen-2.5-VL-72B)
31.7
Appendix
Table 5: Inference throughput (environment steps/s) for representative VLA baselines and our framework with a Qwen-based planner.
Controller architecture
Diamond
Dreamer-V3-style RSSM (ours)
11.0
Transformer world model ( Chen et al., 2022 )
11.2
RNN policy (model-free)
9.7
Value-only world model
10.2
Appendix
Table 7: Controller architecture ablation on Minecraft Diamond. The gains are not specific to an RSSM controller, though world-model controllers perform best.
Task
Gemma
Qwen
GPT-4o
Minecraft Diamond
193/200
190/200
194/200
Pico Park
186/200
180/200
189/200
Atari
181/200
172/200
185/200
Appendix
Table 8: Instruction-following accuracy measured on 200 held-out planner instructions per domain.
Figure 3: Per-setting performance (left) and efficiency (right) results for the model-size and reasoning scaling analyses in Sec. 4.4 . Agent scaling and planning modes are shown in Figure 2 in the main paper. Model scales for controller and VLM are small (50M, 2B), medium (200M, 7B), and large (800M, 72B).
Minecraft Diamond
Overcooked
Online
Offline
Online
Offline
VLM latency mean (ms)
1037.4
1221.9
1106.3
1010.4
VLM latency p95 (ms)
1701.1
2035.7
1872.5
1523.7
Staleness mean (steps)
51.2
0.0
165.1
0.0
Blocking steps mean
0.0
35.4
0.0
0.0
Throughput (env steps/sec)
37.5
13.4
169.9
92.7
Appendix
Table 9: Latency, instruction staleness, and throughput for online (asynchronous) versus offline (blocking) planning. Overcooked staleness and latency are averaged over both agents.
Figure 4: Training curves of the individual loss components for a 200M-parameter controller on Minecraft Diamond (seed 0). The overall objective is a linear combination of the world-model losses (reconstruction, dynamics, and reward prediction), the actor and value objectives, the behavior-cloning (BC) loss on instruction-relabeled segments, and the stop-token loss.
Figure 5: Training summarization prompt.
Figure 6: Single-agent prompt.
Figure 7: Multi-agent prompt (to be appended to single-agent prompt).