Organizations: Tencent ARC Lab · University of Macau · The Chinese University of Hong Kong · City University of Hong Kong · Huazhong University of Science and Technology · Peking University · Great Bay University
Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model. In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions and is optimized through supervised learning and dependency-aware GRPO. The hidden states of special goal tokens that terminate each atomic instruction provide continuous goal embeddings that are cached and reused across control steps. A causal Transformer controller combines the embedding of the active goal with recent observations to predict discrete keyboard and mouse-button actions and continuous camera motion at every game step. We train the system in three stages: planner training, controller pretraining, and joint alignment. Experiments in Minecraft show that TaskFrame is competitive with general-purpose vision-language models on instruction decomposition, and that in closed-loop evaluation the agent achieves a higher combat success rate than the evaluated baselines.
Figures & tables
Figure 1: Overview of TaskFrame. Once per task, the task planner decomposes a high-level instruction into ordered atomic goals, each terminated by a special goal token whose hidden state is cached as a goal embedding. At every frame, the frame controller takes the embedding of the active goal and the latest visual input and emits native keyboard-and-mouse actions.
Figure 2: Overview of TaskFrame , a hierarchical planner–controller agent that separates task-level planning from frame-level control. (A) Task Planner. A vision–language model decomposes a high-level goal (e.g., “collect oak logs across the river” ) into an ordered sequence of atomic goals, each terminated by a special goal token. The hidden states at these tokens are cached as goal embeddings z1,…,zk . (B) Frame Controller. A per-frame autoregressive policy. Each frame consumes the active goal embedding zj , an action-query token ain , a SigLIP image token ot , and the action at , which only later frames attend to. A causal Transformer aggregates them into the action-output token aout , which is decoded into discrete keyboard and mouse-button presses and continuous camera motion. The planner runs once per high-level goal, while the controller acts at every game step.
Table 1: Closed-loop success rate (%) on atomic Minecraft tasks over 3 evaluation seeds, with 12 episodes per kill task and 6 per mining task in each seed. Averages are reported with a binomial estimate of their standard deviation across seeds; per-task mining results are in Section C.3 .
Figure 3: Closed-loop rollouts of TaskFrame. Each row shows four frames of one successful episode on an atomic task. Comparisons with baselines are shown in Figure 6 .
Model
SSF1 ↑
Task2
Acc. (%) ↑
Unparsed (%) ↓
Gemini 3.5 Flash ( Google DeepMind, 2026a )
0.3815
44.4
13.6
Gemini 3.6 Flash ( Google DeepMind, 2026b )
0.3808
46.8
15.2
GPT-4o ( OpenAI, 2024 )
0.2720
36.0
14.0
Qwen3-Omni-30B-A3B ( Qwen Team, 2025 )
0.2643
34.0
11.6
Qwen3.5-397B-A17B ( Qwen Team, 2026a )
0.2990
35.6
12.4
Table 2: Planner performance. SSF1 is evaluated without a supplied step count, whereas Task2 provides the target count. Unparsed is the percentage of questions without a valid decision.
Table 6
Camera head
Yaw ↓
Pitch ↓
Discrete bins
2.1
1.2
Flow matching
1.7
1.0
Table 5: Camera MAE ( ∘ ) on held-out gameplay.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
SFT
GRPO
Initialization
Qwen3.5-9B
SFT checkpoint
Trainable modules
language model, connector
language model, connector
Training data
40,068 examples, 1 epoch
2,207 examples
Peak learning rate
2×10−5
1×10−6
Schedule
cosine, 0.5% warmup
cosine, 3% warmup
Weight decay
0.01
–
Appendix
Table 6: Planner training hyperparameters.
Steps
Total reward
Count score
Order score
KL
Grad. norm
1–50
0.313
0.630
0.178
0.003
1.669
51–150
0.331
0.664
0.188
0.021
1.619
151–250
0.356
0.694
0.211
0.050
1.408
251–350
0.379
0.702
0.240
0.044
1.496
351–450
0.369
0.704
0.226
0.040
1.150
Appendix
Table 7: GRPO training statistics over the first 450 steps (means within each window).
Figure 4: Statistics of the controller training data. Left: per-frame action frequency on a sample of 150 clips (785K frames). Right: task category mix of clean segments, grouped by instruction keyword.
Figure 5: Causal attention mask of the frame controller, adopted from Yue et al. (2026) and shown for two consecutive frames. Rows are query tokens and columns are key tokens; green cells are attended and gray cells are masked. Each frame contributes the goal condition zj , the visual token ot , the action-output token aout , and the ground-truth action at . No token attends to aout , and aout at frame t attends to previous ground-truth actions but not to the same-frame action at .
Table 9: Per-task closed-loop success rate (%) on mining tasks over 3 evaluation seeds (6 episodes per seed), with binomial estimates of the standard deviation across seeds.
Figure 6: Qualitative comparison on (a) kill sheep and (b) mine oak log. Each row shows four frames of one episode, from left to right. Green and red bars indicate success and failure.
Table 10: Closed-loop goal-conditioning ablations on kill tasks (success rate, %, over 3 seeds with 48 episodes per task each; ± values are binomial estimates of the standard deviation across seeds). The model is unchanged; only the goal supplied at inference differs.
Table 11: Success rate (%) under the original JARVIS-VLA task configurations over 3 seeds, with binomial estimates of the standard deviation across seeds (12 episodes per task in each seed).
Recent work shows that pre-trained, instruction-tuned vision-language models (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world-model controllers excel at fast observation-to-action control, but lack open-ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act autonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train controllers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling objectives. We evaluate our proposed approach across seven embodied environments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actuators. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks.
Zineng Tang, Kelsey R. Allen, Sjoerd van Steenkiste +2
Most vision-language-action (VLA) models map observations directly to actions without explicit intermediate planning, which limits performance on long-horizon tasks where early mistakes compound. We propose Coarse-to-Control, a plan-execute VLA that introduces planning natively in the action-token space. The key idea is to let the policy first predict a compact sequence of coarse action tokens that summarize the intended future trajectory, and then generate executable action tokens conditioned on this plan. Because both planning and execution share a unified discrete action vocabulary, the plan stays close to the control manifold and provides directly actionable guidance rather than an abstract hint that must be translated back to motor commands. Experiments on LIBERO, SimplerEnv-WidowX, and real-world manipulation tasks show that action-token planning consistently improves over direct action generation, with the largest gains on long-horizon multi-stage tasks.
Jinhao Wu, Shiduo Zhang, Yicheng Liu +9
1Nanjing University · 2Shanghai Innovation Institute · 3Fudan University +1
Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io