Organizations: Tencent ARC Lab · University of Macau · The Chinese University of Hong Kong · City University of Hong Kong · Huazhong University of Science and Technology · Peking University · Great Bay University
Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model. In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions and is optimized through supervised learning and dependency-aware GRPO. The hidden states of special goal tokens that terminate each atomic instruction provide continuous goal embeddings that are cached and reused across control steps. A causal Transformer controller combines the embedding of the active goal with recent observations to predict discrete keyboard and mouse-button actions and continuous camera motion at every game step. We train the system in three stages: planner training, controller pretraining, and joint alignment. Experiments in Minecraft show that TaskFrame is competitive with general-purpose vision-language models on instruction decomposition, and that in closed-loop evaluation the agent achieves a higher combat success rate than the evaluated baselines.
Figures & tables
Figure 1: Overview of TaskFrame. Once per task, the task planner decomposes a high-level instruction into ordered atomic goals, each terminated by a special goal token whose hidden state is cached as a goal embedding. At every frame, the frame controller takes the embedding of the active goal and the latest visual input and emits native keyboard-and-mouse actions.
Figure 2: Overview of TaskFrame , a hierarchical planner–controller agent that separates task-level planning from frame-level control. (A) Task Planner. A vision–language model decomposes a high-level goal (e.g., “collect oak logs across the river” ) into an ordered sequence of atomic goals, each terminated by a special goal token. The hidden states at these tokens are cached as goal embeddings z1,…,zk . (B) Frame Controller. A per-frame autoregressive policy. Each frame consumes the active goal embedding zj , an action-query token ain , a SigLIP image token ot , and the action at , which only later frames attend to. A causal Transformer aggregates them into the action-output token aout , which is decoded into discrete keyboard and mouse-button presses and continuous camera motion. The planner runs once per high-level goal, while the controller acts at every game step.
Table 1: Closed-loop success rate (%) on atomic Minecraft tasks over 3 evaluation seeds, with 12 episodes per kill task and 6 per mining task in each seed. Averages are reported with a binomial estimate of their standard deviation across seeds; per-task mining results are in Section C.3 .
Figure 3: Closed-loop rollouts of TaskFrame. Each row shows four frames of one successful episode on an atomic task. Comparisons with baselines are shown in Figure 6 .
Model
SSF1 ↑
Task2
Acc. (%) ↑
Unparsed (%) ↓
Gemini 3.5 Flash ( Google DeepMind, 2026a )
0.3815
44.4
13.6
Gemini 3.6 Flash ( Google DeepMind, 2026b )
0.3808
46.8
15.2
GPT-4o ( OpenAI, 2024 )
0.2720
36.0
14.0
Qwen3-Omni-30B-A3B ( Qwen Team, 2025 )
0.2643
34.0
11.6
Qwen3.5-397B-A17B ( Qwen Team, 2026a )
0.2990
35.6
12.4
Table 2: Planner performance. SSF1 is evaluated without a supplied step count, whereas Task2 provides the target count. Unparsed is the percentage of questions without a valid decision.
Table 6
Camera head
Yaw ↓
Pitch ↓
Discrete bins
2.1
1.2
Flow matching
1.7
1.0
Table 5: Camera MAE ( ∘ ) on held-out gameplay.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
SFT
GRPO
Initialization
Qwen3.5-9B
SFT checkpoint
Trainable modules
language model, connector
language model, connector
Training data
40,068 examples, 1 epoch
2,207 examples
Peak learning rate
2×10−5
1×10−6
Schedule
cosine, 0.5% warmup
cosine, 3% warmup
Weight decay
0.01
–
Appendix
Table 6: Planner training hyperparameters.
Steps
Total reward
Count score
Order score
KL
Grad. norm
1–50
0.313
0.630
0.178
0.003
1.669
51–150
0.331
0.664
0.188
0.021
1.619
151–250
0.356
0.694
0.211
0.050
1.408
251–350
0.379
0.702
0.240
0.044
1.496
351–450
0.369
0.704
0.226
0.040
1.150
Appendix
Table 7: GRPO training statistics over the first 450 steps (means within each window).
Figure 4: Statistics of the controller training data. Left: per-frame action frequency on a sample of 150 clips (785K frames). Right: task category mix of clean segments, grouped by instruction keyword.
Figure 5: Causal attention mask of the frame controller, adopted from Yue et al. (2026) and shown for two consecutive frames. Rows are query tokens and columns are key tokens; green cells are attended and gray cells are masked. Each frame contributes the goal condition zj , the visual token ot , the action-output token aout , and the ground-truth action at . No token attends to aout , and aout at frame t attends to previous ground-truth actions but not to the same-frame action at .
Table 9: Per-task closed-loop success rate (%) on mining tasks over 3 evaluation seeds (6 episodes per seed), with binomial estimates of the standard deviation across seeds.
Figure 6: Qualitative comparison on (a) kill sheep and (b) mine oak log. Each row shows four frames of one episode, from left to right. Green and red bars indicate success and failure.
Table 10: Closed-loop goal-conditioning ablations on kill tasks (success rate, %, over 3 seeds with 48 episodes per task each; ± values are binomial estimates of the standard deviation across seeds). The model is unchanged; only the goal supplied at inference differs.
Table 11: Success rate (%) under the original JARVIS-VLA task configurations over 3 seeds, with binomial estimates of the standard deviation across seeds (12 episodes per task in each seed).