HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

Zecong Tang; Zian Guan; Yuncheng Guo; Zhanqiu Zhang; Yiwen Guo; Bo Dai

cs.CVOct 22, 2026

HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

Authors: Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai

Organizations: The University of Hong Kong · Zhejiang University · Fudan University · LIGHTSPEED ·

Abstract

Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.

Figures & tables

Figure 1: System overview of HiSTEVE, illustrating the agent harness mechanism.

Figure 2: Architecture of the VLA policy model and task completion state detector.

	MineStudio Bench				OpenHA				LongBench
Method	Mine	Kill	Craft	Smelt	Emb.	GUI	Combat	Avg.	R&C	E&M	Rec.	Open
VPT	20.0	13.3	0.0	6.7	10.1	0.7	3.6	4.8	0.0	1.4	0.0	0.0
STEVE-1	20.0	13.3	0.0	6.7	8.4	0.0	4.9	4.4	0.0	0.7	0.0	0.0
ROCKET-1	33.3	40.0	0.0	6.7	19.2	0.0	29.8	16.3	1.4	2.9	0.7	0.0
JARVIS-VLA	66.7	60.0	26.7	60.0	31.0	25.3	18.3	24.9	7.1	5.7	1.4	0.7
LatentHA	46.7	33.3	6.7	20.0	27.3	3.5	8.2	13.0	2.1	2.9	0.7	0.0

Table 1: Comparison on MineStudio Bench, OpenHA, and LongBench. MineStudio reports strict pass@1, OpenHA reports family-wise episode success and macro average, and LongBench reports final-goal success. R&C, E&M, Rec., and Open denote the four LongBench task families. The best and second-best results are highlighted in bold and underline , respectively.

Figure 3: Qualitative examples of HiSTEVE. The left panel shows successful atomic-task executions, while the right panel illustrates multi-stage long-horizon execution.

Method	#Params	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	FPR $\downarrow$	FNR $\downarrow$	F1 $\uparrow$
Random	–	0.500	0.500	0.500	0.500	0.500	0.500
Qwen3.5	9B	0.500	0.504	0.030	0.030	0.970	0.057
Qwen3.5	35B	0.502	0.530	0.042	0.037	0.958	0.078
GPT-4o	–	0.698	0.708	0.673	0.277	0.327	0.690
GPT-5.5	–	0.780	0.791	0.761	0.201	0.239	0.776
Ours	1.4B	0.863	0.861	0.865	0.140	0.135	0.863

Table 2: Atomic-subtask completion detection. All models use the same temporal sampling; the random baseline has an expectation of

0.5

on every metric under the balanced validation set.

Method components

MineStudio

OpenHA

LongBench

Method

WKI

Full H.

World

GUI

All

World

GUI

Avg.

Goal SR

Progress

Staged policy training

Imitation learning only

–

\checkmark

\checkmark

–

54.7

46.2

50.4

36.4

15.7

29.5

7.1

11.9

+ world knowledge injection

\checkmark

–

\checkmark

\checkmark

–

71.4

58.9

65.1

41.8

22.4

35.3

14.3

20.7

+ environment-feedback RL

\checkmark

\checkmark

\checkmark

\checkmark

–

89.7

71.6

80.7

43.1

26.8

37.7

14.3

21.9

Regime-specific representation

Table 3: Ablation of staged VLA training, regime-specific representation, and agent harness design. For MineStudio, World averages Mine and Kill, GUI averages Craft and Smelt. For OpenHA, World averages Embodied and Combat, GUI denotes the GUI family. LongBench reports final-goal success (Goal SR) and hidden diagnostic progress (Progress); Full H. denotes the complete agent harness.

Figure 4: Overview of the cross-regime gradient-conflict analysis. (a) Mean conflict gap over all layers and components throughout training; the shaded region denotes

\pm 1

standard deviation across components, and the dashed line is a linear fit. (b) Mean conflict gap by decoder depth, averaged over training steps and components. (c) Mean conflict gap by shared component, averaged over training steps and decoder layers. All panels are computed from the same set of gradient probes.

Index	Action	Human Input	Description
1	Forward	key W	Move forward.
2	Back	key S	Move backward.
3	Left	key A	Strafe left.
4	Right	key D	Strafe right.
5	Jump	key Space	Jump or swim upward.
6	Sneak	key Left Shift	Sneak and move cautiously.

Table 4: Native Minecraft action space used by HiSTEVE. Non-GUI and GUI mouse movements share the same displacement codebook but are represented by disjoint token sets.

Family	Task	Primary long-horizon requirement
Resource & Crafting	wooden_age	Resource acquisition and tool progression
	stone_age	Multi-stage gathering and crafting
	iron_tool	Resource, furnace, and tool-production chain
	glass_maker	Gathering, smelting, and output retrieval
Exploration & Memory	cave_survey	Exploration and persistent environmental state
	camp_return	Landmark memory and return navigation

Table 5: LongBench task suite. Each episode exposes only the final goal to the agent; the decomposition and diagnostic checkpoints are withheld during execution.

FFN intermediate width
Variant	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	FPR $\downarrow$	FNR $\downarrow$	F1 $\uparrow$
Ours	0.863	0.861	0.865	0.140	0.135	0.863
Wanda, width $\tfrac{1}{2}$	0.865	0.863	0.867	0.138	0.133	0.865
Wanda, width $\tfrac{1}{4}$	0.739	0.739	0.740	0.262	0.260	0.739
FFN channel selection
Prefix pruning, width $\tfrac{1}{3}$	0.768	0.768	0.768	0.232	0.232	0.768

Table 6: Ablation of the state detector design. Each group changes one design factor while holding the remaining settings fixed.

	MineStudio Bench				OpenHA				LongBench
Method	Mine	Kill	Craft	Smelt	Emb.	GUI	Combat	Avg.	R&C	E&M	Rec.	Open
VPT	$20.0^{\pm 20.0}$	$13.3^{\pm 11.5}$	$0.0^{\pm 0.0}$	$6.7^{\pm 11.5}$	$10.1^{\pm 4.1}$	$0.7^{\pm 1.1}$	$3.6^{\pm 1.5}$	$4.8^{\pm 1.5}$	$0.0^{\pm 0.0}$	$1.4^{\pm 2.4}$	$0.0^{\pm 0.0}$	$0.0^{\pm 0.0}$
STEVE-1	$20.0^{\pm 20.0}$	$13.3^{\pm 11.5}$	$0.0^{\pm 0.0}$	$6.7^{\pm 11.5}$	$8.4^{\pm 2.5}$	$0.0^{\pm 0.0}$	$4.9^{\pm 2.6}$	$4.4^{\pm 1.5}$	$0.0^{\pm 0.0}$	$0.7^{\pm 1.9}$	$0.0^{\pm 0.0}$	$0.0^{\pm 0.0}$
ROCKET-1	$33.3^{\pm 11.5}$	$40.0^{\pm 20.0}$	$0.0^{\pm 0.0}$	$6.7^{\pm 11.5}$	$19.2^{\pm 2.9}$	$0.0^{\pm 0.0}$	$29.8^{\pm 3.9}$	$16.3^{\pm 1.0}$	$1.4^{\pm 3.8}$	$2.9^{\pm 3.9}$	$0.7^{\pm 1.9}$	$0.0^{\pm 0.0}$
JARVIS-VLA	$66.7^{\pm 11.5}$	$60.0^{\pm 20.0}$	$26.7^{\pm 11.5}$	$60.0^{\pm 34.6}$	$31.0^{\pm 4.6}$	$25.3^{\pm 3.6}$	$18.3^{\pm 4.3}$	$24.9^{\pm 1.9}$	$7.1^{\pm 4.9}$	$5.7^{\pm 4.5}$	$1.4^{\pm 3.8}$	$0.7^{\pm 1.9}$
LatentHA	$46.7^{\pm 23.1}$	$33.3^{\pm 30.6}$	$6.7^{\pm 11.5}$	$20.0^{\pm 20.0}$	$27.3^{\pm 4.2}$	$3.5^{\pm 1.4}$	$8.2^{\pm 3.6}$	$13.0^{\pm 1.7}$	$2.1^{\pm 3.9}$	$2.9^{\pm 2.7}$	$0.7^{\pm 1.9}$	$0.0^{\pm 0.0}$

Table 7: Benchmark performance with variability on MineStudio Bench, OpenHA, and LongBench. Values are mean

\pm

standard deviation in percentage points; bold and underlined means denote the best and second-best results within each column.

Panel of Table 8 — Table 8: Representative examples of the three data formats used for world knowledge injection.

Panel of Table 9 — Table 9: Representative imitation-learning examples from the non-GUI and GUI interaction regimes. Each example contains a single-frame observation, an atomic-subtask instruction, and its aligned target action sequence.

RL for Game AI

See all 29 papers

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
Sep 24, 2026Yehang Zhang, Haojian Huang, Yifan Chang +13Tool-Using Vision-Language Agents Robotic Manipulation
HKUST(GZ) · Knowin AI · CUHK
WISE: A Long-Horizon Agent in Minecraft with Why-Which Reasoning
Jun 11, 2026Renmin Cheng, Changhao ChenCausal Reasoning Long-Horizon Agent Tasks
The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)
EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent
Sep 1, 2026Wei Wang, Wenqiao Zhang, Yutong Lin +14Robotic Control Large Language Model-Based Robot Planning
College of Computer Science and Technology, Zhejiang University · Nanjing University of Aeronautics and Astronautics · Cornell University +3

Sep 24, 2026J/K move · Enter open · S save

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

Yehang Zhang, Haojian Huang, Yifan Chang, Jianchong Su+12

HKUST(GZ) · Knowin AI · CUHK

Tool-Using Vision-Language Agents Robotic Manipulation Robot Imitation Learning VLMs for Robotics

Open paper AskPDF arXiv

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

Abstract

RL for Game AI

Explore similar work

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

WISE: A Long-Horizon Agent in Minecraft with Why-Which Reasoning

EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent