HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

Zecong Tang; Zian Guan; Yuncheng Guo; Zhanqiu Zhang; Yiwen Guo; Bo Dai

cs.CVOct 22, 2026

HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

Authors: Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai

Organizations: The University of Hong Kong · Zhejiang University · Fudan University · LIGHTSPEED ·

Abstract

Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.

Figures & tables

Figure 1: System overview of HiSTEVE, illustrating the agent harness mechanism.

Figure 2: Architecture of the VLA policy model and task completion state detector.

	MineStudio Bench				OpenHA				LongBench
Method	Mine	Kill	Craft	Smelt	Emb.	GUI	Combat	Avg.	R&C	E&M	Rec.	Open
VPT	20.0	13.3	0.0	6.7	10.1	0.7	3.6	4.8	0.0	1.4	0.0	0.0
STEVE-1	20.0	13.3	0.0	6.7	8.4	0.0	4.9	4.4	0.0	0.7	0.0	0.0
ROCKET-1	33.3	40.0	0.0	6.7	19.2	0.0	29.8	16.3	1.4	2.9	0.7	0.0
JARVIS-VLA	66.7	60.0	26.7	60.0	31.0	25.3	18.3	24.9	7.1	5.7	1.4	0.7
LatentHA	46.7	33.3	6.7	20.0	27.3	3.5	8.2	13.0	2.1	2.9	0.7	0.0

Table 1: Comparison on MineStudio Bench, OpenHA, and LongBench. MineStudio reports strict pass@1, OpenHA reports family-wise episode success and macro average, and LongBench reports final-goal success. R&C, E&M, Rec., and Open denote the four LongBench task families. The best and second-best results are highlighted in bold and underline , respectively.

Figure 3: Qualitative examples of HiSTEVE. The left panel shows successful atomic-task executions, while the right panel illustrates multi-stage long-horizon execution.

Method	#Params	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	FPR $\downarrow$	FNR $\downarrow$	F1 $\uparrow$
Random	–	0.500	0.500	0.500	0.500	0.500	0.500
Qwen3.5	9B	0.500	0.504	0.030	0.030	0.970	0.057
Qwen3.5	35B	0.502	0.530	0.042	0.037	0.958	0.078
GPT-4o	–	0.698	0.708	0.673	0.277	0.327	0.690
GPT-5.5	–	0.780	0.791	0.761	0.201	0.239	0.776
Ours	1.4B	0.863	0.861	0.865	0.140	0.135	0.863

Table 2: Atomic-subtask completion detection. All models use the same temporal sampling; the random baseline has an expectation of

0.5

on every metric under the balanced validation set.

Method components

MineStudio

OpenHA

LongBench

Method

WKI

Full H.

World

GUI

All

World

GUI

Avg.

Goal SR

Progress

Staged policy training

Imitation learning only

–

\checkmark

\checkmark

–

54.7

46.2

50.4

36.4

15.7

29.5

7.1

11.9

+ world knowledge injection

\checkmark

–

\checkmark

\checkmark

–

71.4

58.9

65.1

41.8

22.4

35.3

14.3

20.7

+ environment-feedback RL

\checkmark

\checkmark

\checkmark

\checkmark

–

89.7

71.6

80.7

43.1

26.8

37.7

14.3

21.9

Regime-specific representation

Table 3: Ablation of staged VLA training, regime-specific representation, and agent harness design. For MineStudio, World averages Mine and Kill, GUI averages Craft and Smelt. For OpenHA, World averages Embodied and Combat, GUI denotes the GUI family. LongBench reports final-goal success (Goal SR) and hidden diagnostic progress (Progress); Full H. denotes the complete agent harness.

Figure 4: Overview of the cross-regime gradient-conflict analysis. (a) Mean conflict gap over all layers and components throughout training; the shaded region denotes

\pm 1

standard deviation across components, and the dashed line is a linear fit. (b) Mean conflict gap by decoder depth, averaged over training steps and components. (c) Mean conflict gap by shared component, averaged over training steps and decoder layers. All panels are computed from the same set of gradient probes.

Index	Action	Human Input	Description
1	Forward	key W	Move forward.
2	Back	key S	Move backward.
3	Left	key A	Strafe left.
4	Right	key D	Strafe right.
5	Jump	key Space	Jump or swim upward.
6	Sneak	key Left Shift	Sneak and move cautiously.

Table 4: Native Minecraft action space used by HiSTEVE. Non-GUI and GUI mouse movements share the same displacement codebook but are represented by disjoint token sets.

Family	Task	Primary long-horizon requirement
Resource & Crafting	wooden_age	Resource acquisition and tool progression
	stone_age	Multi-stage gathering and crafting
	iron_tool	Resource, furnace, and tool-production chain
	glass_maker	Gathering, smelting, and output retrieval
Exploration & Memory	cave_survey	Exploration and persistent environmental state
	camp_return	Landmark memory and return navigation

Table 5: LongBench task suite. Each episode exposes only the final goal to the agent; the decomposition and diagnostic checkpoints are withheld during execution.

FFN intermediate width
Variant	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	FPR $\downarrow$	FNR $\downarrow$	F1 $\uparrow$
Ours	0.863	0.861	0.865	0.140	0.135	0.863
Wanda, width $\tfrac{1}{2}$	0.865	0.863	0.867	0.138	0.133	0.865
Wanda, width $\tfrac{1}{4}$	0.739	0.739	0.740	0.262	0.260	0.739
FFN channel selection
Prefix pruning, width $\tfrac{1}{3}$	0.768	0.768	0.768	0.232	0.232	0.768

Table 6: Ablation of the state detector design. Each group changes one design factor while holding the remaining settings fixed.

	MineStudio Bench				OpenHA				LongBench
Method	Mine	Kill	Craft	Smelt	Emb.	GUI	Combat	Avg.	R&C	E&M	Rec.	Open
VPT	$20.0^{\pm 20.0}$	$13.3^{\pm 11.5}$	$0.0^{\pm 0.0}$	$6.7^{\pm 11.5}$	$10.1^{\pm 4.1}$	$0.7^{\pm 1.1}$	$3.6^{\pm 1.5}$	$4.8^{\pm 1.5}$	$0.0^{\pm 0.0}$	$1.4^{\pm 2.4}$	$0.0^{\pm 0.0}$	$0.0^{\pm 0.0}$
STEVE-1	$20.0^{\pm 20.0}$	$13.3^{\pm 11.5}$	$0.0^{\pm 0.0}$	$6.7^{\pm 11.5}$	$8.4^{\pm 2.5}$	$0.0^{\pm 0.0}$	$4.9^{\pm 2.6}$	$4.4^{\pm 1.5}$	$0.0^{\pm 0.0}$	$0.7^{\pm 1.9}$	$0.0^{\pm 0.0}$	$0.0^{\pm 0.0}$
ROCKET-1	$33.3^{\pm 11.5}$	$40.0^{\pm 20.0}$	$0.0^{\pm 0.0}$	$6.7^{\pm 11.5}$	$19.2^{\pm 2.9}$	$0.0^{\pm 0.0}$	$29.8^{\pm 3.9}$	$16.3^{\pm 1.0}$	$1.4^{\pm 3.8}$	$2.9^{\pm 3.9}$	$0.7^{\pm 1.9}$	$0.0^{\pm 0.0}$
JARVIS-VLA	$66.7^{\pm 11.5}$	$60.0^{\pm 20.0}$	$26.7^{\pm 11.5}$	$60.0^{\pm 34.6}$	$31.0^{\pm 4.6}$	$25.3^{\pm 3.6}$	$18.3^{\pm 4.3}$	$24.9^{\pm 1.9}$	$7.1^{\pm 4.9}$	$5.7^{\pm 4.5}$	$1.4^{\pm 3.8}$	$0.7^{\pm 1.9}$
LatentHA	$46.7^{\pm 23.1}$	$33.3^{\pm 30.6}$	$6.7^{\pm 11.5}$	$20.0^{\pm 20.0}$	$27.3^{\pm 4.2}$	$3.5^{\pm 1.4}$	$8.2^{\pm 3.6}$	$13.0^{\pm 1.7}$	$2.1^{\pm 3.9}$	$2.9^{\pm 2.7}$	$0.7^{\pm 1.9}$	$0.0^{\pm 0.0}$

Table 7: Benchmark performance with variability on MineStudio Bench, OpenHA, and LongBench. Values are mean

\pm

standard deviation in percentage points; bold and underlined means denote the best and second-best results within each column.

Panel of Table 8 — Table 8: Representative examples of the three data formats used for world knowledge injection.

Panel of Table 9 — Table 9: Representative imitation-learning examples from the non-GUI and GUI interaction regimes. Each example contains a single-frame observation, an atomic-subtask instruction, and its aligned target action sequence.

RL for Game AI

See all 29 papers

Sep 24, 2026

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

Yehang Zhang, Haojian Huang, Yifan Chang +13

HKUST(GZ) · Knowin AI · CUHK

Tool-Using Vision-Language Agents Robotic Manipulation Robot Imitation Learning VLMs for Robotics

Ask PDF

Jun 11, 2026

WISE: A Long-Horizon Agent in Minecraft with Why-Which Reasoning

Rapid advances have been made in developing general-purpose embodied agent in environments like Minecraft through the adoption of LLM-augmented hierarchical approaches. Despite their promise, low-level controllers often become performance bottlenecks due to repeated execution failures. We argue that a key limitation is not only the lack of episodic memory, but also the decoupling of what-where-when memory from which-why reasoning. To address this, we propose WISE (Which-Why Informed Semantic Explorer), a long-horizon agent framework with an enhanced low-level controller equipped with a Causal Event Graph that augments episodic memory with explicit causal structure linking observations to task relevance. Unlike prior work such as MrSteve, which relies on feature similarity for retrieval, WISE enables robust recall under viewpoint changes and supports opportunistic task reordering through causal reasoning. Building on this memory, we propose an Opportunistic Task Scheduler that dynamically re-prioritizes subtasks when causally relevant opportunities are detected. We further equip WISE with a multi-scale progressive exploration strategy to provide spatially comprehensive observations for downstream reasoning. Experiments show that WISE largely improves task success and efficiency on long-horizon sparse tasks, particularly in settings requiring adaptive decision-making.

Renmin Cheng, Changhao Chen

The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

Causal Reasoning Long-Horizon Agent Tasks Long-Horizon LLM Agents Memory-Augmented Agents LLM Agent Planning Agentic AI

Ask PDF

Sep 1, 2026

EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent

Vision-language-action (VLA) policies generate executable robot actions, but plausible proposals can have different consequences for the task. We present EmbodiedSkills, a skill-based agent framework that augments frozen VLA policies with world action models for consequence-aware action selection. The framework defines reusable skills for scene observation, context encoding, candidate generation, consequence prediction, outcome evaluation, candidate comparison, and action execution. Five stages organize their input-output dependencies in a shared feedback loop, accommodating both continuous action chunks and pick-and-place primitives. At each decision point, independently trained world action models predict the latent consequences of a common set of policy proposals. Selection combines their outcome scores and prediction disagreement with a lightweight comparator that estimates the benefit of an alternative from agreement in its predicted effects. The predictive and comparison modules learn from matched-state candidate branches, while the action policy and pretrained observation encoders remain frozen. On MetaWorld MT50 and CLIPort, EmbodiedSkills improves success rates from 76.28% to 78.64% and from 33.94% to 36.83%, respectively. An additional CALVIN evaluation with the frozen FLOWER policy increases five-task-chain success from 77.10% to 77.90%. These results demonstrate the applicability of consequence-aware selection across different policy backends and action representations.

Wei Wang, Wenqiao Zhang, Yutong Lin +14

College of Computer Science and Technology, Zhejiang University · Nanjing University of Aeronautics and Astronautics · Cornell University +3

Robotic Control Large Language Model-Based Robot Planning Robotics Vision-Language-Action Models

Ask PDF

Abstract

RL for Game AI

Explore similar work

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

WISE: A Long-Horizon Agent in Minecraft with Why-Which Reasoning

EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent