World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

Yehang Zhang; Haojian Huang; Yifan Chang; Jianchong Su; Bohan Zhou; Yingjie Xu; Wosong Chen; Tianhao Zhou

cs.ROSep 24, 2026

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

Authors: Yehang Zhang, Haojian Huang, Yifan Chang, Jianchong Su, Bohan Zhou, Yingjie Xu, Wosong Chen, Tianhao Zhou, +8 more

Organizations: HKUST(GZ) · Knowin AI · CUHK

Abstract

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

Figures & tables

Figure 1: World Action Agent (WAA). WAA combines visual action rehearsal with reusable skills, enabling VLMs to perform diverse manipulation tasks and adapt through expert demonstrations and human guidance.

Figure 2: Overview of the World Action Agent (WAA) harness. (A) The main agent observes global and Contact views in the visual action workspace, rehearses action proposals with an Imagination Agent, consults skills through a Skill Agent, and executes planned motions and local corrections. (B) Expert videos and human teaching are distilled into a multimodal skill library through extraction, editing, and review. (C) Interaction traces train a smaller VLM to operate the same harness. Insets are from two recorded bowl-manipulation tasks; dashed arrows indicate target guides.

Figure 3: Action rehearsal and in-view correction. (a) The Imagination Agent rotates an infeasible grasp preview (purple) until planning succeeds; the main agent then shifts it toward the handle (red circle). (b) Contact A appears aligned, but Contact B reveals an offset; the main agent drags the bowl toward its target in Contact B, and the harness converts the drag into end-effector motion. These interface examples are rendered from RGB-D point clouds.

	Object		Goal		Spatial
Method	Pos.	Task	Pos.	Task	Pos.	Task	Avg.
OpenVLA	0.0	0.0	0.0	0.0	0.0	0.0	0.0
$\pi_{0.5}$	17.0	1.0	38.0	0.0	20.0	1.0	12.8
CaP-Agent0 (CaP-X)	22.0	18.0	26.0	17.0	12.0	14.0	18.2
CaP-Agent0 (Playful)	27.0	31.0	29.0	16.0	13.0	23.0	23.2
RATs (Playful)	61.0	63.0	43.0	36.0	29.0	31.0	43.8

Table 1: Success rates (%) on LIBERO-Pro. Bold marks the best result in each column.

Figure 4: Qualitative examples of WAA. (A) Drag corrections align and lower a bowl before release. Circles mark object and gripper anchors, and arrows summarize the corrections. (B) An infeasible proposal becomes executable after raising the candidate. (C) Revised contact enables a successful soup-can lift after failed grasps.

Method	API cost ( $)$ \downarrow$	Model calls $\downarrow$	Time (s) $\downarrow$	Input token (K) $\downarrow$	Output token (K) $\downarrow$
Show-Harness	0.5021	119.80	874.12	347.10	62.90
WAA	0.1996	31.05	150.47	193.62	5.45

Table 2: Per-episode cost compared with Show-Harness.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Stored reference for unstable cylindrical-object contact. The tilted object is not centered between the fingers. The skill uses this example to guide reassessment and a new approach.

Figure 8: One-demonstration acquisition of a stove-activation skill. (A) RGB observations and aligned end-effector states from one expert trajectory provide contact, motion, and outcome evidence. (B) The learning workflow converts these observations into a procedure and revises an overly broad success condition. (C) The frozen skill is used by a Gemini 3.7 Flash agent through the WAA harness. Source images and role outputs are from the recorded run.

Figure 9: Point-cloud renderings of the same LIBERO-Pro scene through Agentview and Contact cameras. RGB-D + Hand combines Agentview, left, right, and wrist RGB-D with FK gripper geometry. VGGT ( Wang et al., 2025 ) reconstructs from eight RGB views without measured depth or a hand model. Camera poses locate the viewing frames. All inputs are captured in simulation.

Function	Arguments	Operation and feedback
detect_region	query : string within_region_id ∗ : string	Detect and segment a region in the latest observation. Return a region reference and, when available, the centroid of its visible point cloud in the base frame. No robot motion.
propose_grasps	region_id : string	Generate grasp candidates with seed_id references. A candidate must be converted to an action with preview_grasp before execution.
locate_point	query : string within_region_id ∗ : string force_refresh ∗ : bool	Estimate a coarse 3D anchor and display its point reference on the canvas. Optionally restrict grounding to a region or refresh the cached estimate. No robot motion.
preview_pose	point_id : string offset_xyz_m : number[3] quaternion_xyzw ∗ : number[4]	Construct a pending pose and motion plan. Target TCP position equals the referenced point plus the base-frame offset. Omitted orientation preserves the current orientation.
preview_grasp	seed_id : string	Convert a grasp candidate into a pending action and motion plan for visual inspection. Return an action reference without executing or closing the gripper.
imagine_action	instruction : string action_id ∗ : string	Request local geometric inspection and refinement by the Imagination Agent. Start from the referenced action, or the current TCP when omitted. Return a virtual preview without physical execution.

Appendix

Table 4: Main-agent functions. Arguments marked with ∗ are optional. Physical execution is stated explicitly in the last column.

Function	Arguments	Operation and feedback
shift_preview	delta_xyz_m : number[3] frame : base or tool	Translate the virtual target in the selected frame and update the preview for inspection, subject to the metric bounds above.
rotate_preview	angle_deg : number	Rotate the virtual target about its tool-local $+Z$ axis, with a right-handed angle in $[-10,10]$ degrees.
finish_imagination	status : ready or failed	Return the current preview on ready , or abandon the current edits on failed . Readiness does not indicate execution or task success.

Appendix

Table 5: Imagination Agent functions for action rehearsal. These functions edit or return virtual action proposals and do not move the robot.

Function	Arguments	Operation and feedback
select_skill_evidence	state_ids : string[] reference_ids : string[] reason : string	Select at most two state entries and a configuration-bounded number of reference images. The selection reason is limited to 240 characters.
report_skill_guidance	applicability : enum reference_differences : string[] uncertainty : string[]	Report applicable , not_applicable , or uncertain . Each list contains at most two statements, each limited to 120 characters.

Appendix

Table 6: Reporting functions used by the optional Skill Agent. All listed arguments are required.

Setting	Value
Training data	112 successful LIBERO-Pro episodes (Gemini 3.7 Flash, evolved skills)
Samples	1,774 main-agent tool calls
Framework	LLaMA-Factory 0.9.5
Fine-tuning method	LoRA on all linear layers (rank 8, $\alpha=16$ , dropout 0)
Trainable parameters	21.6M
Frozen modules	Vision encoder, multimodal projector

Appendix

Table 7: Fine-tuning configuration for the Qwen3.5-9B pilot.

Panel of Figure 10 — Figure 10: Three failure cases. Panels are cropped from the Canvas received at the labeled decision turns.

Explore similar work

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Goal-Conditioned Dynamic Manipulation Visuomotor Control
Tsinghua University · Striding AI · Purdue University +4
World Pilot: Steering Vision-Language-Action Models with World-Action Priors
Jun 10, 2026Zefu Lin, Rongxu Cui, Junjia Xu +4Efficient World-Action Model World Models
Institute of Automation, Chinese Academy of Sciences (CASIA) · Nanjing University · Beihang University
AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation
Sep 20, 2026Yicheng Jiang, Zesen Gan, Xiaobo Wang +8Efficient World-Action Model Robotic Manipulation
The Hong Kong University of Science and Technology · Shenzhen University of Advanced Technology · Sangfor Technologies Inc. +4

Jul 9, 2026, cs.ROJ/K move · Enter open · S save

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li+13

Tsinghua University · Striding AI · Purdue University+4

Goal-Conditioned Dynamic Manipulation Visuomotor Control Frozen Vision-Language Models

Open paper AskPDF arXiv

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R. Real-world demonstrations on dual-Franka robots further show target redirection, grasp recovery, and new task compositions with the same frozen VLA. Code is available at https://github.com/RLinf/RPent.

Abstract

Explore similar work

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

World Pilot: Steering Vision-Language-Action Models with World-Action Priors

AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation