WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation

Jiaming Wang

cs.ROOct 31, 2026

WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation

Authors: Jiaming Wang

Abstract

Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.

Figures & tables

Figure 1: Three jobs for a VLM in a robot system, shown on the episode of Figure 2 . (a) The VLM picks the next skill from a library, and the skill moves the robot. (b) The VLM writes the motion itself, here the program that Code as Policies wrote for this episode. (c) In WorldSketch, the VLM sketches a minimal kinematic world model with perception tools. The door becomes a link on a hinge with a goal angle, the rest of the kitchen stays as unlabelled voxels, and a planner finds the grasp and a collision-free motion on this sketch.

Figure 2: WorldSketch on a RoboCasa episode ( open the toaster oven door ). The VLM builds the sketch in a short conversation with perception tools and checks it in rendered overlays, here redrawing the glass door as a bar after its first panel failed the depth test. The sketch holds only the door, with its handle, hinge, goal angle and ranked joint hypotheses, and the rest of the kitchen stays as voxels. The planner optimizes the grasp, the base placement and the arm motion along the door’s arc together, and a refuted outcome leads to the next hypothesis, a revision or a fresh sketch.

Figure 3: Sketch construction on two RoboCasa episodes, with the points that the VLM marks (purple dots), the views (dashed circles) and crops (dashed boxes) that it requests, and the accepted sketch in the voxel map of the planner (last image). (a) To find the knob of the front left burner, the VLM reads the burner icon in a label view and sketches the knob with the burner as its effect region. (b) The lever drawn from the first points agrees with depth in 53% of its pixels, and in 92% after the VLM re-points it. Of its four joint hypotheses, the most probable one (ticked) turned the water on. Appendix B gives the transcripts.

method	drawers	doors	knobs	buttons	levers	pick &	all	calls	input	output	time	false
	& racks				& spout	place			tokens	tokens	(min)
trained on demonstrations of the tasks
GR00T N1.5, 3 runs ( Bjorck et al., 2025 )	8.7	8.3	6.0	4.7	8.0	4.3	40.0 (44%)	–	–	–	0.8	–
$\pi_{0.5}$ ( Black et al., 2025a )	10	5	2	1	8	9	35 (39%)	–	–	–	1.2	–
zero-shot, GPT-6.1-Sol
Code as Policies ( Liang et al., 2023 )	3	0	2	2	1	0	8 (9%)	6.1	31k	1.5k	1.9	–

Table 1: RoboCasa kitchen tasks: successes per family (of 15) and in all (of 90), and per episode the VLM calls, the input and output tokens (with reasoning) and the minutes, with each VLM call counted by its response time ( Appendix D ). False : episodes declared solved that were not. The zero-shot methods use GPT-6.1-Sol, and the last row is the better of two runs with GPT-6-Astra ( Table 4 ). The time of ReKep is a lower bound, since 52 of its episodes reach a time limit.

method	pos.	task	pos.	task	pos.	task	all	calls	time	false
	object		goal		spatial
$\pi_{0.5}$ ( Black et al., 2025a )	4	0	9	7	6	16	42	–	0.3	–
CaP-X, single turn ( Fu et al., 2026 )	0	0	1	0	0	0	1	1.9	1.6	–
CaP-Agent0 (CaP-X) ( Fu et al., 2026 )	4	3	4	1	8	3	23	48	13.1	1
our coding agent	25	25	25	23	24	25	147	5.8	0.9	1
WorldSketch	25	25	23	24	24	25	146	3.2	1.4	1

Table 2: LIBERO-PRO, 150 episodes: position (pos.) and task perturbations of three suites, five tasks with five initial states each (successes of 25), with VLM calls and minutes per episode as in Table 1 . False : episodes declared solved that were not. Every method that calls a VLM uses GPT-6.1-Sol, and the last row is WorldSketch with GPT-6-Astra. CaP-X runs as released, with its agent CaP-Agent0 and its single-turn tier with low-level functions (S3), which Appendix H describes with our coding agent.

final system, 90 episodes
change	correct	success
no overlay check †	80 $\to$ 58	–
grounding, 50 ep., 2 runs
masks instead of points	71 $\to$ 46	77 $\to$ 65
hybrid: masks for panels	71 $\to$ 75	77 $\to$ 75
inventory, 50 ep., mask requests

Table 3: Sketch accuracy (first sketch correct) and task success when one part of construction changes, as reference

\to

variant in %. Only the first block uses the final system. The others use earlier versions on other episodes ( Appendices I , J , K and L ), so each row compares only with its own reference. † Recorded VLM answers replayed without execution. ‡ Right part bound, over the episodes that each arm finished.

VLM and effort	success	correct	part	calls	input	output	reasoning
GPT-6-Astra medium, run 1	82	72	88	5.5	90k	1.4k	0.3k
GPT-6-Astra low	77	69	84	5.3	86k	1.2k	0.2k
GPT-6.1-Sol medium	82	68	87	5.4	88k	1.8k	0.7k

Table 4: WorldSketch with different VLMs and reasoning efforts on the 90 kitchen episodes: successes, first sketches that are correct and that bind the right part, VLM calls per episode, and input, output and reasoning tokens per episode.

	ours	CaP	CaP-Agent0	ReKep
failures (episodes)	18/180	82/90	60/90	85/90
world model wrong	11%	67%	23%	78%
wrong part or contact site	11%	52%	5%	56%
wrong mechanism or direction	0%	5%	13%	13%
wrong goal or end state	0%	10%	5%	8%
execution failed	89%	13%	77%	6%

Table 5: Root causes of the failures on the kitchen tasks, as shares of the failures of each method ( Appendix F ). World model : the model of the task was wrong (for WorldSketch the sketch, for the code-writing agents the part, mechanism and goal that their programs assume, and for ReKep its keypoints and constraints). Execution : the model was right but the motion failed. Other : program and parser errors. WorldSketch: the two runs with GPT-6-Astra.

Figure 4: Failure cases on the kitchen tasks: for each method the most common cause in its main failure class ( Table 5 ), and in (a) a sketch error of WorldSketch. Red marks show the error, and dashed grey marks the right part. (a) The VLM sketches the lid of the coffee machine as its start button. (b) Code as Policies turns the rear-left knob instead of the front-left one. (c) ReKep grasps at a keypoint on a rolling pin inside the drawer. (d) The fingers of WorldSketch slip off a cabinet bar 6.6 mm thick (the frame is from after the slips). (e) The espresso head, hidden by the hand, blocks the grasps of CaP-Agent0 at the mug.

method	VLM	SR	SPL	NE	OSR	nDTW
Open-Nav ( Qiao et al., 2025 )	GPT-4	19	16.1	6.70	23	45.8
SmartWay ( Shi et al., 2025 )	GPT-4o	29	22.5	7.01	51	–
Three-Step Nav ( Zheng et al., 2026 )	GPT-5	34	29.1	5.87	39	57.7
STRIDER ( He et al., 2025 )	GPT-4o	35	30.3	6.91	39	51.8
LaViRA ( Ding et al., 2026 )	Gemini 2.5 Pro	38.3	28.3	6.54	48.7	–
EvoNav ( Dai et al., 2026b )	Gemini 2.5 Pro	43	37.8	5.04	51	62.4

Table 6: Zero-shot navigation on R2R-CE val-unseen. Upper block: the 100 Open-Nav episodes, with published results. MIP ran GPT-6.1-Sol at low reasoning effort and reports only its SR. WorldSketch uses GPT-6.1-Sol at medium effort (mean of three runs, s.d. of SR 1.2). Lower block: 376 episodes on paths unused in development (one run). SR, SPL (success weighted by path length), OSR (oracle success) and nDTW (fidelity to the reference path) in %, NE (final distance to the goal) in metres.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

family	tasks
drawers, racks	OpenDrawer, CloseDrawer, SlideOvenRack , SlideToasterOvenRack , CloseFridgeDrawer
doors	OpenCabinet, CloseCabinet, OpenMicrowave, CloseMicrowave, OpenToasterOvenDoor
knobs	TurnOnStove, TurnOffStove, PreheatOven , TurnOnToasterOven , LowerHeat
buttons	TurnOnMicrowave, TurnOffMicrowave, StartCoffeeMachine , TurnOnBlender , OpenElectricKettleLid
levers, spout	TurnOnSinkFaucet, TurnOffSinkFaucet, TurnSinkSpout, TurnOnElectricKettle , AdjustWaterTemperature
pick & place	PickPlaceCounterToSink, PickPlaceMicrowaveToCounter , PickPlaceCabinetToCounter , PickPlaceCounterToDrawer , CoffeeSetupMug

Appendix

Table 7: The RoboCasa kitchen tasks: five atomic tasks per family, three episodes each (seeds 1000–1089, RoboCasa365’s default kitchens). Tasks in italics were drawn with a fixed random seed from the RoboCasa365 atomic tasks that act on the same kind of part as their family.

tool	the VLM writes	grounding	returned	uses
surface point	name, image, pixel	a plane fitted to the depth around the pixel (6 mm RANSAC); the point is where the pixel’s ray meets it	3-D point, normal	78
part point	name, image, pixel	a plane fitted to the ring around the pixel; depth more than 6 mm in front of it is clustered, and the nearest cluster’s centroid is the point	point, protrusion, size, long axis	195
corner	name, image, pixel	depth along the pixel’s ray; when a panel’s corners come from one image, they are placed on the plane fitted inside them, preferring a front plane (glass)	point	291
raw point	name, image, pixel	median depth along the pixel’s ray	point, normal	199
wrist view	pixel, distance (0.4, 0.65 or 0.9 m)	the wrist camera looks at the 3-D point under the pixel, from a pose with a free line of sight that the arm can reach	new image	83
label view	pixel	the wrist camera straight in front of the surface under the pixel, about 45 cm away (the base backs off if needed)	high-resolution image	35

Appendix

Table 8: The tools the VLM calls while sketching (the request fields of its answer) and how the backend grounds them. Uses : requests over the 90 episodes of one run.

outcome $o$	same	flipped	other
verified	0.80	0.01	0.03
satisfied, unconfirmed	0.10	0.05	0.10
moved without effect	0.04	0.25	0.40
moved the full amount, no effect	0.15	0.25	0.40
moved along another joint	0.05	0.20	0.50
resisted	0.04	0.60	0.40

Appendix

Table 9: Outcome likelihoods

P(o\mid\rho)

used in the belief update.

kitchen tasks
method	think	model	plan	execute	other	total
WorldSketch	91	37	157	143	0	429
WorldSketch, GPT-6-Astra	78	36	112	149	0	374
Code as Policies	56	25	3	33	0	117
CaP-Agent0	792	65	8	154	0	1019
ReKep	35	30	5442	334	0	$\geq$ 5842

Appendix

Table 10: Time per episode by stage, in seconds: VLM response time ( think ), building the world model apart from the VLM ( model ), motion planning and inverse kinematics ( plan ), robot motion and outcome checks in the simulator ( execute ), and the rest. Where the logs do not separate model, plan and execute, one value spans the three columns. The VLAs log only the total.

set	episodes	correct	type	axis	point	path	goal	truth among hypotheses
kitchen tasks (2 runs)	180	81%	97%	94%	88%	93%	92%	82/89

Appendix

Table 11: Accuracy of the first sketch against ground truth in the two runs with GPT-6-Astra ( Table 4 ). Correct : right part, joint type, axis (

\leq 10^{\circ}

), interaction point (

\leq 2

cm), path (

\leq 3

cm) and goal inside the success range. Per-joint rates are over target joints; truth among hypotheses is over joints of faucets, spouts, knobs and buttons.

grounding	$n$	correct	axis $\leq 10^{\circ}$	pivot (cm)	path $\leq 3$ cm	direction	success
				median / p90		right	given correct / not
points	220	68%	92%	1.7 / 3.2	76%	92%	85% / 66%
hybrid	100	75%	93%	1.5 / 3.1	80%	92%	77% / 68%
masks	100	46%	93%	2.9 / 35.2	67%	73%	80% / 52%

Appendix

Table 12: Accuracy of the first model for three ways of grounding it, with an earlier version (420 episodes). Axis, pivot, path and direction are scored per joint.

cause	failures	examples
small part the fingers slip off or cannot enter	6	6.6 mm cabinet bars inside a thicker visual handle; a small faucet lever; a toaster tray’s thin lip; a kettle button recessed 1–2 mm in the handle
no reachable contact	3	a cabinet door open flat against the fridge; a plate deep inside a microwave
wrong functional site in the sketch	2	the lid of a coffee machine pressed instead of its start button; a mug placed at the drip tray, 10 cm from the dispenser
placement settles off target	2	a mug released within 0.5 cm of the dispenser settled 4.2–4.4 cm away (limit 4 cm)
held object slipped out	2	a cup dropped during the carry; a mug lost at the release
object wider than the gripper	2	a translucent container, 9.7 cm wide, whose depth hits the shelf behind it

Appendix

Table 13: Root causes of the 18 failures in the two runs with GPT-6-Astra ( Table 4 ), one cause per failure, from the logs and replays.

task	Astra	Sol	CaP	A0	ReKep	$\pi_{0.5}$	coding	task	Astra	Sol	CaP	A0	ReKep	$\pi_{0.5}$	coding
	ours								ours
OpenDrawer	3	3	1	2	0	2	3	TurnOnMicrowave	3	3	0	3	0	1	3
CloseDrawer	3	3	1	2	0	2	3	TurnOffMicrowave	3	3	1	3	0	0	3
SlideOvenRack	3	2	0	1	1	1	2	StartCoffeeMachine	2	3	1	3	0	0	3
SlideToasterOvenRack	3	3	0	0	0	2	1	TurnOnBlender	3	3	0	1	0	0	0
CloseFridgeDrawer	3	3	1	1	1	3	2	OpenElectricKettleLid	2	3	0	2	0	0	3

Appendix

Table 14: Successes per task on the 90 kitchen episodes (of 3): WorldSketch with GPT-6-Astra (the run of Table 1 ) and with GPT-6.1-Sol, Code as Policies (CaP), CaP-Agent0 (A0) and ReKep with GPT-6.1-Sol,

\pi_{0.5}

, and the multi-turn coding agent of Appendix H .

set	functions
perception	get_observation (RGB-D, intrinsics and pose of each camera), segment_sam3_text_prompt , segment_sam3_point_prompt , point_prompt_molmo (answered by the VLM), get_oriented_bounding_box_from_3d_points , plan_grasp (Contact-GraspNet), get_robot_state
motion	solve_ik , move_to_joints , goto_pose , navigate_to_pose , goto_home_joint_position , open_gripper , close_gripper
skill library (CaP-Agent0)	rotation_matrix_to_quaternion , decompose_transform , depth_to_point_cloud , mask_to_world_points , pixel_to_world_point , transform_points , interpolate_segment , normalize_vector , select_top_down_grasp
high level (CaP)	get_object_pose , get_object_poses , sample_grasp_pose , goto_pose , move_to_pose , open_gripper , close_gripper , get_robot_state , goto_home_joint_position

Appendix

Table 15: Functions of the code-writing baselines. CaP-Agent0 gets the low-level set and CaP-X’s synthesized skill library (as in the M4 tier of CaP-X with the library), and Code as Policies gets the high-level functions of CaP-X, built from the same pieces.

method	$n$	success	ours, same	$\Delta$ (95% CI)	calls	input tokens
WorldSketch (earlier version)	50	38.5	–	–	4.2	0.13M
direct poses	47	28	37	$-9.0$ ( $-15.5$ , $-3.0$ )	7.3	0.23M
skill orchestrator	50	26	38.5	$-12.5$ ( $-19.0$ , $-6.5$ )	6.9	0.21M
keypoint constraints	26	16	21.5	$-5.5$ ( $-10.5$ , $-0.5$ )	7.4	0.35M
multi-turn coding agent	21	14	16.5	$-2.5$ ( $-8.0$ , 3.5)	17.9	0.58M

Appendix

Table 16: Further baselines on 50 episodes of the 15 common tasks, with an earlier version of WorldSketch (two runs, mean) and one partial run per baseline, all with GPT-6-Astra. Unfinished arms are compared with ours on the episodes they finished.

\Delta

: difference with a bootstrap 95% interval.

	requests	selection	coarse	full
successes / finished	26/40	14/34	4/50	11/22
right part bound	18/40	17/27	6/27	8/21
$\Delta$ vs requests, shared	–	$-8$ /31	$-22$ /40	$-4$ /22
McNemar $p$	–	0.02	$<10^{-5}$	0.22
primitives extracted	7	1361	91	118
primitives in model	2.8	3.0	1.8	52

Appendix

Table 17: Requesting versus selecting from an inventory, with an earlier version of the system (same segmenter, planner, executor and verifier; 50 episodes, one run each; unfinished arms are compared on shared episodes). Requests : the VLM asks for masks of the parts it needs; selection : the backend segments and fits everything and the VLM selects; coarse : object-level boxes and planes only; full : requests, but every visible articulated part is modelled.

	requests	exhaustive	coarse
primitives extracted	7	1213	94
primitives in model (mean)	2.1	10.9	3.4
correct binding	25/30	17/30	3/30
binding error	1.0 cm	3.1 cm	25 cm
construction time	70 s	97 s	52 s
uncached input and output tokens	55k	79k	22k

Appendix

Table 18: Task-directed requests against selection from an inventory with an early version of the system and executor (9 tasks

\times

5 seeds; medians unless noted). Binding : the bound part lies within a few centimetres of the true handle, knob, lever or button (30 fixture episodes). Task success was low for all three methods with the executor of that time and does not separate them.

family	$n$	points	hybrid	masks	union
drawers	8	12	14	14	14
doors	20	28	23	21	30
knobs	6	10	9	7	10
buttons	6	12	11	11	12
levers, spout	8	12	15	12	16
pick & place	2	3	3	0	4

Appendix

Table 19: Three groundings on 50 episodes of the 15 common tasks (other seeds) with an earlier version, successes summed over two runs (of

2n

). Union : solved by any grounding in the same run.

arm	success	model correct as built
full system (3 runs)	78% (141/180)	69%
no verifier	81% (48/59)	–
no requested views	80% (47/59)	71%
single hypothesis	78% (46/59)	66%
no overlay check	76% (45/59)	59%

Appendix

Table 20: Component ablations of an earlier version on 60 episodes of the 15 common tasks (seeds 120–179), one run per arm (the full system: three runs).

episodes	VLM	WorldSketch	coding agent	note
60	GPT-6-Astra (default)	78% of 180	–
	Claude Sonnet 5.5	61% of 90	–	models correct as built 38% against 69%
	Qwen3.5-9B (open weight)	2% of 57	–	copies the prompt’s example coordinates
30	GPT-6 (Astra; Sol for the agent)	19	22	first sketch correct 23/30
	Gemini 4 Argon	17	24	first sketch correct 18/30; paired $p=0.23$
	Qwen3.6-35B (open weight)	1	0

Appendix

Table 21: Other VLMs, with earlier versions of WorldSketch. Top: 60 episodes of the 15 common tasks (runs summed). Bottom: 30 of the kitchen episodes (the first seed of each task), with a version that solved 53 of the 90, and the multi-turn coding agent of Appendix H on the same episodes.

condition	success	clean, same episodes
clean replay	20/30	–
depth noise $\sigma=0.5$ cm	17/28	19
depth noise $\sigma=1$ cm	9/23	16
depth noise $\sigma=2$ cm	4/30	20
range-proportional noise 2%	1/29	20
10% of pixels dropped	21/30	20

Appendix

Table 22: Perception noise with an earlier version: the VLM’s outputs are replayed from a baseline run and only the perception is degraded (30-episode subset, one run per condition, 23–30 episodes each). Clean: the same episodes without noise.

episodes		SR (%)	SPL (%)	NE (m)	OSR (%)	nDTW (%)	VLM calls
Open-Nav	all (100)	75.7 $\pm$ 1.2	57.5	2.86	83.0	67.4	5.7
	outside dev. (87)	74.7 $\pm$ 1.2	55.9	2.86	82.8	67.2	5.7
	path-disjoint (63)	77.8 $\pm$ 2.8	57.7	2.49	83.6	67.5	5.9
unused	all (376)	76.9	56.6	2.60	82.2	67.2	6.3
	single-floor (306)	75.8	56.5	2.79	81.0	66.6	5.7
	multi-floor (70)	81.4	57.4	1.79	87.1	69.6	9.3

Appendix

Table 23: Navigation results by episode subset (GPT-6.1-Sol, medium effort). For the Open-Nav episodes, SR is the mean and s.d. over three runs, outside dev. leaves out the 13 development episodes and path-disjoint also the 24 that share a path with one. The unused episodes have one run.

Agentic AI for Robotics

See all 60 papers

Oct 30, 2026

SketchMate: Agentic Geometric Reasoning for Zero-Shot Precision Insertion

Inserting an unseen part requires sub-millimetre precision and the correct orientation of keyed features. Current zero-shot agentic manipulation methods built on vision-language models (VLMs) produce spatial outputs that are far too coarse for this. We propose to let a VLM build a sketch model of the task, i.e., a few geometric primitives and two mate frames, which a VLM agent measures with geometric tools in multi-view depth and which a standard planner and compliant controller execute. Because a sketch describes the parts independently of their placement, the robot needs to build it only once per task. A sketch memory stores each agent run and transfers the stored sketch to later placements by point-cloud registration, so that the VLM is called only for unfamiliar parts. After grasping the part, the robot inspects both mating faces at close range and corrects the sketch by shape matching. On 69 simulated insertion episodes, our method inserts 88% of the parts at the assembled pose in a strict kinematic test, while ReKep and MOKA with the same VLM insert none. On new placements of eight tasks, reusing one stored run inserts as many parts as calling the agent in each episode, and it produces the sketch in 3.4 s instead of about 90 s.

Jiaming Wang

Robotic Insertion Robotic Manipulation Zero-Shot Robotic Manipulation Agentic AI for Robotics VLMs for Robotics

Ask PDF

Jul 30, 2026

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io

Xiangcheng Zhang, Yilun Du

Harvard University

World Models for Robotics World Model-Based Planning Action-Conditioned World Models Robot Policy Generalization World Models for VLA Models

Ask PDF

Sep 24, 2026

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

Yehang Zhang, Haojian Huang, Yifan Chang +13

HKUST(GZ) · Knowin AI · CUHK

Tool-Using Vision-Language Agents Robotic Manipulation Robot Imitation Learning VLMs for Robotics

Ask PDF

Abstract

Agentic AI for Robotics

Explore similar work

SketchMate: Agentic Geometric Reasoning for Zero-Shot Precision Insertion

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal