MAESTRO: Orchestrating Robotics Modules with Coding Agents for Zero-Shot Generalist Robots

Junyao Shi; Rujia Yang; Kaitian Chao; Selina Bingqing Wan; Yifei Shao; Jiahui Lei; Jianing Qian; Long Le

cs.RONov 2, 2025

MAESTRO: Orchestrating Robotics Modules with Coding Agents for Zero-Shot Generalist Robots

Authors: Junyao Shi, Rujia Yang, Kaitian Chao, Selina Bingqing Wan, Yifei Shao, Jiahui Lei, Jianing Qian, Long Le,

Organizations: University of Pennsylvania

Abstract

Today's dominant route to generalist robots scales robotics data to train end-to-end policies. We instead use a vision-language model (VLM) as a coding agent that writes robot programs using a curated library of perception, planning, and control tools. MAESTRO's orchestration harness packages multi-step perception and action procedures into callable tools and guides their use through a plan-act-replan loop, using visual and execution feedback to verify outcomes and recover from failures. With this tool library and harness, MAESTRO matches or surpasses state-of-the-art vision-language-action (VLA) models zero-shot in our tabletop comparison and performs diverse mobile manipulation tasks, without collecting new teleoperation data or invoking a VLA tool. Ablations show that expanding the tool library improves average task progress with the full harness, while adding tools without it does not reliably improve progress. Case studies illustrate MAESTRO's potential for autonomous code improvement across trials, faster execution with VLA tools, and automatic robot data generation. We show that expanding robotics tool libraries together with their orchestration harnesses offers a complementary route to general-purpose robots alongside scaling end-to-end robot training data. Project site: https://maestro-robot.github.io. Code: https://github.com/penn-pal-lab/maestro.

Figures & tables

Figure 1: maestro receives language instruction and leverages a set of tools to complete diverse tasks in a zero-shot setting.

Figure 2: Given prompt and images, VLM plans by writing and executing code that integrates perception, spatial reasoning, control, learned visuomotor policies, and image editing. Execution results (images and stdout) provide feedback for reacting and replanning, forming a closed-loop perception–action–learning cycle. This enables adaptive long-horizon manipulation, as shown at right (instruction: Grasp knife by the handle and cut banana in the middle).

Panel of Figure 3 — Figure 3: Tabletop manipulation evaluation tasks. See the project website for videos.

Figure 4: Tabletop manipulation results: mean task progress (0–100; higher is better). Error bars show

\pm 1

standard error across five trials per method and task; numerical values are reported in Table 2 .

Figure 7: Scaling maestro ’s tools and harness together. Mean task progress for cut banana, rotate cube, and open cabinet as tool groups are added to Base CaP (S0): S1 (+perception), S2 (+grasp/control), and S3 (+geometry). The four conditions are Full harness (I+E), Interface only (I), Execution only (E), and No harness. Error bars show per-task standard errors, computed as sample SD divided by $\sqrt{n}$ for each task–stage–condition cell. S0 uses the same five baseline trials per task for all four curves.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Stage / component	Base CaP	maestro
(a) Tool library — curated robotics modules composed as callable tools, by scaling stage (Sec. 3.2 )
S0 (Base CaP)	VLM-internal 2D pointing, bounding boxes, imagined top-down grasps; Cartesian + gripper control [ 2 , 4 ]	Starting point; each row below replaces a default with a curated module
S1: advanced perception	VLM performs perception itself [ 8 , 19 , 2 ]	Grounded-SAM segmentation centers; FoundationStereo [ 30 ] metric depth; ReKep-inspired task-relevant keypoints [ 31 ] ; active perception (zoom / look around with wrist camera); point and 6-DoF-pose image-editing feedback $\dagger$
S2: grasping & control	Imagined top-down grasp only; direct Cartesian moves, no collision awareness [ 2 , 4 ]	GraspGen [ 32 ] 6-DoF grasps (side / angled / functional-part) with multi-view fusion; cuRobo [ 33 ] collision-free motion planning
S3: geometry & spatial reasoning	None (language-only)	Distance, vector construction, rotation between vectors, rotation about an axis
S4: VLA	None	$\pi_{0.5}$ [ 12 ] VLA with high-frequency closed-loop monitoring (Qwen-2.5-VL)

Appendix

Table 1: Comparison of maestro ’s tool library and harness against Base CaP , the closed-loop coding-agent pattern maestro builds on (Sec. 3.1 ; the Gemini Robotics agent [ 2 , Fig. 12, App. B.3.2] is a documented example ). Components in bold are introduced or substantially scaled in maestro ; cells marked None indicate no equivalent in Base CaP. Both parts are organized by the scaling stage, from S0 (Base CaP) through S4, at which each capability enters: part (a) lists the curated robotics modules added at each stage (Sec. 3.2 ), and part (b) lists the harness infrastructure each stage requires (Sec. 3.3 ). Tags ( a )–( e ) in part (b) name the five harness design principles of Sec. 3.3 .

Figure 8: A summarized overview of maestro ’s system prompt.

Figure 9: One replan iteration on “Erase the characters on the whiteboard”. The first actor_code grasps 1 cm too high (top); from the post-execution image and robot state, replan rewrites a single line ( grasp_pos[2] ) and the retry succeeds, completing the task with no further replans (bottom). See the project website for a video of this replanning example.

Figure 10: Self-evolution of maestro ’s generated policy on the “open cabinet door” task. Top: representative frames with completion progress. Middle: semantic code abstraction with changed parts highlighted in blue. Bottom: strategy changes and outcome per trial. Across four autonomous trials, the agent evolves from a naive straight-line pull to a hinge-based arc trajectory with grasp stabilization, achieving full task completion.

Challenge	Task Description	Gemini Robotics Agent	$\pi_{0}$	$\pi_{0.5}$	maestro
Pick–Place	Put item in bowl	$83.6\pm 15.9$	$74.0\pm 16.6$	$70.0\pm 18.4$	$\mathbf{98.0\pm 2.0}$
Deformable object	Fold the four corners of the towel into the center	$29.0\pm 8.7$	$47.0\pm 11.2$	$\mathbf{70.0\pm 6.9}$	$68.0\pm 8.9$
Articulated object	Open cabinet	$2.0\pm 2.0$	$5.0\pm 2.2$	$2.4\pm 2.4$	$\mathbf{68.0\pm 14.0}$
Spatial reasoning	Rotate cube purple side up	$52.0\pm 16.5$	$23.0\pm 15.0$	$30.0\pm 17.0$	$\mathbf{88.0\pm 5.6}$
Tool use	Cut banana with knife	$71.0\pm 12.9$	$29.6\pm 10.7$	$14.0\pm 2.9$	$\mathbf{92.0\pm 2.5}$
Object affordance	Hang mug on mug holder	$46.0\pm 10.4$	$59.0\pm 13.7$	$\mathbf{80.0\pm 6.3}$	$69.0\pm 4.3$

Appendix

Table 2: Tabletop manipulation results: mean

\pm

standard error of task progress (0–100; higher is better) across methods and tasks.

Task Category	Task Description	maestro
Long-horizon manipulation	Collect all toys on table	$85.0\pm 10.0$
Long-horizon loco-manipulation	Throw green ball into garbage can	$76.7\pm 6.7$
Active exploration	Search for item and return	$96.0\pm 4.0$
Object affordance	Press button to open door	$93.3\pm 6.7$

Appendix

Table 3: Mobile manipulation results: mean

\pm

standard error (SE) of task progress (0–100; higher is better) for maestro , with

n=5

trials per task. SEs are computed from the reported rounded SDs as

\mathrm{SE}=\mathrm{SD}/\sqrt{5}

Figure 11: Per-trial visualization of tabletop manipulation results. Each row corresponds to a task and shows the original trial setting alongside the five STAR-Gen evaluation trials. Above each trial image, a gradient scale bar ranges from white (0 progress) on the left to black (100 progress) on the right. Colored triangular markers ( $\blacktriangledown$ ) on this bar denote each method’s score on that trial, with colors corresponding to the methods indicated in the legend.

Method	Fold Towel	Rotate Cube
maestro	$68.0\pm 8.9$	$88.0\pm 5.6$
maestro w/o advanced perception	$40.0\pm 3.2$	$25.0\pm 0.0$
maestro w/o geometry modules	$67.5\pm 1.6$	$42.5\pm 14.2$

Appendix

Table 4: Leave-one-out ablation on the Fold Towel and Rotate Cube tasks. Values are mean task progress out of 100

\pm

standard error (SE), with

n=5

trials per method and task and

\mathrm{SE}=\mathrm{SD}/\sqrt{5}

. The full maestro baseline reuses Table 2 ; ablation SEs are calculated from the reported rounded SDs.

Figure 12: Aggregate tool-use statistics across 62 maestro tabletop trials (2,062 calls across 18 modules in 6 categories). Top left: per-module call frequencies. Top right: calls grouped by module category. Bottom left: most frequent consecutive tool-call pairs. The full interactive passmap with per-task filtering is hosted on our project site .

Figure 13: Full maestro tool-use passmap aggregated across 62 tabletop trials (2,062 calls, 18 modules). Nodes are tool modules color-coded by category (perception, control, gripper, grasping, geometry, VLA); node size is proportional to call count and edge width to transition frequency. The interactive version on our project site supports per-task filtering, drag-and-drop layouts, and highlighting of common workflows.

Figure 14: Per-task tool-use passmap thumbnails. Each panel shows the call graph for one task category (e.g. pick-place, deformable, articulated, long-horizon-with-VLA); node size encodes call count and edge width encodes transition frequency. The dominant edges and most-called modules differ across categories, illustrating that maestro composes task-specific workflows rather than selecting a fixed pipeline. Full-resolution per-task passmaps and an interactive viewer with task-level filtering are available on our project site .

Task category	Trials	First-attempt failures ( $X$ )	Later logged success ( $Y$ )	Rate ( $Y/X$ )
Pick-place	7	2	1	50.0%
Deformable	6	3	3	100.0%
Articulated	5	2	2	100.0%
Tool use	7	8	4	50.0%
Spatial reasoning	9	14	12	85.7%
Object affordance	5	4	2	50.0%

Appendix

Table 5: Per-task rates of later logged success after an initial failure. For each task category, we report the number of trials, the number of subtask instances whose first attempt failed (

X

), the subset with a later logged success judgment (

Y

), and the conditional rate

Y/X

. A “subtask” is a unique step index within a trial; inclusion in

Y

requires a later attempt of the same step index in the same trial to reach Success: True . These log-derived judgments are distinct from human-scored trial completion.

Component	Total (min)	Wall (%)	Calls	Mean (s)	Min (s)	Max (s)
VLM planning	152.8	22.1	570	16.1	5.1	77.6
Initial planning	21.1	3.1	61	20.8	9.6	40.3
Replanning	131.7	19.1	509	15.5	5.1	77.6
GroundedSAM segmentation	104.6	15.1	1433	4.4	0.9	24.8
Keypoint picking	37.9	5.5	59	38.5	8.6	124.8
Gemini pointing	14.3	2.1	41	20.9	3.7	39.4

Appendix

Table 6: Component latency aggregated across a mixture of main and VLA experiments. The listed tool calls account for approximately 47% of total experiment wall time; the breakdown is not exhaustive, with remaining time including robot execution, waiting, logging, and other operations. Counts denote tool invocations, not independent trials; initial planning and replanning subdivide VLM planning and are not counted twice.

Method	Progress (%)	Time
maestro w/o $\pi_{0.5}$	$89\pm 5.1$	$11^{\prime}\,18^{\prime\prime}$
maestro w/ $\pi_{0.5}$	$94\pm 4.9$	$4^{\prime}\,59^{\prime\prime}$
$\pi_{0.5}$	$17\pm 9.1$	$2^{\prime}\,16^{\prime\prime}$
$\pi_{0}$	$21\pm 6.6$	$2^{\prime}\,58^{\prime\prime}$
Gemini Robotics	$80\pm 3.0$	$11^{\prime}\,53^{\prime\prime}$

Appendix

Table 7: Evaluation of maestro with and without

\pi_{0.5}

as a callable pick-and-place tool, on the clean up the table, largest object first task. Reported are average task progress (0–100; higher is better) with

\pm 1

standard error across five trials per method (

\mathrm{SE}=\mathrm{SD}/\sqrt{5}

, in percentage points), and average wall-clock completion time.

Figure 15: Automatic data generation with Tether [ 35 ] : a two-task case study. We compare Bowl from Shelf and Wiping with Cloth . maestro + Tether starts from 10 maestro execution rollouts; Tether starts from 10 human-teleoperated rollouts, and Diffusion Policy [ 59 ] trains on Tether’s human-teleoperated data. The $\pi_{0}$ [ 1 ] baseline is evaluated zero-shot. See the project website for videos of the data-collection tasks.

Agentic AI for Robotics

See all 54 papers

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation
Sep 29, 2026Bingxuan Li, Siqi Song, Yizhuo Wu +3Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation
University of Illinois Urbana-Champaign
Auriga Technical Report: A Task-Agnostic Agent Harness for Zero-Shot Embodied Manipulation
Oct 31, 2026Chengcheng Wang, Yuheng Shi, Siyu Xu +18Agentic AI for Robotics Zero-Shot Robotic Manipulation
Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens
Oct 1, 2026Ruiyang Si, Jianxin Bi, Shunyu Yang +12Vision-Language-Action Models Tool-Augmented Language Model Agents
Peking University · National University of Singapore · NVIDIA +1

Sep 29, 2026J/K move · Enter open · S save

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

Bingxuan Li, Siqi Song, Yizhuo Wu, Jiarui Yao+2

University of Illinois Urbana-Champaign

Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation Robotic Manipulation VLMs for Robotics Reasoning for VLA Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have advanced robotic manipulation, but their zero-shot generalization in new tasks and environments remains limited, and their reliance on specialized training keeps them from benefiting directly from rapidly advancing general-purpose vision-language models (VLMs). In parallel, recent agentic robotic systems leverage VLMs for high-level reasoning or coding agents for robot control, but often depend on extensive external models and tools, introducing additional complexity and cost. This motivates us to ask: Can a general-purpose VLM itself operate a robot more like the human teleoperator by reasoning directly from observations, issuing actions, and continuously adapting to execution feedback, without relying on external models such as learned action experts, coding agents or grounding tools like SAM3? In this work, we introduce MotorMind, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates. Without task-specific policy training, coding agents, or additional grounding tools such as SAM3, MotorMind achieves 66.7% success on the base LIBERO-PRO suites and 53.8% under perturbations, compared with at most 13.3% and 19.2%, respectively, for the prior zero-shot methods we evaluate. The same interface reaches 95% average success on a real xArm6 robot across direct manipulation and human-perturbation settings. Replacing the backbone with a stronger VLM further improves performance, while the remaining failures - primarily due to visual grounding, embodied reasoning, and action knowledge - decrease as VLM capability improves. These results show that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchronous execution harness, can perform effective zero-shot robotic manipulation.

Abstract

Agentic AI for Robotics

Explore similar work

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

Auriga Technical Report: A Task-Agnostic Agent Harness for Zero-Shot Embodied Manipulation

Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens