Auriga Technical Report: A Task-Agnostic Agent Harness for Zero-Shot Embodied Manipulation

Chengcheng Wang; Yuheng Shi; Siyu Xu; Yutong Wang; Jinjing Zhao; Chenghao Xia; Jinqi Liao; Jiankun Peng

cs.ROOct 31, 2026

Auriga Technical Report: A Task-Agnostic Agent Harness for Zero-Shot Embodied Manipulation

Authors: Chengcheng Wang, Yuheng Shi, Siyu Xu, Yutong Wang, Jinjing Zhao, Chenghao Xia, Jinqi Liao, Jiankun Peng,

Abstract

Vision-language models can interpret instructions and plan manipulation tasks, yet struggle when acting directly as robot policies. On RoboDojo, a large planning-execution gap separates high-level task decisions from physical execution: removing arm execution raises GPT-6 Astra's success rate from 22.5% under the reference control interface to 76.2%. We present Auriga, a task-agnostic agent harness that closes 68.3% of this gap in zero-shot manipulation without target-task trials or resets. In Auriga, the model makes all task decisions over an explicit task belief and acts through either direct tool calls or synthesized multi-step Python programs, while the harness supplies calibrated 3D measurements, finger-referenced dual-arm control, and a single admission gateway to the robot. Before motion, the model inspects a visual trajectory preview while simulation is paused; during execution, the harness checks declared stage tolerances and grasp bounds, stopping the robot on divergence and returning the measured discrepancy for in-place recovery. Developed across 13 cross-task capability iterations and frozen for evaluation, Auriga ranks first on the RoboDojo leaderboard with a 57.7% success rate (64.0 Score), more than doubling the reference harness on the same model, and remains first under scene randomization while learned policies drop by up to 33.9%. On RoboCasa365, the same frozen harness reaches a 73.2% success rate, outperforming VLA-assisted agents by up to 23.1% on composite tasks. Project page: https://auriga.stelledge.com/ Code: https://github.com/lose4578/Auriga

Figures & tables

Figure 1: Auriga completes manipulation tasks zero-shot within a single episode. A general-purpose model plans and acts through a shared harness, with no target-task exploration, demonstrations, or resets. The model localizes targets by multi-view triangulation, complemented by depth and point-cloud estimates (1), and reviews candidate motions before they are executed (2). The harness verifies each execution and stops when an action fails (3); the model then revises its plan and recovers from the current state (4–5) until the task succeeds (6).

Figure 2: Planning, execution, and recovery within a single tower-building episode. From left to right, the model grounds the task in visual and geometric observations, constructs a plan, and builds the tower through direct tool calls and generated programs. Action previews and execution feedback support a preview–execute–verify loop. After a failed grasp, the model inspects the scene, updates its plan, and regrasps the object to complete the tower. The harness provides measurements and execution checks; the model selects actions and recovery strategies. The scene is never reset.

Method / setting	Task exploration	Cross-trial refinement	Task code / skills	Trial-derived memory	Demo-derived artifacts	Demo references
	Before evaluation		Available to the test agent
GPT-6 Astra (zero-shot) [ 60 ]	✗	✗	✗	✗	✗	✗
GPT-6 Astra (image-demo ICL) [ 60 ]	✗	✗	✗	✗	✗	✓
GPT-Policy (video ICL) [ 5 ]	✗	✗	✗	✗	✗	✓
GPT-GEAR (full-context ICL) [ 64 ]	✗	✗	✗	✗	✓	✓
SenseNova (RoboDojo ICL) [ 42 ]	✗	✗	✓	✗	✓	✓

Table 1: Target-task preparation and evaluation inputs across the specified settings.

	Average		Generalization		Precision		Long-Horizon		Memory		Open
Method	SR	Score	SR	Score	SR	Score	SR	Score	SR	Score	SR	Score
Direct object motion (privileged)	76.2	79.6	81.7	83.3	65.0	72.0	70.0	75.4	96.7	96.7	67.5	70.6
RoboProbe [ 60 ]	22.5	28.5	31.7	34.1	2.5	8.5	7.5	22.1	40.8 §	45.8 §	30.0	32.1
Auriga	59.2	64.6	43.3	46.2	50.0	55.6	45.0	59.5	90.0	90.5	67.5	71.2

Table 2: Planning–execution gap on 210 episodes, five for each of the 42 tasks: SR (%) and Score (0–100) per dimension and their average over the five dimensions, as in Tab. 4 . With direct object motion, the model moves objects through a privileged action instead of the arm. All methods use GPT-6 Astra, RoboProbe at medium and the others at high reasoning effort. § Memory over 29 episodes, since one episode has no counterpart in the reference results.

Figure 3: General-purpose harness evolution of Auriga across 13 capability iterations ( #0 to #12 ). The horizontal axis lists the task-agnostic tools and harness mechanisms introduced at each iteration, and the vertical axis tracks the resulting performance (overall Score on RoboDojo). Larger nodes, labeled with their Score and gain, mark major capability milestones: motion preview ( #2 ), unified dual-arm actions ( #3 ), harness refactoring with verbatim task specifications ( #4 ), calibrated 3D geometry and history retrieval ( #6 ), ROI zoom ( #7 ), stage tolerances and obstacle checks ( #9 ), and explicit motion control ( #11 ), culminating in the frozen Auriga harness with VGGT-Omega and SAM 3.1 perception ( #12 , star marker: 64.0 Score, 57.7% SR across 6,300 episodes). Top-left inset: 42-task Score by capability dimension from iteration #4 to the full frozen harness.

Figure 4: Overview of Auriga. Top: The model maintains a task belief and an evidence record, uses observations to localize targets, and previews actions while the world is paused. Every physical motion passes through the admission gateway $\mathcal{A}$ , which returns a receipt and a new observation. Bottom: Closed-loop execution on hang_mugs : (1) localization, (2) grasp preview, (3) execution, and (4) placement on the rack.

Capability	Role	Representative tools	World	Details
Measure	Return images, robot state, object estimates, and geometric or state quantities with their reliability	robot_observe , geometry_query , stereo_guide	–	Sec. 5.1
Act	Preview and execute multi-stage action sequences for one or both arms, or let simulated time pass	robot_action , robot_wait	$\mathcal{A}$	Secs. 4.1 , 4.2 and 5.2
Compute	Run programs for loops, search, and numerical work; their physical requests are reviewed by the model	program_run	$\mathcal{A}$	Sec. 4.1
Recall	Revise the task belief, retrieve earlier interactions, and record or query operation lessons	task_plan , history_* , operation_memory	–	Secs. 3.3 and 5.1

Table 3: Capabilities available to the model. Only physical calls advance the world, and they do so only through the admission gateway

\mathcal{A}

Panel of Figure 5 — Figure 5: The three nested loops of Auriga within one episode. The stage loop runs inside $\mathcal{A}$ and stops execution when a declared condition is violated; the model closes the action and belief loops. Deliberative calls extend the evidence record without advancing the world.

Figure 6: Preview and verification in two rounds, drawn on the head-camera image. In the first round, the previewed gripper path moves to the object, closes on it, and lifts it to a checkpoint placed by the model, where the robot pauses and the model receives a fresh wrist view (inset 1). The measured path follows the preview until a finger lands on the basket rim while the object is lowered. Once the gap exceeds the tolerance, the harness stops the arm and reports with a fresh view (inset 2), so the gripper does not open. In the second round, the model previews a revised motion from the stop that lifts clear of the rim and lowers into the basket, where the gripper opens (inset 3). The rim contact and the second round are illustrative.

Figure 7: Perception and action interfaces in Auriga. (a) Objects separated and placed in 3D from the head-camera image, and the position of a selected point from its pixel ray and a plane of known height or a second view. (b) The object estimates keep a held object at its pose in the hand at the grasp, and a measurement against this reference shows that it has turned in the hand. (c) A new plan that returns to where earlier motions failed carries a warning, and earlier frames remain retrievable. (d) A goal refers to the contact point on the fingers rather than the wrist, with a coarse tolerance in transit and a tight one at contact. (e) Both arms share one timeline, with declared waits, a speed per movement that slows near contact, and a gripper that opens during a movement.

Learned policies: WAM or VLA, alone or with an agent
		Average		Generalization		Precision		Long-Horizon		Memory		Open
Method	Type	Score	SR	Score	SR	Score	SR	Score	SR	Score	SR	Score	SR
Rex-M1 Preview	VLA	33.79	27.44	34.05	26.05	32.82	23.83	46.89	33.00	48.53	48.22	6.67	6.08
Simate-beta	VLA	33.95	27.96	35.09	27.95	34.35	26.92	57.84	43.42	33.33	33.00	9.12	8.50
Awomo-0.5	WAM	35.34	29.64	30.31	23.84	39.65	33.50	40.36	26.75	42.27	41.22	24.09	22.92
PhysicalRSI	Agent + VLA	36.27	31.38	21.30	15.62	38.05	32.50	46.37	37.33	46.74	46.56	28.88	24.92

Table 4: Results on the RoboDojo simulation benchmark, with external entries from the official leaderboard [ 41 ] ( Sec. 6.1.1 ), reporting Score (0–100) and SR (%) in two blocks ordered by ascending average Score: the eight leading entries that use a world action model (WAM) or a vision-language-action (VLA) policy, alone or with an agent, and four agent methods on the leaderboard (Agent, acting through tools or code) together with Auriga. In each column, the highest value is in bold and the second highest is underlined. † Uses gpt-6-astra .

Figure 8: Five-dimension profile of Auriga and seven other methods of Tab. 4 , in Score (a) and SR in % (b). Each axis runs from zero at the center to the value at its outer vertex, with four evenly spaced rings. Below each dimension name is the value of Auriga, followed after a slash by the highest value of the other methods where it is higher. Auriga is the thick blue line with the shaded area; the other agent methods are drawn with solid lines and the WAM / VLA methods with dashed lines.

Figure 9: Generalization of the entries of Tab. 4 on the 12 Generalization tasks under scene randomization, ordered by

G

. (a) Change in SR (percentage points,

\Delta\text{SR}=\text{SR}_{\mathrm{rand}}-\text{SR}_{\mathrm{std}}

) from standard to random scenes. (b)

G=\tfrac{1}{2}(S_{\mathrm{std}}+S_{\mathrm{rand}})-|S_{\mathrm{rand}}-S_{\mathrm{std}}|

, the mean of the standard- and random-scene Scores minus their absolute difference.

Pretraining kitchens: learned WAM or VLA policies
				Composite
Method	Type	Overall	Atomic-Seen	Seen	Unseen	All
$\pi_{0}$	VLA	14.8	34.6	6.1	1.1	3.6
$\pi_{0.5}$	VLA	16.9	39.6	7.1	1.2	4.2
WorldDreamer	WAM	35.3	66.3	26.7	9.0	17.9
Target kitchens: VLA policy and agent methods

Table 5: Success rates (%) on the 50 Target50 tasks of RoboCasa365 [ 32 ] , at seeds 1–5 per task (250 episodes), ordered by ascending Overall SR within each evaluation setting. Overall is the mean over the 50 tasks, and Composite All covers the 160 composite episodes. Agent + VLA methods hand contact-rich phases to the RLDX-1 policy. WorldDreamer,

\pi_{0.5}

, and

\pi_{0}

are evaluated in the pretraining kitchens, all other methods in the target kitchens. –: not reported. Bold: highest; underlined: second highest.

Figure 10: Success rates (%) of the methods of Tab. 5 on RoboCasa365 Target50, in descending order of Overall SR. Grey bars are learned policies, and light blue bars are agent methods that hand contact-rich phases to the RLDX-1 policy. PyRUA-Lean reports composite tasks only as a whole, so its two composite bars are outlined and both show that rate. Above the bar of Auriga is its difference from the highest other method of the group.

Variant	Plane intersection	Triangulation	Calibration and object estimates	Successes	Score	First grasps held (%)
No geometry tools	$\times$	$\times$	$\times$	30/50	67.1	65.3
No triangulation	✓	$\times$	$\times$	40/50	84.0	85.7
Full	✓	✓	✓	46/50	94.9	–

Table 6: Spatial measurement ( Secs. 5.1 and 4 ): successes over the 50 cases, mean Score, and the share of first grasps that hold (%). The highest successes and Score are in bold. ✓: available;

\times

: removed.

Variant	Clearance review	Divergence stop	Successes	Score	Stops per episode
No clearance review	$\times$	✓	31/50	69.1	1.20
No divergence stop	✓	$\times$	34/50	75.1	0.00
Full	✓	✓	36/50	79.8	–

Table 7: Collision handling ( Secs. 4.1 and 4.2 ): successes over the 50 cases, mean Score, and the stops per episode at a movement that exceeded its arrival tolerance (

\delta_{p}

\delta_{\theta}

). The highest successes and Score are in bold. ✓: available;

\times

: removed.

Variant	Preview	Checkpoints	Successes	Score	Pauses per episode
No preview	$\times$	placed by the model	36/50	74.7	0.96
Pause after every gripper action	✓	after every gripper action	37/50	78.7	13.82
Full	✓	placed by the model	42/50	88.5	–

Table 8: Preview and checkpoints in

\mathcal{A}

( Secs. 3.2 , 4.1 and 4.2 ): successes over the 50 cases, mean Score, and the checkpoint pauses per episode. The highest successes and Score are in bold. ✓: available;

\times

: removed.

Variant	Finger-referenced goals	Shared timeline	Successes	Score	Planning failures per episode
One arm at a time	✓	$\times$	35/50	74.8	1.14
Wrist-flange goals	$\times$	✓	37/50	77.1	3.42
Full	✓	✓	39/50	83.9	–

Table 9: Expressive actions ( Sec. 5.2 ): successes over the 50 cases, mean Score, and the planning failures per episode (requested stages that could not be planned). The highest successes and Score are in bold. ✓: available;

\times

: removed.

Variant	Programs	Physical actions from programs	Review and receipts	Successes	Score	Model calls per episode
Programs cannot act	✓	$\times$	✓	19/30	73.0	67.1
No programs	$\times$	$\times$	✓	19/30	75.3	66.0
Code as policy	✓	✓	$\times$	20/30	77.2	71.8
Full	✓	✓	✓	30/30	100.0	46.2

Table 10: Programs ( Secs. 3.2 , 4.1 and 2.2 ): successes over the 30 cases, mean Score, and the model calls per episode. The highest successes and Score are in bold. ✓: available;

\times

: removed.

System	Successes	SR (%)	Exclusive	Mean	Median	Batches / ep.	Steps /ep.	Steps / batch
	Task completion			Model calls / ep.		Physical execution
RoboProbe [ 60 ]	3/15	20.0	0	36.6	38	33.4	365.1	10.9
Auriga	14/15	93.3	11	25.4	24	3.1	336.2	107.3

Table 12: Task outcomes and execution profiles on 15 matched standard-variant T-block exchange ( swap_T ) scenes (detailed trace analysis in App. F ).

Panel of Figure 14 — Figure 14: Recovery from grasp failure and transport slippage across RoboDojo and RoboCasa365. Rather than repeating the failed command or discarding prior progress, the model updates the contradicted retention condition in $B_{t}$ and revises the contact region, active arm, or transport mode.

Panel of Figure 15 — Figure 15: Scope-matched replanning after contact and state discrepancies. Each revision addresses the specific assumption contradicted by the execution receipt or visual state—restoring a displaced container, re-posturing the wrist for rotational range, or overriding an ambiguous handle angle.

Panel of Figure 16 — Figure 16: 3D geometry-guided manipulation on matched scenes. Estimated 3D bounding boxes, orientations, and surface heights guide container alignment, below-rim grasps, and shelf clearance across both benchmarks.

Panel of Figure 17 — Figure 17: Failure modes after repeated corrections under multi-body coupling. When a recovery action displaces an already assembled stack or traps an object beneath a fixture rack, local motion completions do not restore cumulative task progress before the budget expires.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Tasks	What it covers	Task identifiers
Generalization	12	Manipulation across varied objects and layouts, including randomized scene variants	stack_bowls , push_T , pack_objects_into_box , fold_clothes , hang_mugs , sweep_blocks , pour_liquid_into_cup , make_toast , arrange_largest_number , sort_nesting_dolls_by_size , store_laptop_and_headphones , stack_blocks
Precision	8	Fine-grained manipulation under tight spatial constraints	fasten_screws , plug_in_charger , insert_tubes , pour_balls_into_vase , play_Xylophone , deposit_coin , insert_key , build_tower
Long-Horizon	8	Multi-step tasks that require completing several subgoals	put_bottles_into_dustbin , fill_pen_holder , classify_objects , play_tic_tac_toe , fill_egg_holder , organize_table , make_kong , play_stacking_toy
Memory	6	Tracking state, recalling a sequence, or matching after a delay	cover_blocks , match_and_pick_from_conveyor , swap_blocks , swap_T , press_by_number , imitate_sorting_sequence
Open	8	Open-ended tasks and manipulation conditioned on language or images	align_blocks , general_pickup , stack_blocks_by_language , solve_equation , classify_objects_by_language , pick_from_conveyor_by_image , store_tools_in_toolbox , pour_by_language

Appendix

Table 13: The five capability dimensions of RoboDojo and their tasks.

Table	Tasks
Tab. 6	Generalization: stack_blocks , stack_bowls , push_T ; Long-Horizon: play_tic_tac_toe ; Memory: cover_blocks , press_by_number , imitate_sorting_sequence ; Open: align_blocks , stack_blocks_by_language , classify_objects_by_language
Tab. 7	Generalization: stack_bowls , sort_nesting_dolls_by_size ; Precision: fasten_screws , insert_tubes ; Long-Horizon: classify_objects , play_stacking_toy ; Memory: cover_blocks , swap_T ; Open: classify_objects_by_language , align_blocks
Tab. 8	Generalization: fold_clothes , arrange_largest_number , sort_nesting_dolls_by_size ; Precision: fasten_screws ; Long-Horizon: classify_objects ; Memory: swap_T , cover_blocks , imitate_sorting_sequence ; Open: pick_from_conveyor_by_image , classify_objects_by_language
Tab. 9	Generalization: fold_clothes , arrange_largest_number ; Precision: fasten_screws , play_Xylophone ; Long-Horizon: classify_objects , play_tic_tac_toe ; Memory: press_by_number , cover_blocks ; Open: classify_objects_by_language , align_blocks
Tab. 10	Generalization: sort_nesting_dolls_by_size , pack_objects_into_box ; Precision: fasten_screws ; Long-Horizon: play_stacking_toy , play_tic_tac_toe , classify_objects ; Memory: press_by_number , imitate_sorting_sequence ; Open: pick_from_conveyor_by_image , classify_objects_by_language

Appendix

Table 14: The ten tasks of each ablation table by dimension: five standard layouts per task (50 cases) in Tabs. 6 to 9 , and 30 cases in Tab. 10 .

Method	Type	Overall	Atomic-Seen	Composite-Seen	Composite-Unseen
$\pi_{0}$	VLA	14.8	34.6	6.1	1.1
$\pi_{0.5}$	VLA	16.9	39.6	7.1	1.2
WorldDreamer	WAM	35.3	66.3	26.7	9.0
Xiaomi-Robotics-1	VLA	57.4	80.2	57.1	32.1

Appendix

Table 15: Success rates (%) of four entries of the official RoboCasa365 leaderboard [ 40 ] on the 50 Target50 tasks, evaluated in the pretraining kitchens and ordered by ascending Overall SR. Overall is the mean over the 50 tasks.

Figure 18: Success rates (%) on RoboCasa365 by group: Auriga, RPent, and RLDX-1 from Tab. 5 (target kitchens, 250 episodes) and the leaderboard entries of Tab. 15 (pretraining kitchens), with Auriga first. In each group, the bar of Auriga shows its rate and, above it, its difference from the highest other method of Tab. 5 ; another method’s bar is labeled with its rate only where it reaches that of Auriga ( Tabs. 5 and 15 list all values).

Figure 19: Outcome of every episode of Auriga on the 250 Target50 episodes, by task and seed: success, failure after the 100 model requests of the episode were used, and failure after the model stopped acting for several turns. Within each group, tasks are ordered by their number of successes.

Figure 20: Score (top) and SR in % (bottom) of Auriga, the seven other methods of Fig. 8 , and RoboProbe, on average and by dimension. Auriga is the first bar of each group, followed by the other methods in order of average Score. Above the bar of Auriga are its value and, higher up in grey, its difference from the highest of the other eight methods, whose value is given above its own bar where it is higher than Auriga.

Dimension	Episodes	Model	Input	Cached	Output	Steps	Time
		calls	(M)	(%)	(k)	used (%)	(min)
Generalization	1,800	54.2	6.57	84.8	21.9	79.6	21.6
Precision	1,200	50.2	6.02	85.9	19.5	86.9	20.2
Long-Horizon	1,199	54.5	6.70	85.2	21.8	88.8	23.0
Memory	900	30.0	3.11	85.5	9.0	68.9	11.3
Open	1,200	40.7	4.67	86.1	15.4	71.8	14.9

Appendix

Table 16: Cost of an Auriga episode by dimension across the 6,299 evaluated episodes: episodes, mean model calls, input tokens and their cached share, output tokens, the share of the native-step limit used, and wall-clock time.

	SR (%)				Score
Method	Standard	Random	Change	Rel. change	Standard	Random	Change
PhysicalRSI	25.67	5.56	$-$ 20.1	$-$ 78%	31.57	11.04	$-$ 20.5
VPP2	46.44	12.56	$-$ 33.9	$-$ 73%	52.14	20.63	$-$ 31.5
Awomo-0.5	30.89	16.78	$-$ 14.1	$-$ 46%	37.70	22.92	$-$ 14.8
Rex-M1 Preview	34.78	17.33	$-$ 17.5	$-$ 50%	42.64	25.46	$-$ 17.2
GO-3 Nano Preview	43.00	24.33	$-$ 18.7	$-$ 43%	45.93	26.97	$-$ 19.0

Appendix

Table 17: Generalization under scene randomization on the 12 Generalization tasks for the entries of Tab. 4 : SR (%) and Score on the standard and random scenes, with the change (random

-

standard) and relative SR change. Above Auriga, entries are ordered by ascending composite generalization score

G

( Fig. 9 ). Bold: best entry per column.

	Auriga		SR (%) of the four other agent methods
Task	SR	Score	PyRUA- Lean	HexaShell- Preview	EmbodiedEvo	RoboProbe
Generalization
stack_bowls	91.3	92.5	70.0	86.0	56.0	62.0
push_T	86.0	86.0	86.0	96.0	86.0	60.0
pack_objects_into_box	14.7	27.3	30.0	16.0	0.0	0.0
fold_clothes	69.3	74.5	52.0	40.0	82.0	72.0

Appendix

Table 18: Per-task results of Auriga (SR in % and Score) and per-task SR (%) of the four other agent methods in Tab. 4 ( Sec. 6.1.1 ). Bold: highest SR of a task (none where all five are at 0.0).

Step	Recorded state and feedback	Decision or verified outcome
72	Both blocks are lifted; the belief records retention as supported and the exchange as pending .	Preserve both initial outlines; place pink first and inspect its release.
169	Pink has been released at blue’s initial pose; blue remains to be transferred.	Keep the completed placement and move blue with the right arm.
278	The transfer motion completes, but the final aperture is zero and the image shows blue on the table.	Mark blue retention as contradicted ; preserve pink as supported ; switch recovery to the left arm.
332	The left-arm re-grasp and slow lift complete; the right arm returns.	Use a short, slow placement at the saved target, followed by release and withdrawal.
380	The evaluator reports task success within the 400-step budget.	Both object goals are satisfied after the local recovery.

Appendix

Table 19: Task state, execution feedback, and correction in a successful T-block exchange. Entries condense the recorded task belief and actions; steps are cumulative native actions from receipt budget counters. The destination remains the opposite block’s initial pose throughout.

Mechanism	Interface	Episodes	Count and unit
Task-belief updates	task_plan	15/15	25 updates
Geometric queries	geometry_query	15/15	17 calls
Preview evaluations	robot_action	15/15	106 evaluations
Generated action plans	robot_action	15/15	74 plans
Multi-stage plans	move , gripper , observe	15/15	73 plans
Plans involving both arms	arm	15/15	54 plans

Appendix

Table 20: Mechanism use in the 15 T-block exchange episodes. Belief updates exclude the initial submission. Preview evaluations include unsuccessful candidates; generated plans are the candidates returned with a plan identifier. Plan and guard counts refer to specifications, including revisions. Observation pauses count actual receipt-reported stops.

Candidate	Native steps	Status
Initial correction	95	Rejected: insufficient budget
Revised route	57	Rejected: insufficient budget
Coordinated approach and return	54	Planned
Final plan with explicit release	55	Planned
Executed correction to task success	51	Success at step 396

Appendix

Table 21: Recorded preview costs during the late orientation correction. The world remains at native step 345 during these previews, with 55 steps available. The first two candidates are rejected because their planned lengths exceed the remaining 55-step native budget. Candidate rows give planned lengths; the last row gives steps actually used before the evaluator signals success.

Figure 23: Recovery from grasp failure and slippage in RoboCasa365 drawer-to-counter transfer ( PickPlaceDrawerToCounter ). After grasps on the thin ladle handle slip during lifting and base motion, the model pulls the ladle forward and switches contact to the wider bowl.

Panel of Figure 24 — Figure 24: Additional RoboDojo episodes replanning after contact and execution disruptions. The model supports the basket during cup placement and corrects the charger’s alignment before insertion.

Panel of Figure 25 — Figure 25: Additional RoboCasa365 episodes replanning after contact and execution disruptions. The model changes the approach to the drawer handle and then switches to the adjacent drawer when the first is empty ( DeliverStraw ), and reopens a drawer pushed inward by arm contact before approaching from the side ( SetUpCuttingStation ).

Agentic AI for Robotics

See all 54 papers

Feb 13, 2026

UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs

Achieving general-purpose robotic manipulation requires robots to bridge high-level semantic intent with low-level physical interaction in unstructured environments. However, zero-shot generalization remains challenging: end-to-end Vision-Language-Action (VLA) models often lack the precision required for long-horizon tasks, while traditional hierarchical planners suffer from semantic rigidity under open-world variations. We present UniManip, a unified framework centered on an Agentic Operational Graph (AOG), an explicit cognitive and operational backbone for task orchestration, state tracking, execution verification, and closed-loop failure recovery. High-level semantic intents are grounded into actionable primitives via a physically grounded scene graph that provides geometric affordances, contact-aware interaction parameters, and collision-free trajectory synthesis. By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, UniManip continuously aligns abstract planning with geometric constraints, enabling robust execution. Unlike static pipelines, UniManip operates as a dynamic agentic loop: it instantiates object-centric scene graphs from unstructured perception, parameterizes these representations into collision-free trajectories via a safety-aware local planner, and exploits structured memory to autonomously diagnose and recover from execution failures. UniManip achieves state-of-the-art performance on LIBERO-Pro, while extensive real-world experiments demonstrate robust zero-shot generalization across diverse tasks, embodiments, and dexterous hands, outperforming leading VLA models and hierarchical baselines by 22.5% and 25.0% in task success rate, respectively. Open-source project page: https://henryhcliu.github.io/unimanip.

Haichao Liu, Yuanjiang Xue, Yuheng Zhou +5

School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Automation, Tsinghua University, China

Robotic Manipulation Long-Horizon Robotic Manipulation Zero-Shot Robotic Manipulation

Ask PDF

Sep 11, 2026

Agent as Policy for Robotic Manipulation

We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves success rates of at least 80% in seven of eight task configurations and significantly outperforms previous agentic robot systems. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.

Mengzhao Jia, Yang Lin, Xixin Zhang +3

University of Notre Dame · University of California San Diego · San Diego State University

Robotic Control Robot Policy Learning Generalization in Robotic Manipulation Robotic Manipulation Robot Task Planning

Ask PDF

Sep 27, 2026

CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation

How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce CodeActionBench, a benchmark of 25 manipulation tasks that evaluates this capability through agentic Code-as-Policy. Without task-specific fine-tuning, demonstrations, external specialist perception or grasp modules, privileged scene state, or predefined task policies, agents should select visual evidence, form task-relevant 3D estimates, construct manipulation targets, and iteratively execute and revise their policies. A shared robot API provides RGB observations, calibrated geometric operations, robot feedback, and bounded motion, leaving task-dependent decisions to the evaluated agent. Fixed task instances, resource budgets, and a hidden physical-outcome verifier support controlled comparisons across models and harness configurations. Extensive evaluations across nine configurations and 675 attempts achieve success rates ranging from 2.7% to 73.3%. The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts, demonstrating the best performance while still leaving substantial room for improvement. Trajectory analyses reveal difficulties in spatial alignment, object retention, and completion judgment, including task failures despite successfully completed motions. CodeActionBench provides a controlled testbed for measuring how general-purpose models translate their capabilities into manipulation behavior and for examining typical failure scenarios in that process.

Yiheng Lyu, Xueying Jiang, Wenhao Li +2

Nanyang Technological University, Singapore · Independent Researcher

Robotic Policy Evaluation Robotic Manipulation Robot Manipulation Benchmarks VLMs for Robotics Embodied Agent Benchmarks

Ask PDF

Abstract

Agentic AI for Robotics

Explore similar work

UniManip: Open-World Robotic Manipulation via Agentic Operational Graphs

Agent as Policy for Robotic Manipulation

CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation