Recent work on robot manipulation has focused on specialist policies trained on demonstrations. We show that generalist AI models can control manipulators more reliably than specialist policies such as vision-language-action models (VLAs) and world-action models (WAMs). We systematically analyze agentic manipulation policies (AMPs), in which a generalist AI model commands the manipulator directly, adapting its actions as the task unfolds. We present AMP1, a reference harness that runs unchanged across single-arm, bimanual, and mobile manipulation. AMP1 achieves state-of-the-art results on the RoboLab-120 and CaP-Gym benchmarks, with no demonstrations or task-specific training. On RoboLab-120, AMP1 reaches an 85.6% success rate zero-shot, compared to 49.2% and 42.9% for leading VLA and WAM policies. The agent succeeds zero-shot in 104 of 108 real-world trials spanning twelve tasks and three robot instances, reducing failure rates by 20x and 15x relative to strong VLA and WAM baselines. The basic harness supplies no grasping, pushing, or placing skills; the agent develops its own strategies, such as searching for hidden objects, using objects as tools, and calibrating its cameras. With the addition of a persistent memory store, the agent can also learn from experience: an augmented harness with a memory representation that is carried across episodes lowers the failure rate on RoboDojo bimanual manipulation tasks by 4.6x relative to zero-shot deployment. Our analysis suggests that AMPs are substantially more general and robust than specialist manipulation policies, providing a promising foundation for robot manipulation.
Figures & tables
Figure 1: Zero-shot real-world manipulation with AMP1. AMP1 solves diverse, long-horizon manipulation tasks directly in the real world from a natural language instruction without task-specific fine-tuning or predefined skills. It devises complex strategies, such as ( A ) searching for hidden objects, ( B ) using a balance scale as a measuring instrument, ( C ) planning and executing a winning strategy in a game of chess, ( D ) combining non-prehensile and prehensile actions where a direct grasp is infeasible, and ( E ) manipulating articulated objects such as a kitchen cabinet door. Task instructions given to AMP1 for each episode are shown below the corresponding image sequence.
Figure 2: Simulation evaluations. ( A ) We evaluate AMP1 on four open-source simulation benchmarks: RoboLab-120 [ 22 ] and the LIBERO-PRO, robosuite, and BEHAVIOR-1K suites of CaP-Gym [ 18 ] . The tasks span short and long horizons, one and two arms, fixed and mobile bases, and prehensile and non-prehensile manipulation. ( B ) Success rates on RoboLab-120 with default ( ∙ ), vague ( ▼ ), and specific ( ▲ ) instructions. Numbers on the right show the success rate with default instructions. AMP1 has the highest success rate on the benchmark and the smallest relative spread across the three instruction conditions among entries that report all three ( ∗ not an official submission on the RoboLab-120 leaderboard; ‡ our evaluation on the vague and specific instruction conditions). ( C to E ) Evaluation on the CaP-Gym suites, zero-shot and with the same harness as on RoboLab-120 ( ⧫ : zero-shot, without training on the benchmark or rollouts of the evaluated tasks before testing; ◊ : few-shot, learning from rollouts of the evaluated tasks; □ : fine-tuned on the benchmark’s demonstrations; dashed lines: programs written by human experts [ 18 ] ). ( C ) On LIBERO-PRO, AMP1 achieves a 95.2% success rate, outperforming the zero-shot CaP-Agent0 baseline as well as fine-tuned VLAs and all but one few-shot agent baselines. ( D ) On robosuite, which includes both single-arm and bimanual tasks, AMP1 achieves a 100.0% success rate zero-shot, outperforming all baselines. ( E ) On the two BEHAVIOR-1K mobile manipulation tasks in CaP-Gym, AMP1 achieves a 96.0% success rate, outperforming the zero-shot CaP-Agent0 baseline by a wide margin and matching or exceeding all but one few-shot baselines.
Task features
Success
Average time to success (seconds)
Task Instruction
Manip.
Horizon
Reasoning
π0.5
FLUX
AMP1
π0.5
FLUX
AMP1
Exec.
Reason.
Total
Put the rubiks cube to the left of the bowl
P
SS
SR
5 / 9
9 / 9
9 / 9
18
20
22
72
94
Put the cube and the banana in the bowl
P
MS
–
9 / 9
9 / 9
9 / 9
26
30
44
138
182
Put all the red things in the grey bin
P
MS
SG
5 / 9
7 / 9
9 / 9
53
39
46
146
192
Pack canned foods into the bin
P
MS
SG
2 / 9
7 / 9
9 / 9
78
40
54
154
208
Table 1: Real-world trials. Evaluation on diverse tasks on physical hardware. ‘Average time to success’ is the mean over successful trials only, in seconds from the start of the episode to task completion. A dash (–) indicates that the method did not succeed on any trial of that task. The timing for AMP1 is decomposed into execution (Exec.) and reasoning (Reason.). AMP1 is paired with Opus 5.5 for these experiments. AMP1 is evaluated with robot error recovery enabled; the baseline deployments terminate trials when robot controller errors interrupt motion. An ablation without error recovery is reported in Tab. S1 . The legend below the table lists the various task features.
Figure 3: Emergent behaviors in AMP1. Four episodes in which AMP1 overcomes an unforeseen challenge: ( A to C ) on the real robot and ( D ) in RoboLab simulation. Quotes under each image are from the agent’s thinking traces. ( A ) The target is hidden from view, so AMP1 reasons about where it could be hidden and moves the occluding object to look. ( B ) The bowl offers no viable grasp, so AMP1 moves the plate against the bin instead and pushes the bowl off the higher bin edge onto the plate. ( C ) The bottle is out of the arm’s reach, so AMP1 uses a wooden spoon to tip it over toward itself, then grasps it by the cap. ( D ) The pomegranate is too wide to grasp, so AMP1 presses it against the wall of the crate and rolls it up the wall. ( E ) Skills that AMP1 writes for itself, anew in each episode, and the share of the 1,200 RoboLab episodes in which each appears. † Appears only in its reasoning, not as code.
Figure 4: Learning from experience. (A) Example RoboDojo simulation tasks. ( B to E ) Few-shot learning on the Mahjong task. (B) AMP1+ must push over the three tiles that match the opponent’s tile, take a tile from the stack and stand it at the right end of the row. (C, D) Comparing two episodes from the same round ( k=2 ); from the binary success signal (one episode failed, the other succeeded), the agent learns that it must stand the tile the right way up. (E) Successful practice episodes per round. AMP1+ records the right-way-up rule after round 2 ① and a rule about when to open the gripper after round 3 ②, and stops practicing after round 6. ( F to H ) Few-shot learning results. After practice, AMP1+ reaches a success rate of 84.1%, up from 26.4% zero-shot. Practice is run five times per task, and the run that succeeds most often is evaluated. (F) Success rate after k rounds of practice. (G) Number of model calls on successful episodes, relative to zero-shot deployment. (H) Success rate of AMP1+ and the top entries on the RoboDojo simulation leaderboard. ( I to K ) Continual learning across tasks, from a single run over one random permutation of the tasks. (I) Success rate through continual learning. (J) Number of model calls relative to zero-shot deployment. (K) Per-segment success rates at the end of each segment, colored by whether the segment has not yet been seen (green), was just seen (yellow), or was seen earlier (blue). In (F, G, I, J) , shaded bands show 95% confidence intervals across tasks. All evaluations are on unseen layouts.
Figure 5: Controlled experiments on RoboLab-120. ( A ) Success rate of 13 models with less and more reasoning, using the basic harness ( Sec. 2D.1 ). ( B ) Success rate with the basic and assistive harnesses, each model at its more-reasoning setting from A ( Sec. 2D.2 ). ( C ) Sensory ablation for four Claude models with all cameras, using the basic harness: removing the perception tools (RGB+C+D), then depth (RGB+C), then calibration (RGB), and finally replacing images with textual descriptions (Text). ( D ) Opus 5.5 with different camera constellations.
Figure 6: AMP1 harness. The generalist AI model controls the robot via Python code that runs in a persistent interpreter. The generalist model builds its own skills on top of the harness SDK. The SDK’s functions fall into three groups: sensing, perception, and motion. The task instruction and all sensory input reach the model via the harness. In addition to the interpreter, the model has a number of auxiliary tools, such as viewing a saved image and terminating the episode (not shown for clarity). The memory (shown dashed) is used when learning from experience ( Sec. 2C ).
Figure S1 : Three instances of our DROID-like physical evaluation platform. The instances use different tabletop backgrounds, among other unit-level differences. The two shoulder cameras (ZED 2i) are annotated in red and the wrist camera (ZED Mini) in blue.
Figure S2 : The 12 real-world evaluation tasks. Each panel shows the initial scene of one task, labeled with its language instruction. Note that the images only show initial conditions from one platform instance, whereas the evaluations are conducted on all three platform instances shown on Fig. S1 independently.
Task Instruction
Success
Average Time to Success (seconds)
AMP1
AMP1
AMP1
AMP1 (w/o RfE)
(w/o RfE)
Exec.
Reason.
Total
Exec.
Reason.
Total
Put the rubiks cube to the left of the bowl
9 / 9
9 / 9
22
72
94
21
67
88
Put the cube and the banana in the bowl
9 / 9
9 / 9
44
138
182
50
151
201
Put all the red things in the grey bin
9 / 9
8 / 9
46
146
192
54
156
210
Pack canned foods into the bin
9 / 9
3 / 9
54
154
208
43
114
157
Table S1: Ablation of recovery from error (RfE) in the real-world evaluations. Success (successes/trials) and average time to success of AMP1 and of a variant without RfE (w/o RfE) on the 12 tasks of Tab. 1 . In both conditions, the Franka arm’s collision reflex interrupts motion. Without RfE, we terminate the trial; with RfE, the agent can autonomously request error recovery through the harness and continue if recovery succeeds. Time to success is averaged over successful trials only.
Figure S3 : Additional emergent behaviors in AMP1 (continued from Fig. 3 ). Four more episodes in which AMP1 works around an unforeseen obstacle with a strategy its prompt does not describe: ( A to C ) on the real robot, ( D ) in RoboLab simulation. Quotes under each image are from AMP1’s thinking trace. ( A ) The bowl is too wide to grip and sits flat on a bin, so AMP1 slides it until its edge sticks out past the bin, then grips that edge from the side (cf. Fig. 3 B). ( B ) The cubes start in a tightly packed grid that leaves no room for the fingers, so AMP1 first pushes a column of the grid to make space. ( C ) The gripper does not line up with the knob in the wrist camera, which suggests a calibration error, so AMP1 closes its gripper to measure the offset and corrects for it in the later subtasks. ( D ) The keyboard is too long to set down flat in one motion, so AMP1 first rests one end on the table, then lowers the other along an arc.
Figure S4 : Additional emergent behaviors in AMP1 (continued). Three more episodes in which AMP1 works around an unforeseen obstacle with a strategy its prompt does not describe, all in RoboLab simulation. Quotes under each image are from AMP1’s thinking trace. ( A ) The mug lies on its side, so AMP1 hooks its closed fingers into the mug’s opening and pivots it upright. ( B ) While reaching toward the shelf, the arm is blocked by a structure none of its cameras can see, so AMP1 infers an upper shelf beam from the blocked motion and keeps its arm below it. ( C ) The drill keeps slipping from friction grasps, so AMP1 switches to gripping the handle so that the drill head rests on top of the fingers.
Figure S5 : One episode, turn by turn. AMP1 with Claude Opus 5.5 at high reasoning effort and the basic harness solves the BananaInBowlTask RoboLab task (“Pick up the banana and place it in the bowl”). Each panel is one turn: AMP1 reasons (yellow), implements and runs Python code (blue), and receives the execution output (green), which it reads at the start of the next turn.
Skill
Description
Code
Episodes (of 1,200)
Scene reasoning skills
Measure container
Measures a container’s rim height, inner width, and floor height before placing an object in it.
✓
66% (792)
Reason about object’s axes
Finds an object’s long and short axes from its points, typically to grasp it across its short axis.
✓
47% (560)
Find free placement space
Finds a free space for each object so that placed objects do not overlap.
✓
40% (476)
Ordering and stacking
Plans the order of moves when stacking / handling several objects.
–
12% (149)
Identify objects based on their category
Decides which objects fit a category in the instruction.
–
7% (82)
Table S2 : Skills AMP1 implements for itself. Skills extracted from the AMP1 run on RoboLab-120 (Opus 5.5, high reasoning effort, 1,200 episodes), grouped into scene reasoning, motion, and diagnostics. A skill is listed only if it appears in at least five episodes across at least two tasks.
Figure S6 : Wrist-camera views collected by AMP1 to calibrate the camera, rendered from a fixed scene camera. Each frustum is the wrist camera’s true pose where a view was taken. Dots mark SIFT features matched across the images. ( A ) First calibration round. Five views collected after known shifts, a descent and a yaw of the gripper. ( B ) Second calibration round. Two tilted views, added after AMP1 found that the views in ( A ) leave the camera’s offset along the tool axis unobservable.
Figure S7 : 3D estimation from the self-calibrated wrist camera. Each frustum is a wrist view that AMP1 triangulated from, linked to its image, in which AMP1’s picked pixels are circled. Rays from the views meet at the triangulated points. ( A ) The top centers of the tuna can (green) and the soup can (red), from six calibration views. ( B ) The bin’s two far rim corners (yellow) and the mustard bottle’s cap (green), from two views over the bin.
Figure S8 : Scene perception with only a wrist camera. The task is “Put all the cubes and blocks in the bin.” Each column is one stage before AMP1 decides to grasp the first target object: ( A ) the start pose, ( B ) the overview at control step 60, and ( C ) the first close look at control step 86. The top row is what the wrist camera sees, with a quote from AMP1’s thinking trace at that view. The bottom row shows the same moment in 3D space: the camera (dark dot), the space it can see, the path it has taken so far (blue, with arrows for the direction), and the objects AMP1’s perception tools have reported by that point. In ( A ), only part of the table is in view, so AMP1 raises the camera for a broader view. In ( B ), all seven targets and the whole bin are perceived, but their boxes are rough, so the agent decides to hover above each object before grasping it. In ( C ), objects are measured again after the gripper hovers 0.30 m above the table. AMP1 uses the cube’s newly measured orientation to choose a closing angle that keeps a finger off the neighboring blue block.
Figure S9 : Robustness to instruction phrasing on RoboLab-120. Success rate across the default (circle), vague ( ▼ ), and specific ( ▲ ) instruction conditions for all seven AMP1 configurations and 12 baseline policies evaluated across all three conditions, ranked by relative spread from lowest (most robust) to highest. Shaded bars span the range from minimum to maximum success rate across the three conditions. Columns on the right report each policy’s default success rate (%), absolute spread ( max−min , in percentage points), and relative spread ( spread/default success rate ). ‡ Evaluated by us using the released checkpoint on the vague and specific instruction conditions (1,200 trials per condition), with default from the leaderboard.
Figure S10 : Outcome and failure breakdown across all 1,200 RoboLab-120 trials. ( A ) AMP1 with Gemini 4 Argon (1,027 succeeded, 173 failed). ( B ) AMP1 with Opus 5.5 (994 succeeded, 206 failed). In each panel, all 1,200 trials in the ‘default’ instruction condition (10 trials across 120 tasks) branch into succeeded and failed trials; failed trials then branch by whether the failure originates in the policy ( AMP1 failure ), the benchmark ( Eval failure ), or the model API ( Provider failure ), and finally into specific causes. Band widths are proportional to the number of trials at a single scale shared across ( A ) and ( B ). Roughly two-thirds of all failed trials – 121 of 173 for Gemini 4 Argon (70%) and 135 of 206 for Opus 5.5 (66%) – are evaluation failures caused by objects that are incompatible with a standard parallel-jaw grasp ( Object defect ) or by flawed verifiers that reject valid end states ( Flaky verifier ).
Model
Input
Cache write
Cache read
Output
Gemini 4 Argon
4
4
0.2
20
Opus 5.5
4
5
0.2
20
Sonnet 5.5
2
2.5
0.1
10
Haiku 5.5
0.10
0.125
0.01
0.5
Opus 5
5
6.25
0.5
25
Opus 4.5
5
6.25
0.5
25
Table S3 : List prices of a number of models. ($ per million tokens). Thinking is billed as output. Gemini 4 Argon has no separate cache-write price; its first pass is billed as input.
Figure S11 : Cost, model calls, and token use on RoboLab-120. (Per-episode averages.) Left column: less vs. more reasoning (basic harness). Right column: basic vs. assistive harness (more reasoning). ( A , D ) Success rate vs. cost. ( B , E ) Model calls vs. cost. ( C , F ) Generated (output and thinking) vs. input tokens (uncached, cache write, and cache read; log scale). Error bars on cost and success rate are 95% confidence intervals. Costs at list price ( Tab. S3 ).
Vision-language models can interpret instructions and plan manipulation tasks, yet struggle when acting directly as robot policies. On RoboDojo, a large planning-execution gap separates high-level task decisions from physical execution: removing arm execution raises GPT-6 Astra's success rate from 22.5% under the reference control interface to 76.2%. We present Auriga, a task-agnostic agent harness that closes 68.3% of this gap in zero-shot manipulation without target-task trials or resets. In Auriga, the model makes all task decisions over an explicit task belief and acts through either direct tool calls or synthesized multi-step Python programs, while the harness supplies calibrated 3D measurements, finger-referenced dual-arm control, and a single admission gateway to the robot. Before motion, the model inspects a visual trajectory preview while simulation is paused; during execution, the harness checks declared stage tolerances and grasp bounds, stopping the robot on divergence and returning the measured discrepancy for in-place recovery. Developed across 13 cross-task capability iterations and frozen for evaluation, Auriga ranks first on the RoboDojo leaderboard with a 57.7% success rate (64.0 Score), more than doubling the reference harness on the same model, and remains first under scene randomization while learned policies drop by up to 33.9%. On RoboCasa365, the same frozen harness reaches a 73.2% success rate, outperforming VLA-assisted agents by up to 23.1% on composite tasks. Project page: https://auriga.stelledge.com/ Code: https://github.com/lose4578/Auriga
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves success rates of at least 80% in seven of eight task configurations and significantly outperforms previous agentic robot systems. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
Mengzhao Jia, Yang Lin, Xixin Zhang +3
University of Notre Dame · University of California San Diego · San Diego State University
Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.