Robot learning has advanced by scaling demonstration datasets and model capacity; however, deploying a new robot embodiment or task typically necessitates a dedicated teleoperation corpus. The resulting policies remain bounded by their imitation distributions, limiting generalization and continual self-improvement. We introduce RAX (Robot Agent eXploration), an open-source framework for robot agents centered on scaling exploration: enabling agents to improve by acquiring online experience and synthesizing executable programs. In RAX, a general-purpose coding agent generates robot control code, operating without action demonstration data, hand-engineered action primitives, or value-function guidance. Through online exploration, the agent identifies decision sequences that achieve task goals and compiles them into modular programs encapsulating the full perception-planning-control loop. Using RAX, we present a systematic multi-embodiment study on scaling online exploration across precision and long-horizon manipulation tasks using single-arm, bimanual, dexterous, mobile, and humanoid systems in simulation and on real robots. Generalization performance on unseen environments improves consistently as exploration scales. In simulation, RAX outperforms top baseline models by +37.4% on average. On real robots, RAX autonomously learns memory-dependent, articulated, high-precision, deformable, and long-horizon mobile manipulation tasks without human demonstrations, continually improving performance by composing verified skills from prior exploration sessions. As exploration scales, we observe emergent capabilities including active perception, error recovery, safety guardrails, and human-robot collaboration. We open-source RAX as a generalist robot-agent platform in which new robot embodiments are integrated through their APIs alone, without teleoperated data collection or policy training.
Figures & tables
Figure 2: System architecture and self-improving loop of RAX. The coding agent, powered by a frozen LLM, interacts with physical hardware across an isolation boundary using multimodal perception and control SDK interfaces. By reflecting on execution outcomes, the agent continually updates a persistent memory system containing working context, task notes, and a versioned repository of verified skills. During exploration, the agent compiles its perception and planning logic into standalone Python control programs that execute high-frequency loops directly on the robot. When encountering perception or planning failures due to environment variations, the program raises an exception ( raise Unsure() ), delegating control back to the agent to make adjustments and refine the codebase.
Figure 3: Agentic 3D perception and motion planning on a real robot. (a) The articulated air fryer model constructed programmatically by the agent from point-cloud measurements and fit to RGB-D observations, consisting of a fixed chassis and a drawer articulated via a prismatic joint along pull direction d relative to central axis c . (b) Action trajectories planned on the instantiated model with the drawer opened by 13.1cm . End-effector poses are visualized as pyramids whose bases face the approach vector and whose thick edges indicate the finger closure axis. The underlying geometric estimator maintains high fidelity, yielding only a 6.5mm displacement between the predicted basket rim and the post-opening physical measurement. Figure 27 illustrates the corresponding execution sequence on the physical robot.
Figure 4: Frontend–backend separation. The agent and its programs act only through the tool interface; the backend owns control, sensing, and safeguards on every platform.
Figure 5: A long-horizon task on the real RB-Y1. The agent composes six sub-tasks from skills learned in earlier sessions. It finds the cups from Gemini’s rim points and depth, measures the desk edge by picking points in the head image, and plans the base routes on a model of the scene.
Platform
Low-level control
Agent action space
Model-based helpers
LIBERO, LIBERO-Pro single-arm Franka Panda
EEF pose control (OSC)
EEF position and rotation deltas; gripper command
Pixel-to-3D projection from depth
RoboDojo dual-arm ARX X5
Joint position control (PD)
Joint-target chunks for both arms and grippers; bounded EEF steps
Inverse kinematics; path checks against workspace and table
Table 1: Low-level control on each platform. EEF: end effector; OSC: operational-space control; PD: proportional-derivative.
Make toast
Hang mugs
Sweep blocks
Plug
Swap
Make
Pour by
Method
Budget
random
random
random
charger
blocks
kong
language
VPP2
–
38.0
13.5
1.3
16.7
26.0
78.7
0.0
Awomo-0.5
–
7.0
9.6
0.0
7.3
4.7
30.0
0.0
Simate-beta
–
7.7
11.4
0.0
29.3
38.7
76.0
0.0
Rex-M1 Preview
–
14.3
20.2
8.0
3.3
13.3
33.3
0.0
VPP2-Preview
–
13.7
14.5
2.7
6.0
22.7
34.7
0.0
Table 2: Scaling exploration on RoboDojo. Score (%) on unseen layouts; baselines from the top five policies of the RoboDojo-Sim leaderboard ( Chen et al., 2026a ) . Cells with more than one run report the mean. –: not run. † No program submitted. ‡ Agent + VLA; listed among agents and unranked on the leaderboard.
Boxing
Getting
Sorting
Clearing
Clean up
Loading
Method
books
organized
items
food
desk
the car
Avg.
Robot Learning Collective
0.0
2.0
6.2
8.0
12.7
20.0
8.2
Comet
0.0
0.0
0.0
0.0
0.0
0.0
0.0
SimpleAI Robot
0.0
0.0
0.0
0.0
0.0
0.0
0.0
The North Star
0.0
0.0
0.0
0.0
0.0
20.0
3.3
Embodied Intelligence ‡
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Table 3: BEHAVIOR-1K. Q-score (%) on the ten public test instances; baselines from the five highest-ranked entries of the 2025 BEHAVIOR Challenge ( Li et al., 2023 ) . † Fully automated program. ‡ Privileged track.
Figure 6: The program RAX writes for boxing books on BEHAVIOR-1K ( Table 3 ). It detects the books, shelves and box in the head camera, tracks the robot’s motion by visual odometry, maps the scene, and plans paths with A*; the policy flow shows the camera view at each step.
Human–object interaction
Human–scene interaction
Method
Demos
Budget
Football
DoubleDesk
P&PBox
OpenDoor
SitSofa
Boxing
VisNavi
Avg.
ACT
100
–
16.7
18.3
56.7
78.3
73.3
56.7
33.3
47.6
Diffusion Policy
100
–
45.0
36.7
75.0
85.0
78.3
76.7
23.3
60.0
Flow matching
100
–
13.3
38.3
73.3
70.0
15.0
70.0
38.3
45.5
π0.5
100
–
10.0
43.3
71.7
66.7
73.3
70.0
23.3
51.2
WB-WAM
100
–
70.0
65.0
86.7
98.3
95.0
81.7
76.7
81.9
Table 4: HumanoidArena. Success rate (%) on the 60 official test episodes of each task, all methods with the SONIC tracker. Baselines from Wang et al. (2026b) and Qin et al. (2026) . Max: each task at its largest budget. ⋆ Three demonstrations without actions. –: not run.
Single-arm
Bimanual
Method
Budget
Hammer
Mouse
Bucket
Tongs
Glasses
Water
iPad
Hanoi
Assembly
Microwave
Photo
Avg.
DP-T
–
81.3
62.0
83.3
22.7
53.3
84.0
8.0
24.7
4.7
73.3
56.7
50.4
DP-C
–
58.7
74.0
70.0
57.3
54.0
63.3
52.0
12.7
3.3
54.0
24.0
47.6
ACT
–
50.0
61.3
64.0
31.3
47.3
47.3
9.3
6.0
0.0
66.0
7.3
35.5
π0.5
–
84.7
64.7
84.0
24.0
72.0
88.7
36.0
23.3
2.7
60.0
54.7
54.1
GR00T N1.5
–
67.3
85.3
72.0
12.7
27.3
72.7
12.7
0.7
0.7
50.7
40.7
40.2
Table 5: DexJoCo, rand-obj setting. Success rate (%) on the 150 official evaluation episodes of each task. Baselines from Wang et al. (2026a) , Park et al. (2026) (GR00T N1.6, Fast-WAM, PointWAM), Zhang et al. (2026) , and Li et al. (2026) . DP-T and DP-C: Diffusion Policy with transformer and CNN backbones. † One policy for all tasks. ‡ Mean over the ten tasks other than iPad. § With simulated tactile input. –: not reported.
LIBERO
LIBERO-Pro, position perturbation
Method
Demos
Spatial
Object
Goal
Long
Avg.
Spatial
Object
Goal
Long
Avg.
OpenVLA
50
84.7
88.4
79.2
53.7
76.5
0.0
0.0
0.0
0.0
0.0
OpenVLA-OFT
50
97.6
98.4
97.9
94.5
97.1
–
–
–
–
–
π0
50
96.8
98.8
95.8
85.2
94.2
0.0
0.0
0.0
0.0
0.0
π0.5
50
98.8
98.2
98.0
92.4
96.9
20.0
17.0
38.0
8.0
20.8
X-VLA
50
98.2
98.6
97.8
97.6
98.1
0.0
2.0
1.0
0.0
0.8
Table 6: LIBERO and LIBERO-Pro. Success rate (%) on the 50 official initial states of each task and on the 50 position-perturbed states of LIBERO-Pro. Baselines from their papers, openpi ( π0 , π0.5 ), and the LIBERO-Pro leaderboard ( Zhou et al., 2025 ) ; PatchWAM reports only the mean, and VPP2 uses 10 initial states per task. † Developed on the perturbed setting under evaluation ( Section 12 ). ‡ Zero-shot with a skill library built on LIBERO-90; included in the average. –: not reported.
Agent
Agents
Envs
Budget
Time (h)
Success
Score
Codex
1
1
32
14.2
40.6
66.4
Codex
2
2
32
9.2
56.3
65.6
Claude Code
1
1
48
13.7
50.0
63.3
Claude Code
4
4
32
4.9
87.5
92.2
Table 7: Parallel exploration on make toast. Agent time until submission, and success rate and score (%) of the fully automated program on 32 unseen scenes.
Task
Budget
RGB-D
RGB-only
Δ
Make toast random
16
2.5
0.0
+ 2.5
Make toast random
32
31.7
7.8
+ 23.9
Hang mugs random
16
36.9
13.9
+ 23.0
Sweep blocks random
16
53.3
33.3
+ 20.0
Average
31.1
13.8
+ 17.3
Table 8: Depth versus RGB-only perception. Score (%) on unseen layouts at matched exploration budgets. Cells with more than one run report the mean.
Figure 7: From exploration to code. Inference time (left) and output tokens (right) of each successful exploration episode, on base-2 logarithmic vertical axes; for rehearsals, inference time is the program’s computation time as measured by the harness, without the overhead of running in isolation. The token axis is linear below one thousand tokens so that zero can be shown. Filled dots mark episodes in which the agent takes part, alone or together with its program; hollow circles mark rehearsals, in which the program runs alone. On loading the car, raw points are faint and the line is a running median over five successes; output tokens of parallel episodes are attributed to the environment an agent was assigned, or split evenly across the episodes active at that moment.
Task
Actions from
Inference
Robot time
Output tokens
Air fryer
Agent
72.3 min
2.9 min
262.4 k
Program
18 s
2.5 min
2.0 k
Bandu tower
Agent
17.2 min
9.3 min
88.4 k
Program
10 s
10.9 min
2.2 k
Table 9: From the agent’s actions to a fully automated program on the real Franka. The first success with actions generated by the agent and a later unattended success of its program. Inference time: wall-clock time outside robot calls. The agent’s Bandu rebuild places six pieces, the program seven.
Method
Demos
Budget
Football
DoubleDesk
P&PBox
OpenDoor
SitSofa
Boxing
VisNavi
Avg.
WB-WAM
100
–
70.0
65.0
86.7
98.3
95.0
81.7
76.7
81.9
RAX
3
32–128
98.3
56.7
83.3
100.0
96.7
100.0
93.3
89.8
RAX ⋆
3
64–256
98.3
46.7
70.0
95.0
100.0
86.7
76.7
81.9
Table 10: HumanoidArena without action demonstrations. Success rate (%) on the 60 official test episodes of each task. RAX: largest budgets of Table 4 . ⋆ Three demonstrations without actions.
Figure 8: Active perception. Simulation, DexJoCo bimanual Tower of Hanoi ( Section 11.2 ): (1) in its first episode, the agent moves single joints of the Allegro hand and finds the white fingertip caps in the wrist camera; (2) from the caps’ points it fits the fingertip offsets relative to the flange, and, without a model of the hand, its thumb tip lies 1.1 mm from the official model; (3) after each placement, the program measures the disk again, on average 0.66 mm below the height it predicted over 150 evaluation episodes. Real robot, ManipulationNet peg-in-hole ( Section 14.4 ): the agent probes with closed fingers to measure the height of the transparent board and the tilt of the table.
Figure 9: Error recovery and safety on real robots. Each incident and the rule the agent derived from it. RB-Y1: a turn planned by hand pushes a cabinet, after which every base move is checked against the area the base sweeps; squatting to lift a bucket overloads a wrist, and standing up first resolves it. Franka: an unusual arm posture hits a camera, after which the agent plans only natural postures that keep the arm away from the cameras.
Figure 10: Human–robot interaction. Requests that the agent sent to the operator, summarized, with the scene before and after. Right: the fryer poses that the agent requested across runs 6 to 10, from its own measurements of the fryer’s axis and handle direction.
Figure 11: Continual learning on the real-world RB-Y1 mobile manipulator. Top: Modular skills acquired during prior exploration sessions and stored in the skill repository, complete with descriptions from their respective skill files. Middle: The final execution of a complex tidy-up task—a single high-level instruction carried out across six sequential stages, annotated with completion timestamps and invoked skills. Bottom: The agent’s generated code for each stage (condensed from execution logs with passed arguments). Code invoking pre-stored skills is color-coded by skill ( chair , cups , bucket , and box correspond to each skill’s run.py ), while ad-hoc control logic written directly by the agent is shown in black.
Figure 12: Adapting skills and programs on the real Franka. Each row is one program that already worked, with the task it learned (left image) and the operator’s change (green). Right: the program’s entry point after the change, condensed from the agent’s code, with new lines in green, modified lines in blue, and reused lines in black. Top: find the cube, where each image overlays frames from the left camera a quarter of a second apart, earlier ones fainter, to show how the object is hidden; the cube is first dropped into a cup and then replaced by a red wedge. Bottom: wiping with paper towels from a box; S and P are the towel program’s modules, imported unchanged.
Franka FR3
Rainbow RB-Y1
Perception
RGB-D images from two exterior and one wrist camera; pixel to 3D point
Head RGB-D images and gaze control; pixel to 3D point; a vision-language model
Motion
Straight-line end-effector moves; joint moves with path checks
Joint, end-effector, and torso targets; IK over torso and both arms; base drive
Gripper
Grasp to a width with a set force
Close with a current limit, or set an opening
Model
None (the agent writes its own kinematics)
Forward kinematics; dry runs for collision, joint load, and center of mass
Table 11: Tools on the real robots.
Skill
What it asks of the agent
Manual robot control
Look first, explore with small moves, choose each action from the current observation, and confirm every condition of the task before handing off.
In-context action learning
When images of a completed task are supplied, infer the goal and useful strategy from them and adapt it to the current layout, without copying actions.
Experience memory
Record what worked right away, with the observable conditions under which it worked; retrieve it for similar situations and distill it as evidence grows.
Autonomous control
Start from a confirmed manual success, turn it into observation-driven code that can be supervised and resumed, and test it across layouts.
Transfer hand-off
Before exploration ends, write how each subtask and each object is handled, for a later session on new layouts.
Multi-environment exploration
Run parallel environments, track their scenes separately, and share one episode budget.
Table 12: Robot-agnostic skills given to every session.
Task
Benchmark id
Step limit
Partial credit
Evaluation
Agent
Make toast random
make_toast_random
1400
25/50/100
45
Codex
Hang mugs random
hang_mugs_random
800
15/40/100
50
Claude Code
Sweep blocks random
sweep_blocks_random
1000
–
30
Codex
Plug in charger
plug_in_charger
400
–
50
Claude Code
Swap blocks
swap_blocks
700
–
50
Codex
Make kong
make_kong
600
–
50
Codex
Table 13: RoboDojo tasks. Step limit at 25 Hz; partial credit: native score tiers; evaluation: number of official layouts.
Task
Benchmark id
Agent
Step limit
Budget
Envs
Program
Boxing books
boxing_books_up_for_storage
Codex †
36,341
128
6
fully automated
Getting organized
getting_organized_for_work
Claude Code ∗
23,506
128
2
agent-in-the-loop
Sorting items
sorting_household_items
Claude Code
23,711
128
8
agent-in-the-loop
Clearing food
clearing_food_from_table_into_fridge
Claude Code
19,602
128
4
agent-in-the-loop
Clean up desk
clean_up_your_desk
Claude Code
32,126
128
4
agent-in-the-loop
Loading the car
loading_the_car
Claude Code ∗
28,839
128
2
agent-in-the-loop
Table 14: BEHAVIOR-1K tasks and sessions. Step limit at 30 Hz; budget: exploration episodes allowed; envs: parallel environments; program: the program evaluated in Table 3 . ∗ Maximum reasoning effort. † Without demonstration keyframes or the vision-language model; 32 episodes used.
Task
Type
Step limit
Success condition
Budget
Football
HOI
2000
the ball passes the goal line between the posts
32
DoubleDesk
HOI
2000
the hammer rests inside the basket on the second desk
128
P&PBox
HOI
1450
the box rests on the shelf, inside its outline
128
OpenDoor
HSI
1800
the latch is released and the door is open by at least 60 ∘ , robot standing
32
SitSofa
HSI
2000
the pelvis is over the seat and below 0.66 m
32
Boxing
HSI
900
a palm comes within 0.18 m of the target on the bag
32
Table 15: HumanoidArena tasks. Step limit at 50 Hz; budget: largest exploration budget in Table 4 . HOI and HSI: human–object and human–scene interaction.
Task
Benchmark id
Arms
Cameras
Step limit
Hammer
hammer_nail
one
front, wrist
1000
Mouse
click_mouse
one
ego_right, wrist
1200 ∗
Bucket
pick_bucket
one
front, wrist
1200
Tongs
pinch_tongs
one
front, wrist
1000
Glasses
fold_glasses
one
front, wrist
1200
Water
water_plant
one
front, wrist
1000
Table 16: DexJoCo tasks, rand-obj setting. All cameras are RGB-D. ∗ Including 30 steps before control is handed over.
Budget
Interactive
Rehearsal
Used / budgeted
Submitted early
Agent time (h)
2
11
11
22 / 22
0 of 11
0.70 (0.44–1.28)
4
26
18
44 / 44
0 of 11
1.05 (0.51–2.46)
8
57
29
86 / 88
2 of 11
1.34 (0.59–3.02)
16
93
67
160 / 176
5 of 11
1.52 (0.72–2.89)
32
179
101
280 / 352
8 of 11
1.76 (0.98–3.89)
Table 17: DexJoCo sessions by exploration budget. One session per task. Episodes are summed over the eleven sessions; agent time runs from the start of the agent to evaluation and is the median over the eleven sessions, with the range in parentheses.
Figure 13: Perception and planning on a LIBERO-Pro layout, re-run with the agent’s code on the stored frames of one evaluation episode. (a) The agent-view image with Gemini’s points (diamonds) and the geometry that the program derives from depth: the top and axis of each can and the basket rim. (b) The point cloud of the agent-view camera, completed over the basket by the wrist camera and coloured by height above the table on the scale of (c) and (d), with the fitted cans, the basket rim, and the two planned placements, numbered in the order of execution. (c) The basket measured by the wrist camera in 4 mm cells, with the interior outlined. (d) Footprint reach, the highest point under the footprint of a can, and the two spots that minimize it; the circles show the footprint.
Figure 14: Why the program transfers to LIBERO-Pro. (a) Mean object positions over the 50 official and the 50 perturbed layouts, seen from above with the robot at the top; the perturbation exchanges the soup with the ketchup and the tomato sauce with the orange juice. Crosses mark the program’s first estimate of each can in every perturbed episode. (b) The distance between the program’s estimate of a can’s axis and the nearer can in the simulator, for the first agent-view estimate and for the wrist refit before each grasp, against the displacement of the two cans. The task takes the cans in either order, and in two perturbed episodes Gemini exchanged their labels; the rejected refits are those on the ketchup bottle.
Figure 15: Measuring the fingertips with the wrist camera, in the agent’s first episode, re-run on its stored frames. (a) The right wrist camera with the hand in its grasp posture; the agent segments the white fingertip caps and splits them into two clusters. (b) The hand’s point cloud in the frame of the measured flange, with the two clusters and the tip offsets the agent adopted (stars). (c) A side view of the clusters, their centroids and the adopted offsets, with the fingertips of the official hand model, which the agent never saw, for comparison.
Figure 16: The grasps the agent tried, seen by the wrist camera of the grasping hand in its exploration episodes, with panels 3 and 4 rotated so that the peg stands upright. The first four failed for the reasons given below them; the rolled vertical pinch carried both disks and became the grasp of the program.
Figure 17: Perception on an unseen layout and precision over the 150 formal episodes, re-run with the agent’s code. (a) The egocentric point cloud, coloured by height, with the measured disks, the peg axes, the target peg, the thumb and index targets of both pinches, and the predicted seat of the red disk. (b) The height of the red disk measured after placement, minus the height the program predicted. (c) The program’s estimate of the target peg against the hidden pose of the board, both relative to their means over the 150 layouts.
Figure 18: Perception on an unseen DoubleDesk scene, re-run with the agent’s code on the stored frames of one formal episode. (a) The head camera at the start, with the hammer the program finds and, enlarged, its axis and length. (b) The point cloud in the level frame, coloured by height, with the hammer and the bar fitted to it. (c) The last stance before the crossing, seen from above in the frame of the bar: the soles, the window from which the program starts a crossing, and the position of the left sole that the stance model predicted. (d) The basket found at the placing stance.
Figure 19: The stance model and the crossing window. (a) How far a left step from a squared stance places the left sole ahead of the right one, against the step amplitude: the agent’s measurements and the piecewise-linear model it wrote. (b) The positions of the two soles before the bar at every start of a crossing in exploration, rehearsal, and formal episodes, with the window that the program uses.
Versions
Main changes
Failures that motivated them
Completed
up to v8
First full chain; localization from the bar and the basket; crossing window and stance planner; grasp retries; basket approach by turning, then walking
Odometry off by 0.55 m; a walk onto the bar; a fall from 25.6 cm; a false miss that dropped the hammer
4/13
v9–v9g
Left step aimed by the stance model; handle at 60 ∘ ; deadlines; a subagent’s grasp measured at 3/13 against 13/22 and reverted
Late grasps; sweeps of the desk
5/10
v9h–v9k
Looser carrying posture; limited fine steps and backing off; window back to 21.5 cm
The arm over the head; the hammer lost; a fall from 22.4 cm
3/7
v9l–v9o
Limits on the trailing sole; stance model refitted on 60 transitions with a calibration episode; correction of the handle axis
A fall with the trailing sole 41.8 cm back; a handle at 85 ∘ ; a flipped hammer
8/12
v9p–v9t
Trailing-sole margin from the 27/27 against 4/13 record; search turn; stall guard; placing in a partly seen basket
A fall in rehearsal; stalled steps; a basket not seen
6/13
v9u–v9u5b
Re-approach; correction of lateral drift (9.5 cm, n=11 ); cycle guard; a faster place near the step limit
Three to six actions at the bar; a rehearsal cut off with 49 steps left
9/18
Table 18: Versions of the DoubleDesk program. From the agent’s own records. Completed counts interactive and rehearsal runs of each group of versions.
Figure 20: The outcome of every stage in every program run, in interactive and rehearsal episodes grouped by version (left), and for the submitted version on the 60 formal episodes, ordered by outcome (right). Each column is one episode; the counts give completed episodes.
Task
Instruction
Episodes
Before → after
Long 3
put the black bowl in the bottom drawer of the cabinet and close it
5
0 → 100
Long 8
put both moka pots on the stove
5
0 → 100
Long 9
put the yellow and white mug in the microwave and close it
4
0 → 2
Goal 1
put the bowl on the stove
5
2 → 96
Goal 5
push the plate to the front of the stove
4
2 → 44
Table 19: Tasks adapted to LIBERO-Pro. Success rate (%) on the 50 position-perturbed states before and after adaptation, with the exploration episodes it used.
Figure 21: Adapting to a mirrored cabinet. (a) An official layout of the third Long task and (b) the same layout under the LIBERO-Pro perturbation, which moves the cabinet to the other side of the table. (c) The adapted program’s check of the side, re-run on both first frames: level cells of the cabinet top above 1.08 m (orange) on either side of the robot, counted on the height map of the agent-view point cloud. (d) The adapted program on the perturbed layout, with the bowl in the drawer.
ASPIRE
RAX
Programs per task
one per perturbation
one, adapted on 5 of 40 tasks
Instances from the evaluation distribution
30
0, or up to 5 when adapted
Rollouts on these instances
up to 75, plus evolutionary search
0, or up to 5 when adapted
Exploration on unperturbed layouts
–
up to 20 episodes
Evaluation under perturbation
after development on it
zero-shot, or after adaptation
Table 20: Exploration before evaluation under position perturbation. ASPIRE’s counts are its debugging and validation splits for one task and perturbation; evolutionary search adds K rollouts per debugging instance and iteration.
Figure 22: Find the cube, trial 9: adapting to a new way of hiding the cube. (a)–(d) Left exterior camera: the operator drops the cube into an upright cup, turns the cup over, and shuffles the four cups. (e)–(f) Right exterior camera: after the agent added a detector for this case, the robot sets the identified cup aside and grasps the cube. The identification comes from the updated detector run on the recorded wrist stream, and the run resumed from locating the cups. Time is from the first frame.
Figure 23: Find the cube, trial 11: a red wedge instead of the cube. (a)–(d) The operator tosses the wedge into an upright cup, turns the cup over, and shuffles; (c) is the wrist camera, which the agent’s detector watches. (e)–(f) On the resumed run, the robot sets the predicted cup aside and grasps the wedge on the first attempt. Left exterior camera except (c); time is from the first frame.
Figure 24: The find-the-cube program. During the shuffle it follows every cup on the wrist camera and remembers the cup that covered the cube (top); it also catches a cube that is moved or dropped into a cup (middle). It then lifts the cup that hides the cube and picks the cube up (bottom).
Figure 25: The air-fryer program. After its elbow hit the left camera in run 2, the agent re-registered the camera and planned only postures clear of both cameras; it releases the banana with the hand tilted 60 ∘ . One program handles four fryer layouts; the bottom row follows the run with the fryer rotated.
Part of the model
In each scene
Source
Position and orientation (axis c , pull direction d )
estimated
lid circle and handle direction in the point cloud
Handle face, arm height, lateral offset
estimated
handle points in the point cloud
Drawer opening (joint state)
measured
shift of the handle face after the pull
Radii, heights, handle dimensions
fixed
the agent’s measurements from its first manual run
Table 21: What the agent’s fryer model takes from each scene.
Figure 26: The agent’s estimators on its final run , computed by its own code. (a) Lid points from the wrist camera above the fryer; blue points are kept and grey points discarded by the trimmed least-squares circle fit (red), whose centre is the axis c . (b) Handle-arm points seen from above, around the lid circle; points outside the ±0.25 rad window (dashed) are discarded, and d points to the median of the rest. (c) Handle points in the plane along d : the outer face in each 1.5 cm height slice (red bars) and the top of the arm (dashed). (d) Banana points along their principal axis, with the fitted centre line, its ends, and its middle (the grasp point). (e) Rim points after opening around the model’s prediction (dashed), and the fitted rim (green), 6.5 mm apart. Distances are in centimetres.
Figure 27: The air fryer task on the real robot. The agent’s final run, from the continuous recording of an exterior camera, with the time since the start of the run under each frame. Each row is one planned action, numbered as in Figure 3 (b): grasping the handle, pulling the drawer open by 13.1 cm in 3 cm steps, carrying the banana and releasing it with the hand tilted 60∘ over the basket, releasing the ball, and closing the drawer by a push that stops about 2 cm short followed by a push with closed fingers, which leaves the handle within 3.5 mm of its initial position. This camera was moved by a collision earlier in the session and was not used for perception.
Figure 28: Wiping a stain, trial 7b, from the left exterior camera. (a) The poured stain and the towel heap; (b) the grasped wad pressed on the stain; (c) scrubbing; (d) the towel is carried back; (e) the table after one pass. Time is from the first frame.
Figure 29: Wiping with paper towels, second wet run, from the left exterior camera. (a) The poured stain, the tissue box, and the bin behind it; (b) a sheet pulled from the box; (c) six sheets piled on the stain; (d) the pile pressed and grasped as a wad; (e) wiping, with the towel program’s zig-zag; (f) the wad released over the bin; (g) a leftover sheet picked up by the sweep; (h) the table afterwards, with a flat wet scrap the sweep missed. Time is from the first frame.
Figure 30: The wiping program. It finds the stain by comparing both exterior cameras with a clean reference and the towel heap in depth, grasps the heap as a wad, and wipes a zig-zag fitted to the reachable area. The wrist camera catches thin streaks that the exterior cameras miss, so the table is checked at both scales.
Figure 31: The ManipulationNet peg-in-hole challenge, final run, from a crop of the left exterior camera. (a) A peg inserted into L1 (3 mm clearance); (b)–(d) the rectangle peg entering L3 (0.1 mm clearance) corner first, turned upright, and seated; (e) the three pegs seated in L3. Time is from the first frame.
Figure 32: The peg-insertion procedure. The agent fits the board’s CAD model and each peg’s top outline to the wrist camera’s image edges. It drops the peg into the 3 mm row, probes by touch for the 1 mm hole, and tilts a corner into the 0.1 mm hole before turning the peg upright and pushing it down.
Figure 33: Bandu tower, final fully automatic assembly, from the left exterior camera. (a) The pieces after disassembly; (b)–(c) pieces are placed on the base; (d) the knob is held over the urn for visual alignment; (e) the completed tower. Time is from the first frame.
Figure 34: The Bandu assembly program, running fully automatically. It finds the pieces with both exterior cameras and Gemini, keeps a slot model of the demonstrated tower, and places each piece by a guarded place, a tactile search for the seat (with a tilt check for the urn), or a camera alignment of the knob over the urn’s opening.
Learning effective robot manipulation policies requires diverse, high-quality demonstrations, yet existing data pipelines are often difficult to scale because they rely on specialized hardware, centralized operators, or fixed task suites. We present AXIS, a growable community-driven data engine and benchmark for scalable robot learning, which enables browser-based teleoperation for large-scale demonstration collection, automatically generates and validates new manipulation tasks, and transforms community-collected demonstrations into training-ready data through automated success checking, quality filtering, trajectory smoothing, and visual and physics-based augmentation. The AXIS dataset currently contains 207 diverse tasks and 50K+ trajectories. Meanwhile, AXIS organizes data into task snapshots and evaluates policies with a systematic held-out protocol. We compare vision-language-action (VLA) policies under a unified AXIS evaluation suite and analyze scaling behavior across different data volumes. Continual pretraining on AXIS substantially improves the overall success rate of π0.5 by 5.8%, outperforms the model pretrained on RoboCasa365 by 37.3%, and exhibits consistent scaling with increasing data volume, with the largest gains observed under layout, sensor-noise, and camera perturbations.
Mengfei Zhao, Dihong Huang, Yikai Tang +12
1Axis Robotics · University of California, Berkeley · 3Georgia Institute of Technology +6
Large-scale demonstrations have driven unprecedented progress in robot learning, yet collecting robot data through teleoperation is expensive and difficult to scale to diverse environments and long-horizon tasks. Simulation offers a scalable alternative, but existing data-generation pipelines often rely on open-loop controllers, scripted skill sequences, or task-specific programs. We introduce SkillWeaver, an agentic framework that autonomously generates robot experience by exploring over Neural Interaction Skills (NIS): reusable, parameterized, closed-loop policies that expose learned physical interaction capabilities to a reasoning agent. Given a task and a simulated environment, a VLM agent reasons about what to do next, invokes and parameterizes NIS to interact with the environment, observes their outcomes, and generates verification, reflection, and memory to guide subsequent exploration. We instantiate NIS as reinforcement-learned policies for closed-loop, contact-rich manipulation and organize exploration as verifier-guided tree search, enabling the agent to discover successful long-horizon behaviors without relying on predetermined execution pipelines. SkillWeaver scales autonomously to 39.1K demonstrations across 14.1K scenes, which we distill into visuomotor policies. Across simulation benchmarks and real-world manipulation, training on SkillWeaver-generated experience substantially improves generalization to novel objects, spatial configurations, tasks, and environments, and enables zero- and few-shot sim-to-sim and sim-to-real transfer. Our results suggest agentic exploration over neural interaction skills as a scalable alternative for robot data generation.
He Zhu, Lusen Zhao, Kwan Man Cheng +2
Carnegie Mellon University · University of Illinois Urbana-Champaign
Robot learning has largely relied on human-teleoperated demonstrations to acquire effective learnable behaviors. However, human-operated data collection processes can be unintuitive, difficult to scale, and inherently asynchronous. We explore an alternative: distilling physical behavior from general-purpose multimodal foundation models into deployable robot policies by using the foundation model itself as an autonomous demonstrator. While sufficiently capable models can generate successful zero-shot manipulation trajectories, repeatedly invoking them during physical execution is slow and expensive, limiting their utility as scalable data generators. As a solution, we introduce CAPEX, an experience-conditioned demonstration collection framework that uses execution experience from previous attempts to adapt how frequently the foundation model must observe, reason, and replan. We evaluate across RoboCasa tasks and on physical Franka and bimanual YAM-arm platforms, measuring task success, model calls, token usage, collection time, and cost. We further train Diffusion Policy and ACT on matched sets of human-teleoperated and foundation-model-generated demonstrations to evaluate the downstream learning value of autonomously collected data. We find that CAPEX increases the number of successful demonstrations by 4.3x while reducing the cost per successful demonstration by 80%. Policies trained on CAPEX-generated data approach the performance of those trained on matched human demonstrations; with longer training, this gap largely closes for policies trained from scratch. These results suggest that foundation models can serve as scalable sources of reusable robot experience. Project page: https://capex-paper.github.io/
Shivam Aarya, Zhang Xi-Jia, Chengyue Huang +6
School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30332, USA