Self-improvement on compositional manipulation tasks requires deciding what to reuse, where to improve, and which changes to retain. We present ReSkill, a selective self-improvement framework that concentrates revision on capability gaps. A coding agent constructs an object-centric task graph from an instruction and a skill library, then uses node-level execution records to locate failures. At the task level, it revises stage organization, hand-off conditions, and recovery routes; at the skill level, it strengthens perception, motion, manipulation, and verification capabilities that repeatedly fail. Candidate revisions undergo full-task evaluation and paired comparisons before adoption, allowing local changes to accumulate into validated skills and an independently executable policy. The agent participates only during development; the deployed policy calls the retained skills without invoking the agent at run time. On the same development set, average success across ten RoboDojo tasks rises from 30.0% for the initial versions to 86.2% for the final adopted programs; retained programs achieve 83.6% in recorded robustness and model-selection evaluations. Across eight real-world tasks, ReSkill succeeds in 72 of 80 rebuilt-simulation trials and 62 of 80 real-robot trials (77.5%). The median revision changes only three functions, approximately 0.5% of a program, while 39% of task-specific functions in later-added tasks directly reuse existing implementations. These results show how locating local deficiencies, validating candidate changes, and accumulating reusable skills can improve execution of compositional manipulation tasks.
Figures & tables
Figure 2: ReSkill constructs and improves an executable task policy. An instruction and a skill library define a task graph and its node–skill bindings. Trial execution exposes planning or skill gaps, prompting task-level graph revision and skill-level implementation changes. Paired evaluation determines which candidates are retained, updates the skill library, and yields a frozen executable policy with no run-time Agent calls. The bottom row shows original simulation examples of perception, motion, manipulation, and verification skills.
Figure 3: Representative examples from the four skill categories. Columns show perception, motion, manipulation, and verification. The images show inputs, execution stages, or inspected states associated with the named skills; these associations are checked against execution traces and the corresponding frozen program implementations.
Figure 4: Ten RoboDojo simulation tasks. Representative head-camera views from recorded ReSkill executions. The two rows follow the task order in Table 1 .
Public report (%)
ReSkill retained program
Task
Astra
π0.5
Initial (%)
Succ.
SR (%)
Steps
Stack blocks
86.0
6.7
25.0
31/32
96.9
178/550
Stack blocks (lang.)
40.0
0.0
25.0
27/27
100.0
202/400
General pickup
84.0
12.0
100.0
51/57
89.5
57/200
Stack bowls
62.0
38.0
37.5
21/22
95.5
248/800
Build tower
2.0
24.0
37.5
37/37
100.0
722/1050
Table 1: Full-task success (%) and execution cost on ten RoboDojo tasks. Astra (Agent-as-Policy) and π0.5 use success rates from the official report under different protocols and are shown for reference. “Initial” is development-set success for the first version the development agent evaluated in full. “Succ.” and “SR” report successes/episodes and success rate on the recorded robustness and model-selection evaluations (seed 1, layouts 10–54). All retained programs make zero run-time API calls. “Steps” gives the median control-step count over the native step limit. The average is taken over the ten task success rates, with equal task weights.
Apple
Red cup
Black spoon
Cup
Drawer
Block
Bimanual
Sweep
Total
to bowl
to box
to plate
nesting
to bowl
sorting
Rebuilt simulation
10
10
10
10
10
10
7
5
72/80
Real robot
9
9
8
8
9
9
6
4
62/80
Table 2: ReSkill successes on eight real-world tasks in rebuilt simulation and on the real robot (ten trials per task).
Figure 5: Real-world task environments and four execution sequences. (a) Eight task families, each pairing a real demonstration frame (left) with its MuJoCo reconstruction (right). (b) Selected keyframes spanning apple-to-bowl, block collection, drawer storage, and bimanual tabletop transfer, using one video per task. The upper label beneath each frame names the stage; the lower label identifies the visible sub-skill. These qualitative sequences include multi-object operations; quantitative task variants are defined in Table 2 . The sweeping reconstruction in (a) uses a single yellow block.
Figure 6: Composition of failed episodes during development. All evaluated candidates on development and robustness layouts are divided into four consecutive, equally sized groups by episode completion order. Bars normalize the five cause groups over failed episodes only; labels give failures/total episodes, so changes in the number of failures remain visible. These periods describe the exploration process, rather than successive retained policies.
Change type
Revisions
Funcs.
Compared
Improved
Δ SR (pp)
Gain share
Skill level
303
2
119
58
+2.8
36%
Task level
47
1
22
11
+3.3
4%
Both levels
299
5
113
56
+5.6
48%
Parameters only
125
0
63
25
+1.2
12%
Total
774
3
317
150
+3.5
100%
Table 3: Revision statistics by change type. “Funcs.” is the median number of changed functions; “Compared” counts revisions whose two versions each have at least eight verified episodes on the same layouts; “Improved” counts revisions with higher success; “ Δ SR” is the mean change in full-task success across paired episodes for those compared revisions, in percentage points; and “Gain share” is each type’s share of the total positive success-rate improvement.
Figure 7: Function-level skill reuse. Each row covers function definitions in the task-specific modules of one adopted program (count in parentheses), split into functions identical to another task, same-named functions with rewritten implementations, and functions unique to the task; the number at right is the verbatim share. The bottom four tasks were added during development to examine reuse in new tasks.
First success (h)
First DEV 8/8 (h)
Best DEV
Best TEST
Task
Demo
None
Demo
None
Demo
None
Demo
None
Build tower
0.7
3.6
4.5
4.2
8/8
8/8
7/8
6/8
Classify objects
1.5
3.6
4.7
6.5
8/8
8/8
8/8
7/8
Stack blocks
0.7
0.5
2.1
3.2
8/8
8/8
7/8
8/8
Stack bowls
2.9
0.5
4.0
3.6
8/8
8/8
7/8
8/8
Table 4: Demonstration-assisted candidates compared with candidates started simultaneously without a demonstration, taking the faster or better of the two demonstration-free candidates. Times are in hours from candidate creation: “First success” is the first successful full episode, and “First DEV 8/8” is the completion time of the first evaluation that succeeds on all eight development layouts; “Best DEV” includes only versions evaluated on the full development set; “Best TEST” gives the highest recorded successes/episodes on model-selection layouts.
Figure 8: Local skill improvement in tool storage. (a) Frames from one round-15 rollout show localization, detection of a tilted placement, high-side pressing, rechecking, and completed storage. Frames 2–4 are wrist-camera details; tilt angles are computed from adjacent observations. (b) Development results with simulator-measured depth, showing only candidates evaluated on the same 40 layouts. The horizontal axis retains the original round numbers; the vertical axis reports whole-task success. Lines indicate revision ancestry: round 13 resumes from round 10. Filled circles, open circles, and crosses denote adopted versions, intermediate candidates, and rejected candidates, respectively. The annotations list each round’s skill changes; the macro task graph is unchanged throughout this window.