ReSkill: Selective Self-Improvement for Robot Manipulation
Organizations: The Chinese University of Hong Kong, Hong Kong SAR, China · Openmind Intelligent Robot Co., Ltd., Wuhu, China
Abstract
Self-improvement on compositional manipulation tasks requires deciding what to reuse, where to improve, and which changes to retain. We present ReSkill, a selective self-improvement framework that concentrates revision on capability gaps. A coding agent constructs an object-centric task graph from an instruction and a skill library, then uses node-level execution records to locate failures. At the task level, it revises stage organization, hand-off conditions, and recovery routes; at the skill level, it strengthens perception, motion, manipulation, and verification capabilities that repeatedly fail. Candidate revisions undergo full-task evaluation and paired comparisons before adoption, allowing local changes to accumulate into validated skills and an independently executable policy. The agent participates only during development; the deployed policy calls the retained skills without invoking the agent at run time. On the same development set, average success across ten RoboDojo tasks rises from 30.0% for the initial versions to 86.2% for the final adopted programs; retained programs achieve 83.6% in recorded robustness and model-selection evaluations. Across eight real-world tasks, ReSkill succeeds in 72 of 80 rebuilt-simulation trials and 62 of 80 real-robot trials (77.5%). The median revision changes only three functions, approximately 0.5% of a program, while 39% of task-specific functions in later-added tasks directly reuse existing implementations. These results show how locating local deficiencies, validating candidate changes, and accumulating reusable skills can improve execution of compositional manipulation tasks.
Figures & tables
| Public report (%) | ReSkill retained program | |||||
|---|---|---|---|---|---|---|
| Task | Astra | Initial (%) | Succ. | SR (%) | Steps | |
| Stack blocks | 86.0 | 6.7 | 25.0 | 31/32 | 96.9 | 178/550 |
| Stack blocks (lang.) | 40.0 | 0.0 | 25.0 | 27/27 | 100.0 | 202/400 |
| General pickup | 84.0 | 12.0 | 100.0 | 51/57 | 89.5 | 57/200 |
| Stack bowls | 62.0 | 38.0 | 37.5 | 21/22 | 95.5 | 248/800 |
| Build tower | 2.0 | 24.0 | 37.5 | 37/37 | 100.0 | 722/1050 |
| Apple | Red cup | Black spoon | Cup | Drawer | Block | Bimanual | Sweep | Total | |
|---|---|---|---|---|---|---|---|---|---|
| to bowl | to box | to plate | nesting | to bowl | sorting | ||||
| Rebuilt simulation | 10 | 10 | 10 | 10 | 10 | 10 | 7 | 5 | 72/80 |
| Real robot | 9 | 9 | 8 | 8 | 9 | 9 | 6 | 4 | 62/80 |
| Change type | Revisions | Funcs. | Compared | Improved | SR (pp) | Gain share |
|---|---|---|---|---|---|---|
| Skill level | 303 | 2 | 119 | 58 | +2.8 | 36% |
| Task level | 47 | 1 | 22 | 11 | +3.3 | 4% |
| Both levels | 299 | 5 | 113 | 56 | +5.6 | 48% |
| Parameters only | 125 | 0 | 63 | 25 | +1.2 | 12% |
| Total | 774 | 3 | 317 | 150 | +3.5 | 100% |
| First success (h) | First DEV 8/8 (h) | Best DEV | Best TEST | |||||
|---|---|---|---|---|---|---|---|---|
| Task | Demo | None | Demo | None | Demo | None | Demo | None |
| Build tower | 0.7 | 3.6 | 4.5 | 4.2 | 8/8 | 8/8 | 7/8 | 6/8 |
| Classify objects | 1.5 | 3.6 | 4.7 | 6.5 | 8/8 | 8/8 | 8/8 | 7/8 |
| Stack blocks | 0.7 | 0.5 | 2.1 | 3.2 | 8/8 | 8/8 | 7/8 | 8/8 |
| Stack bowls | 2.9 | 0.5 | 4.0 | 3.6 | 8/8 | 8/8 | 7/8 | 8/8 |
Robot Skill Learning
Robot skill learning studies how robots acquire, adapt, and combine reusable behaviors from demonstrations, interaction, and other data.
See all 337 papers