ReTrace: Cross-Stage Failure Attribution for Self-Improving Robotic Manipulation
Organizations: College of Computer Science and Artificial Intelligence, Fudan University · Institute of Trustworthy Embodied AI, Fudan University · Singapore Management University
Abstract
General-purpose multimodal foundation models are increasingly capable of directly controlling robots, yet improving their manipulation capabilities through autonomous interaction remains challenging. A key challenge to such self-improvement is learning from failures in complex manipulation tasks involving cross-stage dependencies, where an action may succeed locally but cause a later stage to fail. To address this challenge, we introduce ReTrace, a self-improvement framework for robotic manipulation through autonomous physical interaction and retrospective reasoning, without provided demonstrations or model-parameter updates. ReTrace retrospectively traces downstream failures to earlier decisions, proposes targeted corrections, and evaluates them through subsequent interactions. Across attempts, ReTrace progressively consolidates validated corrections into a reusable manipulation skill that guide subsequent execution. We evaluate ReTrace on four real-world manipulation tasks using a Franka Research 3 robot. On the two tasks with strong cross-stage dependencies, GPT-6 Astra-low improves from 0% to 80% success on plug insertion and from 0% to 100% on ball pouring, while GPT-6 Sol-xhigh improves from 0% to 10% and 0% to 90%, respectively. The learned skills also transfer across model and reasoning configurations and reduce inference-time decision cost in most evaluated settings.
Figures & tables
| Model | Regular Tasks | Cross-stage Dependent Tasks | ||
|---|---|---|---|---|
| Banana Picking | Block Stacking | Plug Insertion | Ball Pouring | |
| GPT-6 Sol-xhigh (w/ skill) | 90% | 80% 50 | 10% 10 | 90% 90 |
| GPT-6 Astra-low (w/ skill) | 100% | 100% 20 | 80% 80 | 100% 100 |
| GPT-6 Sol-xhigh (w/o skill) | 90% | 30% | 0% | 0% |
| GPT-6 Astra-low (w/o skill) | 100% | 80% | 0% | 0% |
| GPT-6 Astra-high (w/o skill) | – | – | 10% | 20% |
Robot Skill Learning
Robot skill learning studies how robots acquire, adapt, and combine reusable behaviors from demonstrations, interaction, and other data.
See all 323 papers