Organizations: College of Computer Science and Artificial Intelligence, Fudan University · Institute of Trustworthy Embodied AI, Fudan University · Singapore Management University
General-purpose multimodal foundation models are increasingly capable of directly controlling robots, yet improving their manipulation capabilities through autonomous interaction remains challenging. A key challenge to such self-improvement is learning from failures in complex manipulation tasks involving cross-stage dependencies, where an action may succeed locally but cause a later stage to fail. To address this challenge, we introduce ReTrace, a self-improvement framework for robotic manipulation through autonomous physical interaction and retrospective reasoning, without provided demonstrations or model-parameter updates. ReTrace retrospectively traces downstream failures to earlier decisions, proposes targeted corrections, and evaluates them through subsequent interactions. Across attempts, ReTrace progressively consolidates validated corrections into a reusable manipulation skill that guide subsequent execution. We evaluate ReTrace on four real-world manipulation tasks using a Franka Research 3 robot. On the two tasks with strong cross-stage dependencies, GPT-6 Astra-low improves from 0% to 80% success on plug insertion and from 0% to 100% on ball pouring, while GPT-6 Sol-xhigh improves from 0% to 10% and 0% to 90%, respectively. The learned skills also transfer across model and reasoning configurations and reduce inference-time decision cost in most evaluated settings.
Figures & tables
Figure 1: Cross-stage dependencies and failure attribution in robotic manipulation. Top: Plug insertion involves grasping, lifting, rotating, and inserting the plug. Although the grasp succeeds, the resulting relative pose between the gripper and plug leaves insufficient clearance for subsequent insertion, causing a collision. Bottom: Retrospective reasoning traces the insertion collision back to the earlier grasp pose, identifying the decision to revise.
Figure 2: Overview of the framework. The Executor plans and executes robot actions using current observations and accumulated experience. Across attempts, the Attributor diagnoses problems, proposes corrections, and refines them based on execution feedback. The Experience Memory stores skill, pending corrections, and execution history to guide subsequent attempts.
Figure 3: Real-world task settings. From left to right: banana picking, block stacking, plug insertion, and ball pouring. The latter two tasks require earlier grasping decisions to support later operations.
Model
Regular Tasks
Cross-stage Dependent Tasks
Banana Picking
Block Stacking
Plug Insertion
Ball Pouring
GPT-6 Sol-xhigh (w/ skill)
90%
80% ▲ 50
10% ▲ 10
90% ▲ 90
GPT-6 Astra-low (w/ skill)
100%
100% ▲ 20
80% ▲ 80
100% ▲ 100
GPT-6 Sol-xhigh (w/o skill)
90%
30%
0%
0%
GPT-6 Astra-low (w/o skill)
100%
80%
0%
0%
GPT-6 Astra-high (w/o skill)
–
–
10%
20%
Table 1: Task success rates. Baseline rows are shaded gray. The best result for each task is shown in bold, including ties. Green triangles indicate absolute improvements in percentage points over the corresponding models without skills.
Figure 4: Learning process through cross-stage failure attribution and correction refinement. Top: Plug insertion improves over three attempts by attributing the collision risk to an unfavorable grasp pose and progressively refining the gripper tilt before grasping. Bottom: Ball pouring succeeds in the second attempt by shifting the grasp position to prevent scoop instability during lifting.
Figure 5: Inference-time decision counts with and without learned skill. Bars show the mean over 10 trials per configuration. The per-trial decision budget is 35 for plug insertion and 25 for each of the other tasks. Unsuccessful trials are assigned the full task-specific decision budget.
Building reliable robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control. We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of robot execution systems without updating model weights. RPG identifies manipulation capabilities in an offline dataset and constructs related practice tasks in simulation. During practice, RPG uses execution feedback, privileged simulator state, and available dataset videos to diagnose failures. It develops new reusable symbolic skills, refines existing skills, and revises the system prompt based on these diagnoses. Cross-task evaluation tests individual candidate changes and merged revisions before they are retained for reuse. At test time, a multimodal LLM uses the resulting system prompt and skill library to coordinate perception and robot control. On held-out initializations of 22 manipulation tasks, RPG improves task success from 28.6% after the first practice round to 95.0% after 15 rounds, outperforming all evaluated baselines, including ASPIRE (75.5%) and CaP-Agent0 powered by GPT-6 Astra Pro (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, with ten trials on each of three tasks. Project Website: https://rpg-robot.github.io/
In robotic manipulation, the tight coupling between grasping and motion planning often obscures the true source of failure, leading to inefficient trial-and-error. To enable efficient long-horizon manipulation, we propose GTP-FA (Grasp-Then-Plan with Failure Attribution), a task-oriented two-stage 'grasp-then-plan' framework that generates grasp candidates and performs downstream motion planning conditioned on the selected grasp. Given a failed manipulation trajectory, we learn a failure attribution model that generalizes to unseen grasps and produces a stable distribution over failure modes for diagnosis-guided optimization. Based on these attribution results, we then optimize both modules in a diagnosis-driven manner: on the grasping side, we inject task-level priors and risk penalties into grasp candidate scoring and optimization to suppress unstable or task-incompatible grasps; on the planning side, we target high-risk initial states through data collection and fine-tuning to address genuine planning bottlenecks. We evaluate the proposed framework in both simulation and real-robot experiments, and show that GTP-FA improves the corresponding base learners across RL, IL, diffusion-policy, and VLA-based settings, achieving substantially higher overall task success rates.
Jiahao Xu, Peiyuan Wang, Hanzhuo Zhang +8
School of Computer Science and Engineering, Southeast University, China
Embodied foundation models have achieved significant breakthroughs in robotic manipulation, yet they still depend heavily on large-scale robot demonstrations. Although recent works have explored leveraging human data to alleviate this dependency, effectively extracting transferable knowledge remains a significant challenge due to the inherent embodiment gap between human and robot. We argue that the intention underlying human actions can serve as a powerful intermediate representation for bridging this gap. In this paper, we introduce a novel framework that explicitly learns and transfers human intention to facilitate robotic manipulation. Specifically, we model intention through gaze, as it naturally precedes physical actions and serves as an observable proxy for human intent. Our model is first pretrained on a large-scale egocentric human dataset to capture human intention and its synergy with action, followed by finetuning on a small set of robot and human data. During inference, the model adopts a Chain-of-Thought reasoning paradigm, sequentially predicting intention before executing the action. Extensive evaluations in simulation and real-world settings, across long-horizon and fine-grained tasks, and under few-shot and robustness benchmarks, show that our method consistently outperforms strong baselines, generalizes better, and achieves state-of-the-art performance. Project page: https://gazevla.github.io .
Chengyang Li, Kaiyi Xiong, Yuan Xu +3
Shanghai Jiao Tong University · Peking University · ShanghaiTech University +1