cs.ROSep 27, 2026

Recursive Harness Distillation across Agents for Robot Manipulation

Authors: Seungyeon Kim, Junhoo Lee, Minkyu Kim, Baekseung Kim, Nojun Kwak

Organizations: Seoul National University · Korea Advanced Institute of Science and Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A central goal in robotics is to enable manipulation across changing tasks and environments. Vision-language-action (VLA) models provide broad manipulation capabilities but can struggle when execution requires diagnosing failures and adapting behavior. Strong agents can discover effective interventions through interaction with these policies. We propose Recursive Harness Distillation to accumulate this experience as reusable guidance across agents. A strong agent distills its experience into a playbook for a light agent, then recursively refines the playbook using the light agent's execution feedback. The resulting playbook enables agents to reuse accumulated intervention knowledge in new task instances without updating model parameters. In real-world manipulation, the harness improves success from 37.3% to 64.0%. On SimplerEnv Bridge, the light agent with the playbook achieves 66.7% success, compared with 41.7% for the GR00T-only baseline, and outperforms the strong agent without a playbook. The same playbook also benefits the strong agent, which reaches 79.2% success. These results demonstrate the feasibility of harness distillation for robotics: intervention experience can be accumulated, refined through execution, and reused across agents to improve manipulation.

Figures & tables

Figure 1: Recursive Harness Distillation. (1) A strong agent interacts with a frozen VLA and reflects on execution outcomes to identify useful intervention guidance. (2) This experience is distilled into a playbook that specifies when to intervene, how to adjust execution, and what to check afterward. The illustrated revision replaces a simple retry instruction with guidance to inspect the failed grasp, refocus on target–finger contact, and verify that the object follows the gripper. (3) A light agent consults the playbook to guide VLA execution through instruction, attention, and action interventions. Its execution experience returns to the strong agent for further playbook refinement, closing the recursive loop.
Figure 2: Experimental setups. (a) Three real-world tasks: Cube-to-Tray, Cube Stacking, and Button Pressing. (b) Four SimplerEnv Bridge tasks: Spoon-on-Towel, Carrot-on-Plate, Cube Stacking, and Eggplant-in-Basket.
Figure 3: Overall execution success and intervention ablations. We compare agents without a playbook and with initial or refined playbooks, and examine the contribution of each intervention site for Luna + GR00T. Each ablation disables one site while retaining the other two and keeping the refined playbook fixed. The results suggest that the gains extend beyond direct action correction, with playbook-guided instruction editing making the largest contribution among the tested sites.
MethodPlaybookCube-to-TrayCube StackingButton PressingOverall (%)
π0.5\pi_{0.5}None15/253/2510/2537.3
Luna + π0.5\pi_{0.5}None0/250/250/250
Luna + π0.5\pi_{0.5}Refined (Ours)18/2513/2517/2564.0
Table 1: Real-world manipulation success rates. We compare the frozen policy with Luna-guided execution, with and without the refined playbook, over 25 trials per task. Task entries report successful trials; Overall reports the aggregate success rate (%).
Table 5
Figure 5: Task success and deployment token use across playbook revisions. Success rate (left axis) and mean agent token usage per episode (right axis) are evaluated on the same 12 held-out instances per task. Token usage is averaged over all episodes, including failures.
Figure 6: Operational insights from the playbook. Selected excerpts address object identity after occlusion, weak versus empty lifts, instruction–action mismatches, attention-based recovery, preservation of corrections, and evidence for placement.
Figure 7: Playbook-guided recovery in three Bridge tasks. Each row connects an observed failure, a relevant playbook rule, an intervention, and subsequent execution. Multiple interventions can contribute to a recovery.

Explore similar work

CardsList

  1. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

    Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Goal-Conditioned Dynamic ManipulationVisuomotor Control

    Tsinghua University · Striding AI · Purdue University +4

  2. World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

    Sep 24, 2026Yehang Zhang, Haojian Huang, Yifan Chang +13Robotic ManipulationEfficient World-Action Model

    HKUST(GZ) · Knowin AI · CUHK

  3. Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learning

    Jul 15, 2026Xiaopeng Zhang, Yueyang Weng, Qi Liu +2Robotic ManipulationAgentic Reinforcement Learning

    School of Inteligence Science and Engineering, the Harbin Institute of Technology Shenzhen, 518055, China. · Faculty of Robot Science and Engineering, Northeastern University, Shenyang, 110819, China.