Agentic robot systems combine language-model reasoning with robot policies to monitor execution and recover from failures. However, reviewing every action chunk incurs substantial overhead, even in familiar situations. We introduce FastHarness, an experience-driven execution harness that adaptively schedules GPT-6 Astra supervision of a π0.5 policy. A fast path uses stage memory and reviewed outcomes to reduce review frequency as experience accumulates, while a complementary recovery path focuses supervision on correcting errors. Once recovery is verified, routine execution resumes, with failure experience retained to guide future interventions. The policy remains frozen and generates fresh actions from current observations. Across four simulated manipulation tasks, reviewer token use decreases by an average of 57.2% per task from the first evaluated episode to the mean of the final five episodes. Mean episode time decreases from 85.5 to 52.0 minutes over the same interval. On the same tasks, mean success increases from 80% for GPT-as-Policy (π0.5 + GPT-6 Astra) to 97.19% with FastHarness. These results highlight adaptive supervision as a practical route to more efficient agentic robotics without additional model training.
Figures & tables
Figure 1: From VLA execution to experience-driven supervision. VLA-only execution generates actions from current observations. GPT-as-Policy adds GPT-6 Astra review at every action chunk. Using the same policy and reviewer, FastHarness skips redundant reviews as task experience accumulates and concentrates supervision on error recovery. Dots denote GPT reviews; the check mark indicates verified recovery, after which adaptive review resumes.
Figure 2: FastHarness adds two paths around the original GPT-as-Policy execution backbone (gray, middle). The experience-driven fast path (top) routes fresh VLA proposals through a memory-guided scheduler, bypassing GPT-6 review when eligible; the gray dashed link denotes the original per-chunk review route. The recovery path (bottom) uses reviewed local repair and, when needed, verified waypoint goals. Ordinary scheduling resumes after verified resolution. Dashed teal feedback retains reviewed evidence across episodes. Image inserts illustrate recorded task scenes.
Figure 3: GPT-6 calls across repeated task executions. Each panel uses a task-specific call-count scale and spans all evaluated episodes: 16 for Build tower, 20 each for Fold clothes and Bottle disposal, and nine for Classify objects. Episode indices follow the evaluation order within each shared-memory chain.
Figure 4: Execution efficiency for the first evaluated episode and the final-five-episode mean. Avg. denotes the equal-task mean. (a) Recorded input and output tokens, with cached input counted once. The mean per-task reduction is 57.2%, and the equal-task mean falls from 1.83 to 0.93 million tokens. (b) Episode time in minutes; the equal-task mean falls from 85.5 to 52.0 minutes (39% lower).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
1
Match the current observation; obtain and validate a fresh policy proposal.
2
Compute the interval and control-step allowance; set the review flag from the ordered triggers.
3
while the review flag is set do
4
Review the observation, proposal, preceding transition, history, memory, and recovery context.
5
On the first review only: attribute outcome and intent error; store any eligible checkpoint.
6
Update the stage example, reset review counters, and update recovery.
7
Retain outcome/intent uncertainty; set the review flag if recovery requests re-review.
Appendix
Algorithm 1 Review scheduling and recovery before each action chunk in FastHarness.
Component
Configuration
Environment
RoboDojo native runtime on Isaac Sim; two X5 arms; CPU physics.
Control and observations
Control step 0.04 s (25 Hz); three RGB views (high, left wrist, right wrist), joint state and end-effector state; no depth input.
JAX seed 0; a fresh policy server for each episode.
Action interface
14 dimensions: six joints and one gripper opening per arm; internal arm actions are increments relative to the current joint state; gripper 0 is closed and 1 is open. The policy predicts 50 actions; an executed proposal contains 1–15 control steps.
Reviewer
gpt-6-astra , reasoning effort xhigh ; structured review output via the Responses protocol.
Appendix
Table 2: Environment, policy, and harness configuration used for evaluation.
Task
Native step limit
Evaluation and analysis
Build tower
1,050
16 episodes: efficiency and completion
Fold clothes
500
20 episodes: review allocation, automatic spans, time and tokens
Bottle disposal
700
20 episodes: review allocation, automatic spans, time and tokens
Classify objects
1,100
9 episodes: efficiency and completion; episodes 1–8: 16 recovery events
Appendix
Table 3: Task coverage and native control-step limits. Task success counts are reported in Table 6 .
Observed event outcome
Count
Resumed normal scheduling
16
Closed without normal resumption
0
Still active at observation end
0
Total recovery events
16
Appendix
Table 4: Recovery outcomes in Classify objects episodes 1–8, comprising 16 events across eight episodes. Resuming normal scheduling is an intermediate recovery outcome, distinct from native task success. Events within an episode and shared memory chain are not independent trials.
Task
First episode (M)
Final five (M)
Reduction
Build tower
1.47
0.47
68%
Fold clothes
0.72
0.25
65%
Bottle disposal
1.64
0.58
64%
Classify objects
3.50
2.42
31%
Appendix
Table 5: Recorded GPT-6 token usage (millions) in the first evaluated episode and the mean of the final five episodes. Counts include input and output tokens, with cached input counted once. Build tower’s first evaluated episode follows earlier task experience. Values are rounded; the unweighted mean of the four per-task reductions is 57.2%.
Task
Episodes
Successes
Rate (%)
Build tower
16
15
93.75
Fold clothes
20
19
95
Bottle disposal
20
20
100
Classify objects
9
9
100
Appendix
Table 6: FastHarness completion counts and success rates, with an unweighted task mean of 97.19%. The evaluation protocol is specified in Appendix B .
External method
Build tower
Fold clothes
Bottle disposal
Classify objects
π0.5
24
21.07
69.30
12.67
π0.5 +GPT-6
60
100
100
60
GPT-6 only
0
40
20
100
Appendix
Table 7: Success rates (%) from the GPT-as-Policy report ( Su et al., 2026 ) for the same four tasks.
Figure 5: Empirical cumulative distributions of automatic-span lengths in episodes 1–5 and 16–20. A rightward shift indicates longer execution spans between GPT-6 reviews.
Figure 6: Running mean recorded tokens per episode on common axes. Each point averages all episodes through the indicated index, including initial experience acquisition.
A central goal in robotics is to enable manipulation across changing tasks and environments. Vision-language-action (VLA) models provide broad manipulation capabilities but can struggle when execution requires diagnosing failures and adapting behavior. Strong agents can discover effective interventions through interaction with these policies. We propose Recursive Harness Distillation to accumulate this experience as reusable guidance across agents. A strong agent distills its experience into a playbook for a light agent, then recursively refines the playbook using the light agent's execution feedback. The resulting playbook enables agents to reuse accumulated intervention knowledge in new task instances without updating model parameters. In real-world manipulation, the harness improves success from 37.3% to 64.0%. On SimplerEnv Bridge, the light agent with the playbook achieves 66.7% success, compared with 41.7% for the GR00T-only baseline, and outperforms the strong agent without a playbook. The same playbook also benefits the strong agent, which reaches 79.2% success. These results demonstrate the feasibility of harness distillation for robotics: intervention experience can be accumulated, refined through execution, and reused across agents to improve manipulation.
Seungyeon Kim, Junhoo Lee, Minkyu Kim +2
Seoul National University · Korea Advanced Institute of Science and Technology
Deploying a general-purpose model as an agent means running it inside a product-specific runtime harness that decides which actions are executable, how state advances, and when the loop may terminate. Ordinary trajectory supervision shows which action was taken but leaves the runtime semantics behind that action implicit. We introduce Harness Policy Distillation (HPD), which makes these runtime control semantics an explicit training target alongside actions to align the harness policy. HPD replays reference trajectories through the harness to recover the runtime semantics that are implicit in ordinary trajectory data, and represents them as control documents at each model-call boundary. These documents provide privileged information for a teacher to generate actions, while harness verification ensures that the resulting targets are executable under the runtime contract. We jointly train a shared model on the runtime semantics recovered from control-state privileged information and native history, thereby internalizing these semantics into the model policy. At deployment, the trained model uses the original harness interface without requiring the control document or any additional supplement. On TRAJECT-Bench with the Pi harness, Qwen3-8B trained with HPD reaches 22.98% autonomous task success and 49.23% runtime compliance, against 3.09% and 19.90% for the unadapted model. Harness-verified teacher actions substantially improve the native deployment policy, while joint control-state supervision provides further gains in task completion and runtime compliance. HPD turns executable runtime semantics into dense supervision for learning harness-compatible actions at each decision boundary.
Junze Zhu, Weihao Chen, Zhiyang Zhou +2
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Agents built around large language models continually accumulate interaction trajectories during deployment, yet their behavior typically remains fixed. Beyond updating model weights, these trajectories can improve the agent harness that constructs context, mediates tools, validates actions, and recovers execution. We introduce Harness-R1, the first method, to our knowledge, that makes failure-conditioned, lifecycle-wide editing of an existing executable runtime a learned capability. It post-trains a dedicated harness engineer with online reinforcement learning so that its edits are optimized for the realized task success they produce, rather than proposed by a fixed editor. A separate 9B engineer converts batches of target-agent failures into validated executable patches; fresh same-batch reruns of the frozen target provide outcome rewards, so training updates only the engineer. Cold-start supervised fine-tuning initializes this editing policy, which is then trained online with group-relative policy optimization. Across WebShop, ALFWorld, and DBBench, Harness-R1 raises vanilla Qwen3.5-9B success from 44.3% to 53.6% (+9.3 percentage points). After direct target-agent fine-tuning, a target-specific engineer raises the average further from 59.2% to 64.2% (+5.0 points); because these gains hold both before and after fine-tuning the target, Harness-R1 points toward co-evolving the harness engineer and the target agent.
Shuai Shao, Kangning Zhang, Qingyao Li +7
Shanghai Jiao Tong University · Xiaohongshu Inc. · Southeast University