Code-as-Policy agents accomplish long-horizon embodied tasks by generating and executing code, yet continually improving them with teachers that are stronger but not globally reliable remains a key challenge. Existing distillation methods typically treat the teacher's complete behavior as the supervision target and thus misassign training credit on states where the teacher fails. We propose RE-0, a recursively verified policy improvement framework: rather than assuming that the teacher globally outperforms the student, RE-0 requests local corrections from the teacher on the student's own failure histories and checks in the environment whether each correction is genuinely beneficial; verified corrections yield immediate improvement. Building on this, we propose RE-OPD, which turns verified interventions into supervision for on-policy distillation. Only counterfactually verified teacher interventions provide distribution-level supervision, weighted by their measured local benefit, and the improvement they induce is projected back into the standalone student, so both where supervision is applied and how much credit the teacher receives co-evolve with the student policy. We further prove that the student's per-round gain is lower-bounded by its verified intervention gain up to verification and projection error terms. Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes.
Figures & tables
Figure 1: RE-0 overview. RE-0 localizes harmful segments in the current student’s trajectories, requests bounded teacher repairs, and verifies them by paired counterfactual rollout from the same snapshot (LCB gate). Verified interventions μn are distilled into the standalone student via cold-start SFT (Stage I) and RE-OPD refinement (Stage II); the student then regenerates trajectories, closing the loop. Left: simulation, real-robot, and stage-level results.
Figure 2: One recursion on a failing two-arm lift program. Locate: the earliest harmful boundary t (failure). Verify: from the same snapshot, student baseline vs. teacher patch under shared randomness; admitted iff Δ>0 . Distill: verified repairs are credit-weighted into the standalone student; deployment uses no teacher calls.
Task
Base
Flash-API
RE-OPD-cold ( n rows)
RE-OPD-refine
restack
4/50 a
2
31 (3)
40 b
spill
34
33
50 (67)
— c
nut
2
26
44 (27)
—
lift
2
6
38 (7)
29 d
door
48
—
50 (40)
—
cubestack
50
50
50 (0, no-train) e
—
Table 1: Main results (successes / 50 held-out seeds). Flash-API: same teacher, single-shot without diagnose–repair–verify. n : verified training rows. “—”: criterion abstained, not unmeasured. Trainer-seed replication (three seeds, Appendix D ): restack 31/44/36, spill 50/33/33, lift 38/33/35, nut 44/40/43, door 50/50/50.
Figure 3: Credit analysis. (a) Headroom alone does not imply credit: all tasks sit below ρ=0.80 yet admissions range 0–4. (b) Admission decay across rounds. (c) Δ separation: admitted vs. abstained over 110 events. (d) Threshold sweep: the admitted set is invariant.
Figure 4: Zero-shot sim-to-real transfer. Real-robot execution of the simulation-trained policies on the dual-arm AgileX PiPER platform: task keyframes (left), hardware success counts on 20 trials per task (middle), and hardware success rates against the simulation base model (right). The deployed policy code is identical to the simulation policy—no hardware-specific edits.
Selection rule (lift journal)
Rows
Successes / 50
Mean
s1
s2
s3
None (all candidates)
25
6
2
3
3.7
Sign ( Δ>0 )
7
5
4
4
4.3
LCB (paired, Eq. 2 )
3
11
5
13
9.7
Outcome (teacher programs)
27
5
9
4
6.0
Table 2: Selection-rule ablation (lift; three trainer seeds; uniform weights). LCB dominates unfiltered and sign rules in every seed pairing.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Round
Admitted credit
J change
Decision
Retained
restack π0→π1 (cold)
6/6 events, Δ≈+1.0
+54 pt ( 8→62% )
J<ρ : continue
π1
restack π1→π2′ (refine)
4/15, Δ∈[0.19,0.88]
+18 pt ( 62→80% )
J≥ρ : halt
π2′
lift π1→π2 (refine)
1 event, w=0.069
−18 pt ( 76→58% )
regression: halt
π1
spill π1→ (refine)
0 admitted
— (no projection)
abstain (Cor. 2)
π1
Appendix
Table 3: Per-round sign accounting: admitted credit against realized standalone change. The retained generation is the measured-better one in every round.
Task
Base
RE-OPD-cold
RE-OPD-refine
restack
[3.2, 18.8]
[48.2, 74.1]
[67.0, 88.8]
spill
[54.2, 79.2]
[92.9, 100.0]
—
nut
[1.1, 13.5]
[76.2, 94.4]
—
lift
[1.1, 13.5]
[62.6, 85.7]
[44.2, 70.6]
door
[86.5, 98.9]
[92.9, 100.0]
—
cubestack
[92.9, 100.0]
[92.9, 100.0]
—
Appendix
Table 4: Wilson 95% CIs (%) for Table 1 .
Task / rule
Rows
Seed 1
Seed 2
Note
lift: none
25
6
2
lift: sign
7
5
4
lift: LCB
3
11
5
lift: outcome
27
5
9
spill: none
6
10
—
mean reward 0.63
spill: sign
2
0
—
collapse; reward 0.00
Appendix
Table 5: Full selection-ablation grid (successes/50; one run per seed).
Teacher
Events
Admitted
Rescue rate
Failure mode
Student self-play
33 +
0
∼ 0
no credit at p≈0
Local task experts
32
0
∼ 10% ceiling
contract-format failures
API frontier
13
4
19–47%
—
Appendix
Table 6: The admission ledger as a teacher-effect instrument.
Policy \ Task
nut
spill
lift
handover
Base
2
34
2
—
π1 (restack student)
22
24
6
0
π2′ (restack refined)
16
45
3
1
Lift expert
0
28
38
0
Appendix
Table 7: Cross-task transfer (successes / 50, @120).
Task
Expert
Equal (42)
Proportional (25)
Full-pool (172)
Credit-equal
restack
31
3
2
2
1
spill
50
31
0
0
32
nut
44
9
0
0
21
lift
38
20
25
5
15
cubestack
50
50
0
48
48
door
50
50
0
49
49
Appendix
Table 8: Pooled joint training fails (successes / 50). Two separable failures: task interference and quota starvation.
Figure 5: Real-robot platform. The dual-arm AgileX PiPER rig used for the zero-shot deployment experiments.