Code-as-Policy agents accomplish long-horizon embodied tasks by generating and executing code, yet continually improving them with teachers that are stronger but not globally reliable remains a key challenge. Existing distillation methods typically treat the teacher's complete behavior as the supervision target and thus misassign training credit on states where the teacher fails. We propose RE-0, a recursively verified policy improvement framework: rather than assuming that the teacher globally outperforms the student, RE-0 requests local corrections from the teacher on the student's own failure histories and checks in the environment whether each correction is genuinely beneficial; verified corrections yield immediate improvement. Building on this, we propose RE-OPD, which turns verified interventions into supervision for on-policy distillation. Only counterfactually verified teacher interventions provide distribution-level supervision, weighted by their measured local benefit, and the improvement they induce is projected back into the standalone student, so both where supervision is applied and how much credit the teacher receives co-evolve with the student policy. We further prove that the student's per-round gain is lower-bounded by its verified intervention gain up to verification and projection error terms. Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes.
Figures & tables
Figure 1: RE-0 overview. RE-0 localizes harmful segments in the current student’s trajectories, requests bounded teacher repairs, and verifies them by paired counterfactual rollout from the same snapshot (LCB gate). Verified interventions μn are distilled into the standalone student via cold-start SFT (Stage I) and RE-OPD refinement (Stage II); the student then regenerates trajectories, closing the loop. Left: simulation, real-robot, and stage-level results.
Figure 2: One recursion on a failing two-arm lift program. Locate: the earliest harmful boundary t (failure). Verify: from the same snapshot, student baseline vs. teacher patch under shared randomness; admitted iff Δ>0 . Distill: verified repairs are credit-weighted into the standalone student; deployment uses no teacher calls.
Task
Base
Flash-API
RE-OPD-cold ( n rows)
RE-OPD-refine
restack
4/50 a
2
31 (3)
40 b
spill
34
33
50 (67)
— c
nut
2
26
44 (27)
—
lift
2
6
38 (7)
29 d
door
48
—
50 (40)
—
cubestack
50
50
50 (0, no-train) e
—
Table 1: Main results (successes / 50 held-out seeds). Flash-API: same teacher, single-shot without diagnose–repair–verify. n : verified training rows. “—”: criterion abstained, not unmeasured. Trainer-seed replication (three seeds, Appendix D ): restack 31/44/36, spill 50/33/33, lift 38/33/35, nut 44/40/43, door 50/50/50.
Figure 3: Credit analysis. (a) Headroom alone does not imply credit: all tasks sit below ρ=0.80 yet admissions range 0–4. (b) Admission decay across rounds. (c) Δ separation: admitted vs. abstained over 110 events. (d) Threshold sweep: the admitted set is invariant.
Figure 4: Zero-shot sim-to-real transfer. Real-robot execution of the simulation-trained policies on the dual-arm AgileX PiPER platform: task keyframes (left), hardware success counts on 20 trials per task (middle), and hardware success rates against the simulation base model (right). The deployed policy code is identical to the simulation policy—no hardware-specific edits.
Selection rule (lift journal)
Rows
Successes / 50
Mean
s1
s2
s3
None (all candidates)
25
6
2
3
3.7
Sign ( Δ>0 )
7
5
4
4
4.3
LCB (paired, Eq. 2 )
3
11
5
13
9.7
Outcome (teacher programs)
27
5
9
4
6.0
Table 2: Selection-rule ablation (lift; three trainer seeds; uniform weights). LCB dominates unfiltered and sign rules in every seed pairing.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Round
Admitted credit
J change
Decision
Retained
restack π0→π1 (cold)
6/6 events, Δ≈+1.0
+54 pt ( 8→62% )
J<ρ : continue
π1
restack π1→π2′ (refine)
4/15, Δ∈[0.19,0.88]
+18 pt ( 62→80% )
J≥ρ : halt
π2′
lift π1→π2 (refine)
1 event, w=0.069
−18 pt ( 76→58% )
regression: halt
π1
spill π1→ (refine)
0 admitted
— (no projection)
abstain (Cor. 2)
π1
Appendix
Table 3: Per-round sign accounting: admitted credit against realized standalone change. The retained generation is the measured-better one in every round.
Task
Base
RE-OPD-cold
RE-OPD-refine
restack
[3.2, 18.8]
[48.2, 74.1]
[67.0, 88.8]
spill
[54.2, 79.2]
[92.9, 100.0]
—
nut
[1.1, 13.5]
[76.2, 94.4]
—
lift
[1.1, 13.5]
[62.6, 85.7]
[44.2, 70.6]
door
[86.5, 98.9]
[92.9, 100.0]
—
cubestack
[92.9, 100.0]
[92.9, 100.0]
—
Appendix
Table 4: Wilson 95% CIs (%) for Table 1 .
Task / rule
Rows
Seed 1
Seed 2
Note
lift: none
25
6
2
lift: sign
7
5
4
lift: LCB
3
11
5
lift: outcome
27
5
9
spill: none
6
10
—
mean reward 0.63
spill: sign
2
0
—
collapse; reward 0.00
Appendix
Table 5: Full selection-ablation grid (successes/50; one run per seed).
Teacher
Events
Admitted
Rescue rate
Failure mode
Student self-play
33 +
0
∼ 0
no credit at p≈0
Local task experts
32
0
∼ 10% ceiling
contract-format failures
API frontier
13
4
19–47%
—
Appendix
Table 6: The admission ledger as a teacher-effect instrument.
Policy \ Task
nut
spill
lift
handover
Base
2
34
2
—
π1 (restack student)
22
24
6
0
π2′ (restack refined)
16
45
3
1
Lift expert
0
28
38
0
Appendix
Table 7: Cross-task transfer (successes / 50, @120).
Task
Expert
Equal (42)
Proportional (25)
Full-pool (172)
Credit-equal
restack
31
3
2
2
1
spill
50
31
0
0
32
nut
44
9
0
0
21
lift
38
20
25
5
15
cubestack
50
50
0
48
48
door
50
50
0
49
49
Appendix
Table 8: Pooled joint training fails (successes / 50). Two separable failures: task interference and quota starvation.
Figure 5: Real-robot platform. The dual-arm AgileX PiPER rig used for the zero-shot deployment experiments.
Reliable repository-level coding requires agents not only to generate correct code but also to preserve constraints and respond to tool feedback across long interactions. However, our diagnostic analysis shows that agents often fail to act on relevant information already available in context, while final Pass/Fail outcomes provide only sparse process supervision. We introduce On-Policy Reminder Distillation (OPRD), which combines Multi-Scope Reminder supervision with outcome-based routing on reminder-free student rollouts. Successful rollouts provide autoregressive self-imitation targets, whereas failed rollouts receive Reminder-conditioned reverse-KL supervision from a teacher that re-evaluates the same student-generated prefixes. A Global Reminder converts final-test diagnostics into trajectory-level guidance, while Turn-local Reminders apply only to the next decision following the relevant environment observation. The teacher and student share a two-stage initialization from Coding SFT and Instruction-Following (IF) SFT. At inference, the student operates without Reminders. On SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual, OPRD improves the benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model. Under the same diagnostic protocol, it reduces the aggregate error-event count from 446 to 222, a 50.2% reduction.
On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3
On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher. We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues. We term this phenomenon distillation overshoot. Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors. Early in training, repair outpaces erosion, allowing the student to combine the teacher's corrections with its own strengths and surpass the teacher. As training continues, erosion overtakes repair, causing these gains to vanish. Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere. SPOD requires only binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD consistently outperforms ordinary OPD and evaluated variants, improving final success by up to 15.2 percentage points and surpassing the teacher by up to 20.0 points. Moreover, SPOD's final checkpoints outperform even ordinary OPD's best intermediate checkpoints in all six settings, demonstrating that beyond-teacher gains can be not only preserved but further extended. Code: https://github.com/Jike338/SPOD
Jike Zhong, Ming Li, Yuxiang Lai +9
University of Southern California · University of Florida · Emory University +6