Compressed manipulation policies can pass offline evaluation while failing in closed-loop execution; this dissociation is established in prior work and is not our claim. We contribute a causal anatomy of one naturally occurring case. An 8-layer distillation of Octo-Base retains 86% of parameters, passes every offline check we applied (0.996 and 1.000 teacher-ratios on the family's own validation metrics), and collapses in closed loop: 0/72 vs. the teacher's 40/72 on a simulated WidowX pick-and-place task. The collapse is structured, not diffuse: early task stages degrade gradually (the student moves the object at 90% of the teacher's rate and grasps at 55%), while transport-to-target fails categorically, at 0% in every training variant. Paired action-trace forensics isolate the signature: a negative, late-heavy z residual, roughly 10x its post-repair magnitude, and persistent across the base distillation and both continuation branches. Four standard therapies fail under matched controls: continued training and in-domain offline data leave success at zero, even though the latter measurably improves marginal action statistics; command-level compensation recovers nothing at any offset, although the same perturbations degrade healthy policies; clamping the symptom in the command channel preserves grasping, yet success stays at floor. A minimal-pair intervention that substitutes half of the training stream with deployment-distribution teacher rollouts, with every other setting held fixed, restores parity with the teacher (18/36 vs. 17/36 held-out), eliminates that signature, and recovers a teacher-like perturbation-response profile. We claim existence, not universality. Operationally, offline gates, including a family's own validation metrics, are insufficient acceptance tests for compressed policies; a few dozen closed-loop trials sufficed to find what they missed.
Figures & tables
Policy
Succ.
Moved
Grasp
Sust.
Target
Teacher (12L)
15/24
20
20
19
15
Student V1 (8L, 300k)
0/24
18
11
11
0
+5k OXE (Step 0)
0/24
19
16
13
0
+5k Bridge (Step 1)
0/24
20
12
10
0
TABLE I: Staged closed-loop outcomes, instrumented slice (seed 0, episodes 0–23). Stages are the environment evaluator’s own predicates.
Policy
Early
Mid
Late
Full
Student V1 (8L, 300k)
−0.0069
−0.0028
−0.0103
−0.0067
+5k OXE (Step 0)
−0.0064
−0.0007
−0.0073
−0.0048
+5k Bridge (Step 1)
−0.0061
+0.0001
−0.0081
−0.0047
Repaired (step 8k)
−0.0005
+0.0037
−0.0007
+0.0009
TABLE II: Paired z residual on the fixed four-episode diagnostic panel (IDs 2, 11, 14, 23). Entries are mean student − teacher z differences, computed per episode and then averaged ( n=4 episodes per cell); intervals are fixed thirds of the 120-step rollout. Negative values indicate the student commands less upward displacement than the teacher on the same initialization. The base distillation and both continuation branches share the residual’s sign in the early-third, final-third, and full-rollout measures, along with its late-heavy temporal shape; under repair these residuals drop to near zero and the negative, late-heavy pattern is eliminated.
Hypothesis
Decisive test
Verdict
H1 Same-data continuation is sufficient
Step 0: continue, same data
Not supported at tested budgets (0/24; 0/36)
H2 Tested offline task-domain augmentation is sufficient
Step 1: add Bridge data
Insufficient (marginals move; success stays 0/24)
H3 Fixed additive z bias in the command channel
z -offset sweep
Excluded over tested offsets (no recovery; healthy policies degrade)
H4 Suppressing the negative z command is sufficient
Phase-gated command clamp
Falsified (grasp retained; success at floor)
H5 State-conditional policy deficiency in the deployment regime
Rollout-data minimal pair (Sec. VI)
Supported (bundle sufficient; active component unresolved)
Deployment-rollout substitution Replace half of each batch with successful teacher rollouts ( ρR=0.5 ). Held-out success returns to teacher parity: 18/36 vs. 17/36 . The matched zero-rollout control ( ρR=0 ) remains at floor. Rollout bundle sufficient; active component unresolved.
↑
Offline task-domain augmentation Add Bridge data through the Step 1 dual-stream schedule. Gripper and rotation marginals change, but success remains 0/24 . Observed marginal-statistics improvements are insufficient.
↑
Same-data continuation Continue training on the unchanged OXE mixture: +5 k steps gives 0/24 , and +10 k steps gives 0/36 . Additional same-data training is insufficient at tested budgets.
↑
Fig. 1: The therapy ladder. Interventions are ordered from shallow output-level corrections at the bottom to upstream changes in training supervision at the top. The four lower rungs fail under counted closed-loop trials, ruling out progressively richer simple explanations. Only the top rung restores teacher parity ( 18/36 vs. 17/36 held-out); its matched zero-rollout control stays at floor, attributing the recovery to the substituted data rather than the additional optimization. This establishes the rollout-data bundle as sufficient for recovery, not any individual component within that bundle.
Fixed additive z offset
Policy
0
.002
.004
.006
.008
.012
.020
Teacher (12L)
19
16
8
3
0
0
0
Repaired (8k)
22
22
14
3
0
0
0
Zero-control (8k)
0
2
2
1
0
0
0
Student V1
1
0
1
0
0
0
0
TABLE IV: Dose response under command-level perturbation. Counted success (out of 36) under fixed additive z offsets applied throughout the rollout, evaluated on the checkpoint-selection configurations (held-out configurations were never used for perturbation sweeps). The teacher and the repaired student show similar dose-dependent degradation; the zero-rollout control and V1 remain at floor at every dose. The same intervention that demonstrably degrades healthy policies does not rescue either failing one (Secs. V , VI ). Zero-offset entries are separate perturbation-sweep evaluations, not part of the checkpoint-selection comparison of Table V .
Selection
Held-out
Held-out
Policy
success
success
grasp
Student V1 (8L, 300k)
1/36
0/36
17/36
Teacher (12L)
19/36
17/36
30/36
Repaired, step 6k
20/36
20/36
34/36
Repaired, step 8k †
24/36
18/36
34/36
Repaired, step 10k
20/36
21/36
33/36
TABLE V: Parity under a pre-committed selection rule. Step 8,000 was selected on the selection set only (36 configurations), where it was the in-band maximum, and locked before any held-out evaluation; for the checkpoint-selection comparison, the held-out set (36) was then evaluated once per checkpoint. The pre-selected checkpoint scores 18/36 held-out versus the teacher’s 17/36 under an identical harness; the higher neighboring 21/36 at step 10k is not used for reporting. Held-out grasp (34/36 vs. the unrepaired student’s 17/36) shows the recovery does not trade away grasp performance. † Pre-selected.
On-policy distillation (OPD) pays twice for each fresh batch: the student generates trajectories and a stronger teacher scores them. Existing methods improve which trajectories are scored and how the teacher signal is constructed, but usually consume it with one actor update. We introduce CLOOPD, a closed-loop framework separating teacher-signal acquisition from student-side realization. CLOOPD selects an adaptive α waypoint inside a KL envelope, freezes the scored batch and its advantages, re-forwards the student after each actor pass, measures realization, and allocates actor work under a separate token budget. The framework includes deterministic two- and three-pass policies, token-priced CLOOPD-TPMR, and a budget-matched control. Across six 300-step runs on an 8-H20 node, every CLOOPD policy improves the one-pass TOP-D anchor at comparable teacher-token scale: macro accuracy rises from 15.41 to 17.78 with CLOOPD-Fixed2 and 19.36 with CLOOPD-Fixed3. At step 100, CLOOPD-Fixed3 reaches 15.35, nearly matching TOP-D at step 300 while using 67.2% fewer teacher-scored tokens and 28.0% fewer GPU-hours. Earlier 8-A100 ablations show adaptive α eliminates observed trust-envelope violations; a third pass adds headroom. These results position CLOOPD as a framework for budgeting how fully students learn from teacher-scored tokens.
Online distillation of vision-language-action (VLA) policies commonly relies on teacher action scores that an inference-only interface may not expose. Trajectory-only Adversarial Distillation (\gad{}) instead turns a fixed bank of teacher joint-position features into adaptive feedback: a discriminator compares feature marginals from teacher and student rollouts and supplies rewards for on-policy learning. The student interacts with the environment without online teacher queries or environment task rewards. On ManiSkill PutOnPlateInScene25Main, \gad{} improves OpenVLA-OFT (7B) from 26.2% to 76.6% \successonce{} (+50.4 percentage points) and from 20.3% to 58.6% \successend{}. It trails white-box on-policy distillation (OPD) at 81.6% / 59.4% by 5.0 / 0.8 percentage points (n=256 evaluation trajectories per checkpoint). The tested four-epoch continuation achieves the highest \gad{} result without additional teacher data. This best run uses 5× OPD's student interaction budget and 12.5× its complete-rollout batch-passes. The comparison demonstrates online VLA improvement without teacher scoring while quantifying unequal interaction and optimization budgets.
Yun Zhu, Da Wang
Independent Researcher, China · School of Computer and Information Technology, Shanxi University, Taiyuan, China
On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher. We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues. We term this phenomenon distillation overshoot. Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors. Early in training, repair outpaces erosion, allowing the student to combine the teacher's corrections with its own strengths and surpass the teacher. As training continues, erosion overtakes repair, causing these gains to vanish. Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere. SPOD requires only binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD consistently outperforms ordinary OPD and evaluated variants, improving final success by up to 15.2 percentage points and surpassing the teacher by up to 20.0 points. Moreover, SPOD's final checkpoints outperform even ordinary OPD's best intermediate checkpoints in all six settings, demonstrating that beyond-teacher gains can be not only preserved but further extended. Code: https://github.com/Jike338/SPOD
Jike Zhong, Ming Li, Yuxiang Lai +9
University of Southern California · University of Florida · Emory University +6