Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong · Harbin Institute of Technology · ERNIE Team, Baidu
In this work, we study on-policy distillation (OPD) for transferring agentic coding capabilities from a large teacher model to a smaller student model. Prior work has focused primarily on reasoning and short-horizon agentic tasks. We present SWE-OPD, a minimal OPD study for software engineering, where the student interacts with executable repositories and the teacher supervises the prefixes visited by the student. Across three teacher--student pairs, two training sources, and three SWE benchmarks, we find that on-policy supervision transfers coding capability: supervision from Qwen3.6-27B increases the SWE-bench Pro resolve rate of Qwen3.5-9B from 35.9% to 39.4%. Refreshing trajectories is not strictly required either, as an offline variant with cached teacher scores still improves over the student baselines while cutting end-to-end GPU-hours by 59.4%--72.6% and halving peak GPU allocation, at benchmark-dependent accuracy. Furthermore, we extend the offline variant to corss-tokenizer scenarios: distilling GLM-4.5-Air from DeepSeek-V4-Flash-0731 raises its SWE-bench Pro and Multilingual resolve rate from 35.8% to 39.7% and from 41.8% to 46.2%, around 4% and 4.6% improvement, respectively. We also find that a decreasing loss, however, does not necessarily indicate improved agent performance: under top-K reverse-KL loss, one student exhibits severe repetition despite a decreasing loss. We identify a support-mass gradient term as a potential mechanism underlying this failure.
Figures & tables
Figure 1: Study design of SWE-OPD . Left: the student interacts with an executable repository; tool observations enter the conditioning context, while only student-generated tokens receive supervision. Middle & Right: The two design axes we compare are the trajectory source (current student in SWE-OPD vs. initial student in Offline SWE-OPD) and the teacher information used per position (top- K candidates in the RKL loss vs. the sampled token in the ST loss objective).
Method
Verified
Pro
Multilingual
Score
Δ
Score
Δ
Score
Δ
Student: Qwen3.5-9B Teacher: Qwen3.6-27B
Student baseline
57.9
35.9
43.0
Teacher Rollout Distillation
54.8
−3.1
34.5
−1.4
40.5
−2.5
RL (UniAgent ( Ding et al., 2026 ) )
59.2
+1.3
–
–
SWE-OPD (RKL)
60.5
+2.6
39.4
+3.5
45.8
+2.8
Table 1: Results after training on 5K ScaleSWE queries. Avg@4 is the mean resolve rate (%) over four independent runs on SWE-bench Verified ( Jimenez et al., 2024 ) , Pro ( Deng et al., 2025 ) , and Multilingual ( Zan et al., 2025 ) . Δ is the change from the student baseline in percentage points; bold marks the best score per benchmark. Subsequent benchmark tables use these conventions within each student--teacher pair. 3 3 3 The experimental setup, training configurations, and agent scaffold are detailed in Appendix A .
Method
Verified
Pro
Multilingual
Score
Δ
Score
Δ
Score
Δ
Student: Qwen3.5-9B Teacher: Qwen3.6-27B
Student baseline
57.9
35.9
43.0
SWE-OPD (RKL)
61.6
+3.7
38.4
+2.5
44.3
+1.3
SWE-OPD (ST)
61.3
+3.4
38.7
+2.8
43.1
+0.1
Student: GLM-4.5-Air Teacher: GLM-5.2
Table 2: Results on the internal dataset. We evaluate SWE-OPD with Qwen3.5-9B and GLM-4.5-Air as students, supervised by Qwen3.6-27B and GLM-5.2, respectively.
Figure 2: Top- K ablation. Gains in percentage points.
Figure 3: Top- K selection and nested support diagnostics. Qwen3.5-9B distilled from Qwen3.6-27B: (a,b) support-size sweep; (c,d) nested readouts for the K=128 ScaleSWE run. All horizontal axes show training steps. In (b), mean overlap over steps 62–71 is 62.5% at K=128 and 55.0% at K=1024 .
Method
Verified
Pro
Multilingual
Score
Δ
Score
Δ
Score
Δ
Student: Qwen3.5-9B Teacher: Qwen3.6-27B
Student baseline
57.9
35.9
43.0
SWE-OPD (RKL)
61.6
+3.7
38.4
+2.5
44.3
+1.3
SWE-OPD (ST)
61.3
+3.4
38.7
+2.8
43.1
+0.1
Offline SWE-OPD (RKL)
60.2
+2.3
38.0
+2.1
45.3
+2.3
Table 3: Results of SWE-OPD and Offline SWE-OPD on the internal dataset. We compare both variants for Qwen3.5-9B distilled from Qwen3.6-27B and GLM-4.5-Air distilled from GLM-5.2, and additionally evaluate Offline SWE-OPD (ST) for GLM-4.5-Air distilled from DeepSeek-V4-Flash-0731, a teacher with a different tokenizer.
Method
Qwen3.5-9B
GLM-4.5-Air
SWE-OPD
1024
2808
Offline SWE-OPD
416
768
GPU-hour reduction
2.46 ×
3.65 ×
Offline SWE-OPD breakdown
Student rollout
176
368
Teacher scoring
48
160
Table 4: Training cost on the internal dataset (GPU-hours) for our online and offline implementations.
Figure 4: GLM training dynamics: ( 4 ) Repetition fraction; ( 4 ) student entropy; ( 4 ) training loss. RKL and ST training are shown in orange and blue, respectively.
Figure 5: teacher-student probability divergence.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Training data
Model pair
Training variant
Loss
K
Kcache
ScaleSWE (5K)
Qwen
Teacher rollouts
SFT
–
–
Qwen
Online
RKL
128
–
Qwen
Online
ST
–
–
Qwen
Online (ablation)
RKL
256/512/1024
–
Internal ( ∼ 4K)
Qwen
Online
RKL
128
–
Qwen
Online
ST
–
–
Appendix
Table 5: Training configurations. Qwen denotes Qwen3.5-9B ← Qwen3.6-27B, GLM denotes GLM-4.5-Air ← GLM-5.2, and GLM/DS denotes GLM-4.5-Air ← DeepSeek-V4-Flash-0731, with student ← teacher. K is the RKL student-support size; Kcache is the offline teacher-cache size. A dash denotes a setting that does not apply.
Figure 6: Constructed comparison of independent-logit updates. Upper panels show initial probabilities (gray) and final probabilities (arrow tips) of two retained tokens and one omitted token. Lower panels track total outside-support mass; dashed lines mark the teacher. ST updates use the exact expected gradient.
Reliable repository-level coding requires agents not only to generate correct code but also to preserve constraints and respond to tool feedback across long interactions. However, our diagnostic analysis shows that agents often fail to act on relevant information already available in context, while final Pass/Fail outcomes provide only sparse process supervision. We introduce On-Policy Reminder Distillation (OPRD), which combines Multi-Scope Reminder supervision with outcome-based routing on reminder-free student rollouts. Successful rollouts provide autoregressive self-imitation targets, whereas failed rollouts receive Reminder-conditioned reverse-KL supervision from a teacher that re-evaluates the same student-generated prefixes. A Global Reminder converts final-test diagnostics into trajectory-level guidance, while Turn-local Reminders apply only to the next decision following the relevant environment observation. The teacher and student share a two-stage initialization from Coding SFT and Instruction-Following (IF) SFT. At inference, the student operates without Reminders. On SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual, OPRD improves the benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model. Under the same diagnostic protocol, it reduces the aggregate error-event count from 446 to 222, a 50.2% reduction.
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On τ2-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.
On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher. We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues. We term this phenomenon distillation overshoot. Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors. Early in training, repair outpaces erosion, allowing the student to combine the teacher's corrections with its own strengths and surpass the teacher. As training continues, erosion overtakes repair, causing these gains to vanish. Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere. SPOD requires only binary episode success and model probabilities already available during OPD. Across two model families and three agentic benchmarks, SPOD consistently outperforms ordinary OPD and evaluated variants, improving final success by up to 15.2 percentage points and surpassing the teacher by up to 20.0 points. Moreover, SPOD's final checkpoints outperform even ordinary OPD's best intermediate checkpoints in all six settings, demonstrating that beyond-teacher gains can be not only preserved but further extended. Code: https://github.com/Jike338/SPOD
Jike Zhong, Ming Li, Yuxiang Lai +9
University of Southern California · University of Florida · Emory University +6