OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents

Yuechen Luo; Zhiyi Lai; Miao Ziqi; Zihang Xu; Jingxuan Xu; Songwei Yu; Wenqiang Zhu; Weihao Li

cs.CLOct 15, 2026

OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents

Authors: Yuechen Luo, Zhiyi Lai, Miao Ziqi, Zihang Xu, Jingxuan Xu, Songwei Yu, Wenqiang Zhu, Weihao Li,

Organizations: Tsinghua University · Independent Researcher · Shanghai Artificial Intelligence Laboratory · National University of Singapore

Abstract

Reliable repository-level coding requires agents not only to generate correct code but also to preserve constraints and respond to tool feedback across long interactions. However, our diagnostic analysis shows that agents often fail to act on relevant information already available in context, while final Pass/Fail outcomes provide only sparse process supervision. We introduce On-Policy Reminder Distillation (OPRD), which combines Multi-Scope Reminder supervision with outcome-based routing on reminder-free student rollouts. Successful rollouts provide autoregressive self-imitation targets, whereas failed rollouts receive Reminder-conditioned reverse-KL supervision from a teacher that re-evaluates the same student-generated prefixes. A Global Reminder converts final-test diagnostics into trajectory-level guidance, while Turn-local Reminders apply only to the next decision following the relevant environment observation. The teacher and student share a two-stage initialization from Coding SFT and Instruction-Following (IF) SFT. At inference, the student operates without Reminders. On SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual, OPRD improves the benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model. Under the same diagnostic protocol, it reduces the aggregate error-event count from 446 to 222, a 50.2% reduction.

Figures & tables

Figure 1: Diagnostic analysis of Qwen3.5-4B on 200 tasks from the SWE-rebench v2 validation set. Left: a Scope and Source Grounding example in which a Turn-local Reminder prevents unsupported patch expansion. Right: aggregate error-event counts across four categories under Reminder-free and Reminder-augmented settings; percentages indicate relative reductions.

Figure 2: Overview of OPRD. OPRD constructs Global Reminders and observation-anchored Turn-local Reminders, and initializes the teacher and student through Coding SFT followed by IF SFT. During on-policy training, successful rollouts are optimized through autoregressive self-imitation, while failed rollouts receive reverse-KL supervision from a Reminder-conditioned teacher.

Stage	Unit	SWE-rebench V2	Multi- SWE-RL
Coding SFT	Trajectory	7,262	738
IF SFT	Example	2,000	0
OPRD	Task	3,630	370

Table 1: Training data used at each stage.

Figure 3: Programming-language distributions for IF SFT, Coding SFT, and the OPRD task pool. Percentages are computed from 2,000 examples, 8,000 examples, and 4,000 examples, respectively. The same language-color mapping is used in all three pies.

Training Configuration	SWE-bench	SWE-bench	SWE-bench	Average (%)
Training Configuration	Verified (%)	Pro (%)	Multilingual (%)	Average (%)
Qwen3.5-4B	50.4	24.2	35.3	36.6
$+$ Coding SFT	53.6	25.6	38.7	39.3
$+$ Coding SFT $\rightarrow$ IF SFT	55.8	26.7	42.3	41.6
$+$ Coding SFT $\rightarrow$ OPRD (w/o IF SFT)	58.4	28.3	46.3	44.3
$+$ Coding SFT $\rightarrow$ IF SFT $\rightarrow$ OPRD (Ours)	61.2	30.5	52.0	47.9

Table 2: Main results with Qwen3.5-4B and Qwen3.5-9B on SWE-bench Verified , SWE-bench Pro , and SWE-bench Multilingual . We report task resolution rates (%) and their unweighted average. Within each backbone, the best and second-best results in each column are shown in bold and underlined, respectively.

Figure 4: Effect of the teacher refresh schedule on Qwen3.5-4B. We compare Frozen Teacher, Periodic Refresh (

2\times

4\times

, and

6\times

), and Shared Parameters on three benchmarks and their unweighted average.

Figure 5: Effect of Reminder scope and supervised preparation on Qwen3.5-4B. We compare reminder-free, Global-only, Turn-local-only, and Multi-Scope Reminder conditioning for the Base, Coding SFT, and IF SFT checkpoints.

Figure 6: Reminder-following rates on 200 SWE-rebench V2 validation tasks. After a category-specific error is detected, the corresponding Reminder is inserted at the relevant interaction point, and the subsequent behavior is judged by whether it corrects or avoids that error. Each task–category pair contributes at most one successful follow event. Overall aggregates the four functional categories.

Figure 7: Diagnostic error reduction on 200 SWE-rebench V2 validation tasks. Bars show case-level trigger counts for Qwen3.5-4B (Base), Coding SFT, and OPRD. Percentages report relative reductions from Qwen3.5-4B to OPRD.

Figure 8: Effect of functional Reminder categories on the Qwen3.5-4B base checkpoint. We evaluate Tool-Feedback Recovery, Scope and Source Grounding, Scenario-Condition Reasoning, and Post-Edit Control and Validation, together with their combination. The dashed line denotes reminder-free performance.

KL Support and Distillation Temperature
Setting	SWE-bench	SWE-bench	SWE-bench	Average (%)
Setting	Verified (%)	Pro (%)	Multilingual (%)	Average (%)
Full Vocabulary, $\mathcal{T}=0.6$	59.8	29.4	50.3	46.5
Top-100, $\mathcal{T}=0.6$	60.2	29.7	50.7	46.9
Top-100, $\mathcal{T}=1.0$ (Ours)	61.2	30.5	52.0	47.9
Top-100, $\mathcal{T}=1.5$	60.4	29.8	51.0	47.1

Table 3: Ablations of KL support, distillation temperature, outcome-based routing, and KL direction with Qwen3.5-4B. Scores are task resolution rates (%), and Average is the unweighted mean across the three benchmarks. The best and second-best results within each group are shown in bold and underlined, respectively.

$\lambda_{+}:\lambda_{-}$	Ver.	Pro	Multi.	Avg.
0.25:1	59.6	28.7	49.7	46.0
0.5:1	60.4	29.5	51.0	47.0
1:1 (Ours)	61.2	30.5	52.0	47.9
2:1	60.8	29.8	51.3	47.3
4:1	59.8	28.9	50.3	46.3

Table 4: Ablation of the relative weights assigned to successful-trajectory autoregressive self-imitation (

\lambda_{+}

) and failed-trajectory Reminder-conditioned distillation (

\lambda_{-}

). Columns report SWE-bench Verified , Pro, Multilingual, and their average.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Parameter	Value
Model	Qwen3.5-4B	Maximum sequence length	131,072
Training framework	VeOmni	LR start	0
Epochs	3	GPU	16 NVIDIA H20
Global batch size	32	Micro batch size	1
Peak learning rate	$1\times 10^{-5}$	Minimum learning rate	$1\times 10^{-6}$
LR schedule	Cosine	Warmup ratio	0.005

Appendix

Table 5: Training configuration for the Qwen3.5-4B OPRD experiment.

ID	Functional Class	Original Rubric Source	Count
R1	Tool-Feedback Recovery	Deterministic hard rules	10
R2	Scope and Source Grounding	Scope/patch and source/path slots	10
R3	Scenario-Condition Reasoning	Scenario/condition slot	5
R4	Post-Edit Control and Validation	Post-edit drift and validation/wrap-up slots	7
	Total	10 hard rules and 22 non-hard strategies	32

Appendix

Table 6: The four reporting classes used for functional Reminder analysis. The mapping is a fixed partition of the 32 original rubric categories.

Figure 9: Complete runtime Reminder category-judge prompt (Part 1 of 5).

Figure 10: Complete runtime Reminder category-judge prompt (Part 2 of 5).

Figure 11: Complete runtime Reminder category-judge prompt (Part 3 of 5).

Figure 12: Complete runtime Reminder category-judge prompt (Part 4 of 5).

Figure 13: Complete runtime Reminder category-judge prompt (Part 5 of 5).

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 321 papers

Opera: A Verbal Critic Framework for Long-horizon Coding Agents
Sep 27, 2026Kai Mei, Zhiyuan Hu, Yutong Dai +7AI Coding Agents AI Agent Auditing
Salesforce AI Research · Rutgers University · Lehigh University
ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation
Aug 11, 2026Ximo Zhu, Ruiqi Liu, Rong Wang +8Language Model Distillation On-Policy Distillation
Hello Group Inc. · Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS +1
RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation
Sep 26, 2026Jiawei Zhang, Xiangrong Zhang, Rui Song +3Robot Policy Learning On-Policy Distillation
Jilin University, Changchun, China · Dalian University of Technology, Dalian, China

Sep 27, 2026J/K move · Enter open · S save

Opera: A Verbal Critic Framework for Long-horizon Coding Agents

Kai Mei, Zhiyuan Hu, Yutong Dai, Juntao Tan+6

Salesforce AI Research · Rutgers University · Lehigh University

AI Coding Agents AI Agent Auditing Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Long-horizon coding agents need timely corrections, yet feedback can be ineffective or even harmful when it misjudges ongoing work or fails to address the underlying problem. Existing critics focus on evaluating trajectories and generating feedback, but rarely track what happens after feedback is delivered. We present Opera, a verbal critic framework that treats each correction as a persistent note, followed until the diagnosed problem is resolved. Opera decides when to review through periodic and event-driven triggers, diagnoses issues with typed operators, audits feedback against visible evidence before delivery, and tracks the agent's subsequent actions to distinguish mere compliance from actual resolution. As a test-time critic, Opera improves the resolve rate of non-critic agents by up to 12.4, 15.0, and 8.9 percentage points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1, respectively, across four policy models, and achieves the highest mean resolve rate among competitive critic baselines on all three benchmarks, and also improves policy models when the policy critiques itself. Beyond inference, Opera-guided rollouts provide approximately on-policy training data: fine-tuning Qwen3.5-9B on them improves its resolve rate on held-out SWE-Bench Pro repositories by 10.2 percentage points without a critic at inference time, matching fine-tuning on rollouts from a stronger model, while preserving its performance when switching harness, i.e., from Openhands to Terminus-2, which the latter substantially degrades. Our code is available at: https://github.com/dongyuanjushi/Opera.

Abstract

AI Coding Agents

Explore similar work

Opera: A Verbal Critic Framework for Long-horizon Coding Agents

ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation

RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation