Organizations: National University of Singapore, Singapore · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Tsinghua University, Beijing, China · Independent Researchers
Long-horizon coding agents can observe a failure and generate a plausible critique yet continue along the same incorrect trajectory. We call this the Critique-to-Action Handoff Gap. We introduce RIPPLE, a recovery-oriented training framework that internalizes process-level guidance into a single self-recovering policy. A prefix-only teacher provides evidence-grounded recovery supervision without access to future actions or terminal outcomes. After lightweight distillation, reinforcement learning assigns stage-specific credit to reflection triggering, diagnosis, and corrective action. Local validity signals gate trajectory advantages so that intermediate recovery decisions do not indiscriminately inherit terminal outcomes. Across four repository-level software-engineering benchmarks, RIPPLE improves the base policy by an arithmetic mean of 4.8 score points, with no auxiliary critic at deployment. Controlled ablations isolate the benefits of local, stage-aligned credit, and matched-prefix interventions show that critique semantics affect subsequent correction. Recovery gains concentrate where the prefix contains sufficient diagnostic evidence and the repository state remains reversible within the remaining budget.
Figures & tables
Figure 1: The critique-to-action handoff gap. Failure feedback may elicit an evidence-grounded critique without changing subsequent behavior (left). RIPPLE learns a recovery policy over reflection triggering, diagnosis, and corrective action (right).
Figure 2: Overview of RIPPLE . A prefix-only recovery teacher supports policy initialization and process evaluation. The policy is then optimized through recovery-oriented reinforcement learning, which assigns stage-specific process credit to trigger, diagnosis, and corrective-action decisions, with the executable verifier as the sole source of task reward.
Table 1: End-to-end results and deployment cost. The first three benchmarks use Resolved@1; SWE-Compass uses its official task-aligned macro average. Actor tokens and p95 latency are realized quantities normalized to the base policy. Values are averages over three training seeds for trained methods and three matched decoding runs for fixed methods.
Figure 3: Locality and stage alignment contribute separately. Seven credit-assignment variants use identical rollouts and process labels. Bars report Pro Resolved@1 and Full-Loop Recovery; the dashed line reports positive-credit contamination (lower is better). Event and Stage/event denote event-local generic and stage-aware event-wide credit; Random and Shuffle are sparsity controls.
Method
Trig. F1
Diag.
Align.
Func.
Full
Periodic reflection
0.43
55.8
48.1
42.5
31.4
Recovery-SFT
0.56
64.9
54.8
49.4
38.1
Outcome-only
0.60
66.9
58.6
53.0
41.6
ICRL-style
0.65
73.2
64.4
60.5
50.8
RIPPLE
0.79
81.5
74.8
70.2
59.6
Table 2: Stage-wise recovery quality on the fixed Recovery Challenge Set, scored by an independent evaluator and real test transitions. Columns are trigger F1, Diagnosis Accuracy, Action Alignment, Functional Recovery, and Full-Loop Recovery; the last four are percentages.
Figure 4: Matched-prefix critique intervention. Holding policy, prefix, repository snapshot, remaining budget, and paired seeds fixed, the left panel reports K -step recovery and the right panel pass-to-fail regression for four critique conditions. The controlled contrasts show that a full critique outperforms both removal and a length- and style-matched generic critique, whereas a wrong-direction critique is worse than removal.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Definition
Trigger P/R/F1
Emit <recover> within w=4 tool turns of an annotated boundary. Non-deviating prefixes contribute false positives.
Diagnosis Accuracy
Root cause and invalidated assumption agree with the independent evaluator; cited evidence occurs in the prefix.
Action Alignment
Actions follow the diagnosed direction within K=8 turns.
Functional Recovery
Real execution gives ∣F2P∣>0 and ∣F2P∣−∣P2F∣>0 .
Full-Loop Recovery
Trigger, diagnosis, alignment, and functional recovery all succeed.
Positive-credit contamination
Fraction of locally invalid blocks receiving positive advantage; lower is better.
Appendix
Table B.1: Process metrics. Trigger scores range from 0 to 1; all other metrics are percentages. Grounded Evidence is auxiliary because prefix-present evidence can still support an incorrect diagnosis.
Setting
Value
Actor / scaffold
Qwen3.5-27B / common edit–execute loop
Tool-turn cap
150
Trigger tolerance
w=4 turns
Action horizon
K=8 turns
Seeds
3
Initialization
Shared Recovery-SFT checkpoint
Appendix
Table C.1: Implementation settings relevant to comparison validity and deployment cost. Optimizer-level hyperparameters are shared across the controlled credit-assignment variants.
Figure D.1: Pro resolution by static-difficulty quartile. Gains increase from 1.8 points (Easy) to 11.9 (Extreme). Difficulty is defined without evaluated-policy trajectory statistics.
Figure D.2: Recoverable-deviation prevalence and benchmark gain. The reported r=0.68 association is computed over repository clusters, not these four aggregate points.
Slice (weight)
Base
RIPPLE
Gain
SWE-Compass
Bug fixing (1/8)
23.4
32.2
+8.8
Feature implementation (1/8)
25.8
33.4
+7.6
Other task types (6/8)
24.2
30.0
+5.8
Multilingual
C/C++/Go/Rust (4/9)
61.4
69.4
+8.0
Appendix
Table D.2: Task, language, and distribution-shift slices. Weighted rows reproduce their benchmark aggregates up to rounding. Pro’s metadata-shift score is fixed before inference and split at its median.
Method
Repair
Damage
Net
Longer single trajectory
4.4
3.5
+0.9
Two restarts
6.0
3.6
+2.4
Prompted periodic reflection
6.6
4.8
+1.8
Recovery-SFT
6.2
3.2
+3.0
Outcome-only GRPO
6.6
2.9
+3.7
Pivotal-aware credit
7.2
2.7
+4.5
Appendix
Table D.3: Paired repair-versus-damage decomposition on Pro. Every row uses all Pro instances as the denominator, so Repair minus Damage exactly equals the Resolved@1 gain over the base policy. RIPPLE achieves both the highest repair rate and the lowest damage rate among the trained comparisons.
Figure E.1: Leave-one-stage-out end-to-end effects. Removing corrective-action credit produces the largest drop in Pro and Full-Loop Recovery.
Figure E.2: Stage-wise ablation. Trigger credit primarily affects initiation; diagnosis credit affects diagnosis and downstream action; removing action credit leaves upstream stages largely unchanged. Trigger F1 is shown on a 0–1 scale; other cells are percentages.
Figure F.1: Matched-prefix recovery–regression frontier. Better outcomes move toward the upper-left. Direction alone retains much of the benefit; delaying the critique forfeits a substantial share.
Condition
Recovery
P2F
Full critique
62.9
2.4
Remove critique
39.8
6.8
Generic critique
42.0
6.1
Wrong-direction critique
25.6
18.9
Direction only
55.4
4.1
Evidence only
44.7
5.7
Appendix
Table F.1: Extended matched-prefix interventions (%). All conditions share the policy, frozen prefix, repository snapshot, remaining budget, and paired seed. The internal critique is within 3.6 recovery points and 0.3 P2F points of the teacher reference.
Reinforcement learning (RL) has driven substantial progress in coding agents by relying on verified successful trajectories as learning signals. However, such trajectories are scarce in long-horizon software engineering tasks, where a minor error can invalidate the entire rollout. Repeatedly restarting from the initial state is commonly used yet inefficient, and resetting only the context with working directory misses state changes such as installed dependencies. Therefore, branching from intermediate steps to preserve earlier progress requires restoring both the agent context and execution environments. In this paper, we present SPROUT (SnaPshot-based RollOUT), an agentic RL rollout framework that scales long-horizon coding tasks by resuming and branching from intermediate states. Specifically, SPROUT was built on top of a lightweight infrastructure that checkpoints both the filesystem and the agent context after each assistant turn. A verifier-guided backtracking policy then selects where to resume a failed attempt and how to proceed. Extensive experiments on long-horizon coding tasks show that SPROUT achieves up to 23.6% improvement over pass@k across various LLMs, while using 32--52 of their tokens. For RL training of Qwen3.8-27B, SPROUT stably outperforms standard GRPO by 13.9 points on three long-horizon coding benchmarks while cutting the time per training step by 39.5. SPROUT thus makes RL on long-horizon coding tasks both more effective and cheaper to scale.
Mengyang Liu, Weiming Wang, Haixin Nan +6
Shanghai Jiao Tong University · Amazon · Nanjing University +2
Code-as-Policy agents accomplish long-horizon embodied tasks by generating and executing code, yet continually improving them with teachers that are stronger but not globally reliable remains a key challenge. Existing distillation methods typically treat the teacher's complete behavior as the supervision target and thus misassign training credit on states where the teacher fails. We propose RE-0, a recursively verified policy improvement framework: rather than assuming that the teacher globally outperforms the student, RE-0 requests local corrections from the teacher on the student's own failure histories and checks in the environment whether each correction is genuinely beneficial; verified corrections yield immediate improvement. Building on this, we propose RE-OPD, which turns verified interventions into supervision for on-policy distillation. Only counterfactually verified teacher interventions provide distribution-level supervision, weighted by their measured local benefit, and the improvement they induce is projected back into the standalone student, so both where supervision is applied and how much credit the teacher receives co-evolve with the student policy. We further prove that the student's per-round gain is lower-bounded by its verified intervention gain up to verification and projection error terms. Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes.
Jiawei Zhang, Xiangrong Zhang, Rui Song +3
Jilin University, Changchun, China · Dalian University of Technology, Dalian, China
Reliable repository-level coding requires agents not only to generate correct code but also to preserve constraints and respond to tool feedback across long interactions. However, our diagnostic analysis shows that agents often fail to act on relevant information already available in context, while final Pass/Fail outcomes provide only sparse process supervision. We introduce On-Policy Reminder Distillation (OPRD), which combines Multi-Scope Reminder supervision with outcome-based routing on reminder-free student rollouts. Successful rollouts provide autoregressive self-imitation targets, whereas failed rollouts receive Reminder-conditioned reverse-KL supervision from a teacher that re-evaluates the same student-generated prefixes. A Global Reminder converts final-test diagnostics into trajectory-level guidance, while Turn-local Reminders apply only to the next decision following the relevant environment observation. The teacher and student share a two-stage initialization from Coding SFT and Instruction-Following (IF) SFT. At inference, the student operates without Reminders. On SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual, OPRD improves the benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model. Under the same diagnostic protocol, it reduces the aggregate error-event count from 446 to 222, a 50.2% reduction.