SPROUT: Scaling Agentic Reinforcement Learning for Long-Horizon Software Engineering

Mengyang Liu; Weiming Wang; Haixin Nan; Yu Wu; Peize Li; Shihong Huang; Xuning He; Xue Jiang

cs.SEOct 9, 2026

SPROUT: Scaling Agentic Reinforcement Learning for Long-Horizon Software Engineering

Authors: Mengyang Liu, Weiming Wang, Haixin Nan, Yu Wu, Peize Li, Shihong Huang, Xuning He, Xue Jiang,

Organizations: Shanghai Jiao Tong University · Amazon · Nanjing University · King’s College London ·

Abstract

Reinforcement learning (RL) has driven substantial progress in coding agents by relying on verified successful trajectories as learning signals. However, such trajectories are scarce in long-horizon software engineering tasks, where a minor error can invalidate the entire rollout. Repeatedly restarting from the initial state is commonly used yet inefficient, and resetting only the context with working directory misses state changes such as installed dependencies. Therefore, branching from intermediate steps to preserve earlier progress requires restoring both the agent context and execution environments. In this paper, we present SPROUT (SnaPshot-based RollOUT), an agentic RL rollout framework that scales long-horizon coding tasks by resuming and branching from intermediate states. Specifically, SPROUT was built on top of a lightweight infrastructure that checkpoints both the filesystem and the agent context after each assistant turn. A verifier-guided backtracking policy then selects where to resume a failed attempt and how to proceed. Extensive experiments on long-horizon coding tasks show that SPROUT achieves up to 23.6% improvement over pass@k across various LLMs, while using $32$ -- $52%$ of their tokens. For RL training of Qwen3.8-27B, SPROUT stably outperforms standard GRPO by $13.9$ points on three long-horizon coding benchmarks while cutting the time per training step by $39.5%$ . SPROUT thus makes RL on long-horizon coding tasks both more effective and cheaper to scale.

Figures & tables

Figure 1: SPROUT: branch from checkpoints instead of restarting. Resuming a failed rollout from the step where it went wrong beats resampling it from scratch, at inference time and in RL. (a) DeepSWE, four models: SPROUT beats Pass@4 on every model while spending 48–68% fewer tokens; Shepherd and entropy-based branching do not. (b) RL with Qwen3.8-27B: training on SPROUT rollouts improves over GRPO by 7–25 pp on three mainstream agentic benchmarks. Each RL step is 40% faster and cheaper than sampling from the beginning.

Figure 2: SPROUT reuses failed rollouts for efficient exploration. (a) Independent rollouts repeat costly prefixes. (b) Guided by verifier feedback, the same agent policy resumes from paired context and VM checkpoints to explore new continuations.

Panel of Table 1 — Table 1: Comparison with representative approaches. full, partial, no support; – not applicable. Consistent : the agent context and the environment are restored to the same step. Full scope : the restored environment covers the whole disk, not only a workspace. Per-step : every tool call of a long trajectory yields a restorable checkpoint. Compact : checkpoints keep incremental disk changes, snapshot is cheap enough for taking snapshot for each tool call. Failure-aware : branching uses verifier feedback on the failed attempt. Self-guided : the trained policy, not a stronger model, makes the branching decisions.

Figure 3: SPROUT workflow: checkpoint, branch, and learn. (a) After each turn, the execution layer saves a filesystem checkpoint paired with the matching agent context, so any eligible boundary can be restored in a fresh sandbox. (b) When all root attempts fail, the actor, prompted as analyst and reviewer, reads the failures together with verifier feedback, selects boundaries, and writes a starting turn for each branch. Each branch restores its paired state, executes the starting turn, and continues under the actor policy. (c) Roots and branches, with their verified rewards, form training groups; the review context is removed from the learner’s input.

Model	Metric	Pass@1	Pass@4	Entropy-	Shepherd	SPROUT
Model	Metric	(Base)		based
Qwen3.8 27B	Resolve rate (%) $\uparrow$	$37.2$	$71.7$	$71.7$	$67.3$	$\mathbf{78.6}$
	Recovery (%) $\uparrow$	$0.0$	$54.9$	$54.9$	$47.9$	$\mathbf{64.8}$
	Steps $\downarrow$	$16{,}708$	$64{,}136$	$45{,}549$	$55{,}280$	$\mathbf{23{,}174}$
	Tokens (M) $\downarrow$	$1921$	$8673$	$8589$	$9026$	$\mathbf{4123}$
Kimi K3	Resolve rate (%) $\uparrow$	$72.3$	$92.0$	–	$78.8$	$\mathbf{98.2}$

Table 2: Main experimental comparison on DeepSWE. Resolve rate, recovery rate, total execution steps, and total tokens (M). A dash (–) indicates that Entropy-based sampling is unavailable. Best values are bolded, excluding Pass@1 (Base).

Benchmark	Metric	Pass@1	Pass@4	Entropy-	Shepherd	SPROUT
Benchmark	Metric	(Base)		based
DeepSWE	Resolve rate (%)	$37.2$	$71.7$	$71.7$	$67.3$	$\mathbf{78.6}$
	Recovery (%)	$0.0$	$54.9$	$54.9$	$47.9$	$\mathbf{64.8}$
	Steps	$16708$	$64136$	$45549$	$55280$	$\mathbf{23174}$
	Tokens (M)	$1921$	$8673$	$8589$	$9026$	$\mathbf{4123}$
Terminal- Bench 2.1	Resolve rate (%)	$49.4$	$66.3$	$76.4$	$79.8$	$\mathbf{89.9}$

Table 3: Experimental comparison across benchmarks for Qwen3.8 27B. Resolve rate, recovery rate, total execution steps, and total tokens (M). Best values are bolded, excluding Pass@1.

Method	SWE-bench Pro v2 Hard	DeepSWE	Terminal-Bench 2.1	Time per step ( $\downarrow$ )
Base (no RL)	$62.7$	$37.2$	$49.4$	n/a
Basic GRPO	$74.5$	$33.6$	$39.3$	$3.77\,\mathrm{h}$
SPROUT	$\mathbf{84.3}\,{\color[rgb]{0.7,0,0}(\uparrow\,9.8\,\mathrm{pp})}$	$\mathbf{40.7}\,{\color[rgb]{0.7,0,0}(\uparrow\,7.1\,\mathrm{pp})}$	$\mathbf{64.0}\,{\color[rgb]{0.7,0,0}(\uparrow\,24.7\,\mathrm{pp})}$	$\mathbf{2.28}\,\mathrm{h}\,{\color[rgb]{0.7,0,0}(\downarrow\,39.5\%)}$

Table 4: Training comparison. Benchmark columns report success rates in percent. Red annotations show SPROUT minus Basic GRPO in percentage points (pp) for success rates, and the relative reduction for time per step.

Variant	Change	Res. $\uparrow$	$\bm{\Delta}$	Branches $\downarrow$	Steps $\downarrow$	Tok. $\downarrow$
SPROUT (tool-call hint)	full method	$\mathbf{85.8}$	–	$325$	$2{,}839$	$261.9$
Continuation guidance
Rollback, no hint	no injected guidance	$49.6$	$-36.2$	$\mathbf{183}$	$\mathbf{1{,}755}$	$\mathbf{209.8}$
Branch selection
Reward-only review	binary outcome only	$53.1$	$-32.7$	$153$	$2{,}296$	$274.2$

Table 5: Ablations of SPROUT on DeepSWE using GPT-5.6 Luna . Each variant changes one component and keeps the rest of the collection procedure fixed. Res. is the resolve rate (%) and

\Delta

the change relative to SPROUT . Branches, Steps (tool calls), and Tok. (M) count the branching stage only, excluding the root run.

Figure 4: Branch scaling and round allocation. Resolve rate on DeepSWE using GPT-5.6 Luna, where labels denote review rounds

\times

branches per round and all methods share the same settings.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Model	Qwen3.8-27B
Fine-tuning	Full parameters
Prompts per update	32
Rollouts per prompt	8
Nominal global batch size	256 rollouts
Maximum context length	128K tokens

Appendix

Table 6: RL hyperparameters and training configuration.

Method	Source / Setting	Total Hit Rate	Luna	Qwen	Precision	Latency	Retained
No Cache	Baseline	0/248	0/136	0/112	—	0 ms	134/134
Exact Full History	Strict Trace Match	2/248 (0.8%)	2/136 (1.5%)	0/112 (0.0%)	100%	0.0003 ms	133/134
Official TVCACHE TCG	TVCACHE ( Kumar et al., 2026 )	8/248 (3.2%)	8/136 (5.9%)	0/112 (0.0%)	100%	0.0017 ms	133/134
Full-Workspace Hash	Exhaustive Scan	91/248 (36.7%)	65/136 (47.8%)	26/112 (23.2%)	100%	79.1 ms	38/134
Ours	Relaxed Projection	35/248 (14.1%)	28/136 (20.6%)	7/112 (6.2%)	100%	0.201 ms	82/134

Appendix

Table 7: Evaluation of intermediate state matching and caching mechanisms. Evaluated on 248 trajectories across 134 candidate tasks. Hit rates report both the total ratio and the breakdown by query policy. Online latency measures the verification time per candidate checkpoint, and retained candidates denote unresolved task slots remaining after cache reuse.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 226 papers

Scaling Test-Time Compute for Agentic Coding
Apr 16, 2026Joongwon Kim, Wannan Yang, Kelvin Niu +13AI Coding Agents Test-Time Scaling
1Meta Superintelligence Labs · University of Washington · 3New York University +4
CodeMidas: Scaling Agentic Coding RL Environments from Code Itself
Sep 18, 2026Bowen Ye, Lei Li, Shicheng Li +16LLM Agent Training RL for Code Generation
LLM Core, Xiaomi · Peking University · University of Hong Kong +1
ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents
Mar 19, 2026Hao Zhang, Mingjie Liu, Shaokun Zhang +10LLM Agent Training LLM Agent Harnesses

Apr 16, 2026J/K move · Enter open · S save

Scaling Test-Time Compute for Agentic Coding

Joongwon Kim, Wannan Yang, Kelvin Niu, Hongming Zhang+12

1Meta Superintelligence Labs · University of Washington · 3New York University+4

AI Coding Agents Test-Time Scaling Long-Horizon LLM Agents

Open paper AskPDF arXiv

Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.

Abstract

Agentic RL

Explore similar work

Scaling Test-Time Compute for Agentic Coding

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents