Reinforcement learning (RL) has driven substantial progress in coding agents by relying on verified successful trajectories as learning signals. However, such trajectories are scarce in long-horizon software engineering tasks, where a minor error can invalidate the entire rollout. Repeatedly restarting from the initial state is commonly used yet inefficient, and resetting only the context with working directory misses state changes such as installed dependencies. Therefore, branching from intermediate steps to preserve earlier progress requires restoring both the agent context and execution environments. In this paper, we present SPROUT (SnaPshot-based RollOUT), an agentic RL rollout framework that scales long-horizon coding tasks by resuming and branching from intermediate states. Specifically, SPROUT was built on top of a lightweight infrastructure that checkpoints both the filesystem and the agent context after each assistant turn. A verifier-guided backtracking policy then selects where to resume a failed attempt and how to proceed. Extensive experiments on long-horizon coding tasks show that SPROUT achieves up to 23.6% improvement over pass@k across various LLMs, while using 32--52 of their tokens. For RL training of Qwen3.8-27B, SPROUT stably outperforms standard GRPO by 13.9 points on three long-horizon coding benchmarks while cutting the time per training step by 39.5. SPROUT thus makes RL on long-horizon coding tasks both more effective and cheaper to scale.
Figures & tables
Figure 1: SPROUT: branch from checkpoints instead of restarting. Resuming a failed rollout from the step where it went wrong beats resampling it from scratch, at inference time and in RL. (a) DeepSWE, four models: SPROUT beats Pass@4 on every model while spending 48–68% fewer tokens; Shepherd and entropy-based branching do not. (b) RL with Qwen3.8-27B: training on SPROUT rollouts improves over GRPO by 7–25 pp on three mainstream agentic benchmarks. Each RL step is 40% faster and cheaper than sampling from the beginning.
Figure 2: SPROUT reuses failed rollouts for efficient exploration. (a) Independent rollouts repeat costly prefixes. (b) Guided by verifier feedback, the same agent policy resumes from paired context and VM checkpoints to explore new continuations.
Table 1: Comparison with representative approaches. full, partial, no support; – not applicable. Consistent : the agent context and the environment are restored to the same step. Full scope : the restored environment covers the whole disk, not only a workspace. Per-step : every tool call of a long trajectory yields a restorable checkpoint. Compact : checkpoints keep incremental disk changes, snapshot is cheap enough for taking snapshot for each tool call. Failure-aware : branching uses verifier feedback on the failed attempt. Self-guided : the trained policy, not a stronger model, makes the branching decisions.
Figure 3: SPROUT workflow: checkpoint, branch, and learn. (a) After each turn, the execution layer saves a filesystem checkpoint paired with the matching agent context, so any eligible boundary can be restored in a fresh sandbox. (b) When all root attempts fail, the actor, prompted as analyst and reviewer, reads the failures together with verifier feedback, selects boundaries, and writes a starting turn for each branch. Each branch restores its paired state, executes the starting turn, and continues under the actor policy. (c) Roots and branches, with their verified rewards, form training groups; the review context is removed from the learner’s input.
Model
Metric
Pass@1
Pass@4
Entropy-
Shepherd
SPROUT
(Base)
based
Qwen3.8 27B
Resolve rate (%) ↑
37.2
71.7
71.7
67.3
78.6
Recovery (%) ↑
0.0
54.9
54.9
47.9
64.8
Steps ↓
16,708
64,136
45,549
55,280
23,174
Tokens (M) ↓
1921
8673
8589
9026
4123
Kimi K3
Resolve rate (%) ↑
72.3
92.0
–
78.8
98.2
Table 2: Main experimental comparison on DeepSWE. Resolve rate, recovery rate, total execution steps, and total tokens (M). A dash (–) indicates that Entropy-based sampling is unavailable. Best values are bolded, excluding Pass@1 (Base).
Benchmark
Metric
Pass@1
Pass@4
Entropy-
Shepherd
SPROUT
(Base)
based
DeepSWE
Resolve rate (%)
37.2
71.7
71.7
67.3
78.6
Recovery (%)
0.0
54.9
54.9
47.9
64.8
Steps
16708
64136
45549
55280
23174
Tokens (M)
1921
8673
8589
9026
4123
Terminal- Bench 2.1
Resolve rate (%)
49.4
66.3
76.4
79.8
89.9
Table 3: Experimental comparison across benchmarks for Qwen3.8 27B. Resolve rate, recovery rate, total execution steps, and total tokens (M). Best values are bolded, excluding Pass@1.
Table 4: Training comparison. Benchmark columns report success rates in percent. Red annotations show SPROUT minus Basic GRPO in percentage points (pp) for success rates, and the relative reduction for time per step.
Variant
Change
Res. ↑
Δ
Branches ↓
Steps ↓
Tok. ↓
SPROUT (tool-call hint)
full method
85.8
–
325
2,839
261.9
Continuation guidance
Rollback, no hint
no injected guidance
49.6
−36.2
183
1,755
209.8
Branch selection
Reward-only review
binary outcome only
53.1
−32.7
153
2,296
274.2
Table 5: Ablations of SPROUT on DeepSWE using GPT-5.6 Luna . Each variant changes one component and keeps the rest of the collection procedure fixed. Res. is the resolve rate (%) and Δ the change relative to SPROUT . Branches, Steps (tool calls), and Tok. (M) count the branching stage only, excluding the root run.
Figure 4: Branch scaling and round allocation. Resolve rate on DeepSWE using GPT-5.6 Luna, where labels denote review rounds × branches per round and all methods share the same settings.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Model
Qwen3.8-27B
Fine-tuning
Full parameters
Prompts per update
32
Rollouts per prompt
8
Nominal global batch size
256 rollouts
Maximum context length
128K tokens
Appendix
Table 6: RL hyperparameters and training configuration.
Method
Source / Setting
Total Hit Rate
Luna
Qwen
Precision
Latency
Retained
No Cache
Baseline
0/248
0/136
0/112
—
0 ms
134/134
Exact Full History
Strict Trace Match
2/248 (0.8%)
2/136 (1.5%)
0/112 (0.0%)
100%
0.0003 ms
133/134
Official TVCACHE TCG
TVCACHE ( Kumar et al., 2026 )
8/248 (3.2%)
8/136 (5.9%)
0/112 (0.0%)
100%
0.0017 ms
133/134
Full-Workspace Hash
Exhaustive Scan
91/248 (36.7%)
65/136 (47.8%)
26/112 (23.2%)
100%
79.1 ms
38/134
Ours
Relaxed Projection
35/248 (14.1%)
28/136 (20.6%)
7/112 (6.2%)
100%
0.201 ms
82/134
Appendix
Table 7: Evaluation of intermediate state matching and caching mechanisms. Evaluated on 248 trajectories across 134 candidate tasks. Hit rates report both the total ratio and the breakdown by query policy. Online latency measures the verification time per candidate checkpoint, and retained candidates denote unresolved task slots remaining after cache reuse.
Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.
Joongwon Kim, Wannan Yang, Kelvin Niu +13
1Meta Superintelligence Labs · University of Washington · 3New York University +4
Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich source of such tasks, while existing methods typically rely on development artifacts such as issues and commits, limiting the range of tasks that can be extracted. To better scale RL environments, we present CodeMidas, an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its only task-specific input. CodeMidas allocates agentic compute to every stage of environment construction: agents explore implemented functionality to formulate behavioral specifications, construct tests grounded in execution of the original code, and validate and filter candidate tasks through execution checks and repeated solution rollouts. The resulting dataset has 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains. Training MiMo-V2.5 on these tasks with GRPO improves performance on all five diverse benchmarks, covering issue repair (DeepSWE + 11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%). Ablations show that increasing the number of high-quality training tasks improves performance. Trajectory analysis shows the RL-trained agent demonstrates better behaviors like increasing codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.
Bowen Ye, Lei Li, Shicheng Li +16
LLM Core, Xiaomi · Peking University · University of Hong Kong +1
Multi-turn LLM agents are increasingly important for solving complex, interactive tasks, and reinforcement learning (RL) is a key ingredient for improving their long-horizon behavior. However, RL training requires generating large numbers of sandboxed rollout trajectories, and existing infrastructures often couple rollout orchestration with the training loop, making systems hard to migrate and maintain. Under the rollout-as-a-service philosophy, we present ProRL Agent , a scalable infrastructure that serves the full agentic rollout lifecycle through an API service. ProRL Agent also provides standardized and extensible sandbox environments that support diverse agentic tasks in rootless HPC settings. We validate ProRL Agent through RL training on software engineering, math, STEM, and coding tasks. ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.