Converting verifier outcomes into useful policy updates and partial progress into complete task resolution is crucial for terminal agents. We present Terminal-RSI, which alternates Recursive Self-Rewrite (RSR) of verified trajectories with critic-inheriting Recursive Policy-Refine (RPR). We formulate Turn-Boundary Skip-Observation GAE (TB-GAE) to retain token-specific baselines while applying decay across interaction turns. Family Dense Reward (FDR) calibrates semantic progress across task families; Success-Prioritized Reward (SPR) separates strict success from bounded partial credit. A Warm-up Critic Model, recursive critic handoff, and stage-specific learning rates coordinate the two refinement phases. The five-stage sequence from Qwen3.8-27B base model through RSR-1, RPR-1, RSR-2, and RPR-2 yields Terminal-RSI-27B: 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard. On LHTB, process reward rises from 0.21 to 0.37 and strict completion from 0/46 to 5/46; SWR100 success doubles from 3.00% to 6.00%. By recursively coupling verified trajectory rewriting with critic-inheriting policy refinement, each retained stage consolidates discovered behaviors and progressively advances the agent from partial local repairs to strict terminal verification.
Figures & tables
Figure 2: The recursive loop carries critic weights through supervised transitions. Blue blocks distinguish actor-only RSR from joint actor–critic RPR; the bypass preserves the critic, and feedback returns the retained policy to the next cycle. The initial actor is RSR-1; Warm-up Critic Model denotes the prepared value network.
Figure 3: Rewriting and policy refinement update different parameters. The left branch discovers, rewrites, and verifies actor demonstrations; Recursive Policy-Refine (RPR) combines inherited values, reward calibration, and Turn-Boundary Skip-Observation GAE (TB-GAE). Phase-local critic calibration comprises four critic-only updates before joint training, not before each PPO minibatch.
Setting
RPR-1
RPR-2
Initial actor
RSR-1
RSR-2
Initial critic
Warm-up Critic Model
Retained RPR-1 critic
Reward
FDR
SPR
Actor learning rate
5×10−7
1×10−7
Critic learning rate
1.5×10−5
1.5×10−5
Critic-only calibration
4 updates
4 updates
Table 1: Refinement adapts policy updates while preserving the critic learning scale. Shared settings appear below; the ordinary RPR-2 branch uses FDR rather than SPR.
Checkpoint
Terminal-Bench 4.0
Terminal-Bench 2.1
Terminal-Bench 3.0
Terminal-Bench Hard
LHTB
SWR100
Base
0.00
51.70
0.00
31.00
0.21
3.00
PPO
0.00 = 0.00
48.31 ↓ 3.39
1.43 ↑ 1.43
39.00 ↑ 8.00
–
–
RSR-1
6.10 ↑ 6.10
70.10 ↑ 18.40
6.80 ↑ 6.80
54.00 ↑ 23.00
0.29 ↑ 0.08
5.00 ↑ 2.00
RPR-1
16.67 ↑ 10.57
77.53 ↑ 7.43
14.29 ↑ 7.49
69.50 ↑ 15.50
0.30 ↑ 0.01
6.00 ↑ 1.00
RSR-2
17.46 ↑ 0.79
86.52 ↑ 8.99
15.00 ↑ 0.71
76.00 ↑ 6.50
0.31 ↑ 0.01
6.00 = 0.00
Terminal-RSI-27B
19.84 ↑ 2.38
86.52 = 0.00
16.95 ↑ 1.95
79.00 ↑ 3.00
0.37 ↑ 0.06
6.00 = 0.00
Table 2: The retained sequence improves all benchmark scores over Base. Retained-stage rows report Mean@3 success (%); the separate PPO row is a historical aggregate under Appendix E ’s protocol. Best scores, including ties, are bold. Arrows give percentage-point changes from retained predecessors, or the primary Base entry for PPO. Terminal-Bench 4.0, Terminal-Bench 2.1, Terminal-Bench 3.0, Terminal-Bench Hard and SWR100 use percentage of accuracy as performance, LHTB use dense process reward as performance for evaluation.
Figure 4: Performance comparison between Terminal-RSI-27B and state-of-the-art frontier models on terminal agent task benchmarks. Blue bars highlight the success rate achieved by our proposed Terminal-RSI-27B model, while gray bars present the corresponding scores of all included reference frontier models. The left and right panels separately demonstrate the comparative results on Terminal-Bench 4.0 and Terminal-Bench 2.1
Figure 5: Mean Success Progression. Evaluation across Terminal-Bench 4.0, 3.0, 2.1, and Hard yields an equal-weight success mean rising from 20.68% to 50.58% . The vertical axis spans 10.00%–60.00%.
Figure 6: Retained gains coexist with benchmark-dependent regressions. Diamonds denote RSR, stars retained RPR, and gray branches discarded alternatives; the larger final star marks Terminal-RSI-27B at RPR-2 step 18. Values are success (%): retained stages use Mean@3, while the separate PPO control retains its historical protocol (Appendix E ). Lower panels begin at 40.00% and 25.00%. Stage spacing is ordinal, not equal compute.
Figure 7: Dense and binary rewards rise over the displayed RPR-1 window. Interaction length varies and train–inference mismatch remains nonzero. Pale traces show raw observations; dark traces show causal means of up to five observations. The horizontal coordinate is calibrated for display, not a common training clock.
Figure 8: RPR-2 reward rises under both settings, while interaction length diverges. Blue solid curves denote SPR; dark-gray dashed curves denote the FDR branch. Pale traces show raw observations and emphasized curves causal means of up to five observations. Reward and turn panels retain distinct native clocks; different shaping scales are not comparable strict-success rates.
Figure 9: RPR-2 critic EV remains positive throughout the recorded window. Warm-Up begins negative and RPR-1 briefly crosses zero. All 16 RPR-2 observations remain positive without smoothing or imputation.
Figure 10: Structural strata reveal uneven capability gains. Darker curves denote harder fixed groups; bold labels mark RPR-2. Values use one linear scale. Easy reaches 72.88%, while Extreme dips and recovers. Scores are task-weighted; the Terminal-Bench 2.1 RSR-1 analysis uses an alternative rewrite checkpoint.
Figure 11: Reward allocation varies across families and phases. Panels show FDR and SPR profiles for the same 46 families. Solid curves encode mean reward; dashed curves show pooled verifier counts on a separate scale. Polygon area and smoothness are not variance measures.
Figure 12: Final-stage replays satisfy the tested execution contracts. Blue stairs show mean checker completion, gray dots scored repeats, and dashed lines Base. The blue star marks RPR-2 (Terminal-RSI-27B), evaluated separately from RSR-2; both attain 100.00% in two repeats per task. Axes begin at 70.00% and 20.00%, respectively. These checker-level means are not benchmark Mean@3 success.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value or behavior
Evaluation repeats
Three independent runs; arithmetic mean (Mean@3)
Sampling
Temperature 0.60; top- p=0.95
Generation / input budget
8,192 output tokens per turn / 96,000 input tokens per call
Table 3: Final benchmark protocol, distinct from training monitoring. Context and interaction budgets are separate limits; benchmark-specific termination and time limits still apply.
Figure 13: RSR-first initialization remains stronger at the observed common local steps. The blue starting diamond marks RSR-1, filled blue circles mark intermediate RPR observations, and the blue star marks retained step 20; gray squares trace PPO from its own measured initialization. Every observed point has a numerical label. Lines join measured success rates without smoothing or imputation. Local steps align update counts rather than total compute because RSR-first refinement also incurs discovery and supervised-rewriting costs. Historical repeat protocols require reconciliation before treating the comparison as uniformly matched.
Component
Comparison
Primary diagnostic
Credit assignment
Generated-token GAE; skip-only; TB-GAE, with matched rewards and terminal/chunk boundaries
Horizon-stratified advantage or gradient variance; strict success; boundary correctness
Reward evolution
FDR; saturation-only; success-offset-only; full SPR, with family calibration varied separately
Strict success; early failure; reward calibration
Critic handoff
Inherit/reset crossed with warmup under matched calibration budgets
Continued RPR; rollback; alternating RSR and RPR, including discovery, SFT, and RL costs
Worst-suite retention under predefined validation rules and matched total compute
Appendix
Table 5: Component-specific controls are needed for mechanistic attribution. These are prospective experiments rather than measured effect estimates. Comparisons should match task streams, initialization, effective budgets, and repeated training seeds, while recording terminal reasons and strict success.
Figure 14: Five-stage auxiliary evaluation shows continued LHTB gains and an SWR100 plateau. Blue-shaded bars follow Base, RSR-1, RPR-1 at step 20, RSR-2, and RPR-2 at step 18. Dashed references mark the same final Terminal-RSI-27B checkpoint rather than per-suite maxima. LHTB bar heights encode process rewards of 0.21, 0.29, 0.30, 0.31, and 0.37, with separate strict-completion entries of 0/46, 0/46, 2/46, 4/46, and 5/46. SWR100 success is 3.00%, 5.00%, 6.00%, 6.00%, and 6.00%. All checkpoints share the 27B backbone; values come from the stage ledger without imputation. Appendix E distinguishes compact completion entries from run-level aggregation.
Figure 15: Separately recorded final-model executions pass the statistical checks. Five rows progress from Base to RPR-2 (Terminal-RSI-27B); percentages report mean checker completion over scored repeats. Each row links three selected action, implementation, and verification excerpts. RSR-2’s numerical probes differ from RPR-2’s six-module repair; both pass all 11 assertions in two trials. Green denotes passing evidence, not an error-free history; yellow denotes residual or mixed failures. Rows are ordered stages, not equal-duration executions.
Figure 16: Separately recorded final-stage trials satisfy the tested output contract. Five rows show separate stage evaluations; percentages average the wrapper and fuzz-equivalence gates. Base and RSR-1 retain a format mismatch; RPR-1 has mixed outcomes. RSR-2 and RPR-2 each pass both gates in two trials. The first final-model trial builds C/Rust components but delegates expression evaluation to Python. Green denotes tested output equivalence, not a wholly native AST path or memory safety; missing scores are not imputed.
Successful trajectories on difficult tasks provide valuable supervision for model improvement, but specialized harnesses introduce interventions that may be unavailable during deployment. We propose Recursive Self-Rewrite (RSR), a framework that uses one base model, Qwen-3.8-27B, to discover successful solutions under diverse harnesses and reconstruct them as training trajectories under a general harness. A planner extracts procedures into runbooks, a critic screens for verifier and solution leakage and guides recursive revision, and an executor follows qualified runbooks in fresh sandboxes. Across approximately 3K self-curated terminal tasks, three harnesses jointly solve 759 tasks, 34.3% more than the strongest individual harness in the recorded pool. RSR expands 2,001 successful source trajectories into 11,094 rewritten trajectories for supervised finetuning. Training on these trajectories outperforms both the base model and direct trajectory SFT. Compared with the base model, pass@3 increases from 57.0% to 74.2% on Terminal-Bench 2, from 1.5% to 9.1% on Terminal-Bench 4, from 39.0% to 63.0% on our self-curated Terminal-Bench Hard, and from 3.0% to 6.0% on our Software Terminal-Bench. Process reward on Long-Horizon Terminal-Bench rises from 0.21 to 0.29. These results show how diverse harness-assisted experiences can be reconstructed into reusable capabilities for a model operating under a general harness.
Zongxia Li, Yucheng Shi, Zhongzhi Li +7
Tencent HY LLM Frontier · University of Maryland, College Park · University of Georgia +4
Terminal agents have shown strong abilities in complex coding tasks. However, they often struggle to recover from errors during long-running execution (e.g., command errors, faulty implementations), eventually leading to task failures. To this end, we present ReTerm, a self-improving terminal agent trained by learning how to recover from its own failed terminal tasks. To support its training, we construct a customized data pipeline and produce ReTerm-10K, a high-quality dataset that provides effective recovery demonstrations for SFT cold start. Furthermore, we propose ReTerm-GRPO, an online RL algorithm that forks from the agent's own failure states to generate multiple independent self-recovery attempts. By learning from whether these attempts successfully complete the task, the agent self-improves through its own recovery experience. Built on Qwen3.5-9B, our ReTerm-9B achieves 34.8% Pass@1 on Terminal-Bench 2.0, improving over the base model by 14.6 points. Further analysis shows that our method achieves consistently higher recovery rates across diverse error categories compared with standard GRPO training. All code, data and model are released at https://github.com/twothsnd/ReTerm.
Wei Xie, Kaituo Feng, Shiming Fu +3
Fudan University · MMLab, The Chinese University of Hong Kong · Shanghai Jiao Tong University
Terminal agents are a compelling application of large language models (LLMs), with the potential to integrate deeply into users' daily workflows. Reinforcement learning (RL) is a key technique for improving their capabilities, making scalable training environments a central challenge. Since public real-user interaction data are scarce, synthetic environments provide a practical alternative, but often suffer from domain gaps and limited fidelity, leading to poor generalization. Existing work mainly scales the quantity and diversity of synthetic environments, while reward-signal quality and the mechanisms governing generalization remain under-explored. We study how RL improves terminal agents and propose the Agentic Compositional Generalization hypothesis: rather than teaching new domain-specific skills from scratch, RL primarily shapes high-level decision-making behaviors that compose and route low-level skills acquired during pre-training and supervised fine-tuning (SFT). This account is consistent with our empirical results and suggests that verifier quality, which determines which behaviors are reinforced, is more important than simply increasing environment quantity or diversity. Motivated by this insight, we propose River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization. Using this recipe, our RL-trained agent achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks. River also generalizes across model families, scales, agent harnesses, and RL objectives. Using fewer than 30% of the TMax training environments, River improves RL gains by 106% and 30% on average for models ranging from 2B to 27B on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively.
Yihang Yao, Bo Pang, Xuan Phi Nguyen +3
Salesforce AI Research · Carnegie Mellon University