cs.AIOct 15, 2026
SaveTerminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine
Abstract
Converting verifier outcomes into useful policy updates and partial progress into complete task resolution is crucial for terminal agents. We present Terminal-RSI, which alternates Recursive Self-Rewrite (RSR) of verified trajectories with critic-inheriting Recursive Policy-Refine (RPR). We formulate Turn-Boundary Skip-Observation GAE (TB-GAE) to retain token-specific baselines while applying decay across interaction turns. Family Dense Reward (FDR) calibrates semantic progress across task families; Success-Prioritized Reward (SPR) separates strict success from bounded partial credit. A Warm-up Critic Model, recursive critic handoff, and stage-specific learning rates coordinate the two refinement phases. The five-stage sequence from Qwen3.8-27B base model through RSR-1, RPR-1, RSR-2, and RPR-2 yields Terminal-RSI-27B: 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard. On LHTB, process reward rises from 0.21 to 0.37 and strict completion from 0/46 to 5/46; SWR100 success doubles from 3.00% to 6.00%. By recursively coupling verified trajectory rewriting with critic-inheriting policy refinement, each retained stage consolidates discovered behaviors and progressively advances the agent from partial local repairs to strict terminal verification.
Figures & tables
Figure 2: The recursive loop carries critic weights through supervised transitions. Blue blocks distinguish actor-only RSR from joint actor–critic RPR; the bypass preserves the critic, and feedback returns the retained policy to the next cycle. The initial actor is RSR-1; Warm-up Critic Model denotes the prepared value network.
Figure 3: Rewriting and policy refinement update different parameters. The left branch discovers, rewrites, and verifies actor demonstrations; Recursive Policy-Refine (RPR) combines inherited values, reward calibration, and Turn-Boundary Skip-Observation GAE (TB-GAE). Phase-local critic calibration comprises four critic-only updates before joint training, not before each PPO minibatch.
| Setting | RPR-1 | RPR-2 |
|---|---|---|
| Initial actor | RSR-1 | RSR-2 |
| Initial critic | Warm-up Critic Model | Retained RPR-1 critic |
| Reward | FDR | SPR |
| Actor learning rate | ||
| Critic learning rate | ||
| Critic-only calibration | 4 updates | 4 updates |
Table 1: Refinement adapts policy updates while preserving the critic learning scale. Shared settings appear below; the ordinary RPR-2 branch uses FDR rather than SPR.
| Checkpoint | Terminal-Bench 4.0 | Terminal-Bench 2.1 | Terminal-Bench 3.0 | Terminal-Bench Hard | LHTB | SWR100 |
|---|---|---|---|---|---|---|
| Base | 0.00 | 51.70 | 0.00 | 31.00 | 0.21 | 3.00 |
| PPO | 0.00 0.00 | 48.31 3.39 | 1.43 1.43 | 39.00 8.00 | – | – |
| RSR-1 | 6.10 6.10 | 70.10 18.40 | 6.80 6.80 | 54.00 23.00 | 0.29 0.08 | 5.00 2.00 |
| RPR-1 | 16.67 10.57 | 77.53 7.43 | 14.29 7.49 | 69.50 15.50 | 0.30 0.01 | 6.00 1.00 |
| RSR-2 | 17.46 0.79 | 86.52 8.99 | 15.00 0.71 | 76.00 6.50 | 0.31 0.01 | 6.00 0.00 |
| Terminal-RSI-27B | 19.84 2.38 | 86.52 0.00 | 16.95 1.95 | 79.00 3.00 | 0.37 0.06 | 6.00 0.00 |
Table 2: The retained sequence improves all benchmark scores over Base. Retained-stage rows report Mean@3 success (%); the separate PPO row is a historical aggregate under Appendix E ’s protocol. Best scores, including ties, are bold. Arrows give percentage-point changes from retained predecessors, or the primary Base entry for PPO. Terminal-Bench 4.0, Terminal-Bench 2.1, Terminal-Bench 3.0, Terminal-Bench Hard and SWR100 use percentage of accuracy as performance, LHTB use dense process reward as performance for evaluation.
Figure 4: Performance comparison between Terminal-RSI-27B and state-of-the-art frontier models on terminal agent task benchmarks. Blue bars highlight the success rate achieved by our proposed Terminal-RSI-27B model, while gray bars present the corresponding scores of all included reference frontier models. The left and right panels separately demonstrate the comparative results on Terminal-Bench 4.0 and Terminal-Bench 2.1
Figure 5: Mean Success Progression. Evaluation across Terminal-Bench 4.0, 3.0, 2.1, and Hard yields an equal-weight success mean rising from 20.68% to 50.58% . The vertical axis spans 10.00%–60.00%.
Figure 6: Retained gains coexist with benchmark-dependent regressions. Diamonds denote RSR, stars retained RPR, and gray branches discarded alternatives; the larger final star marks Terminal-RSI-27B at RPR-2 step 18. Values are success (%): retained stages use Mean@3, while the separate PPO control retains its historical protocol (Appendix E ). Lower panels begin at 40.00% and 25.00%. Stage spacing is ordinal, not equal compute.
Figure 7: Dense and binary rewards rise over the displayed RPR-1 window. Interaction length varies and train–inference mismatch remains nonzero. Pale traces show raw observations; dark traces show causal means of up to five observations. The horizontal coordinate is calibrated for display, not a common training clock.
Figure 8: RPR-2 reward rises under both settings, while interaction length diverges. Blue solid curves denote SPR; dark-gray dashed curves denote the FDR branch. Pale traces show raw observations and emphasized curves causal means of up to five observations. Reward and turn panels retain distinct native clocks; different shaping scales are not comparable strict-success rates.
Figure 9: RPR-2 critic EV remains positive throughout the recorded window. Warm-Up begins negative and RPR-1 briefly crosses zero. All 16 RPR-2 observations remain positive without smoothing or imputation.
Figure 10: Structural strata reveal uneven capability gains. Darker curves denote harder fixed groups; bold labels mark RPR-2. Values use one linear scale. Easy reaches 72.88%, while Extreme dips and recovers. Scores are task-weighted; the Terminal-Bench 2.1 RSR-1 analysis uses an alternative rewrite checkpoint.
Figure 11: Reward allocation varies across families and phases. Panels show FDR and SPR profiles for the same 46 families. Solid curves encode mean reward; dashed curves show pooled verifier counts on a separate scale. Polygon area and smoothness are not variance measures.
Figure 12: Final-stage replays satisfy the tested execution contracts. Blue stairs show mean checker completion, gray dots scored repeats, and dashed lines Base. The blue star marks RPR-2 (Terminal-RSI-27B), evaluated separately from RSR-2; both attain 100.00% in two repeats per task. Axes begin at 70.00% and 20.00%, respectively. These checker-level means are not benchmark Mean@3 success.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value or behavior |
|---|---|
| Evaluation repeats | Three independent runs; arithmetic mean (Mean@3) |
| Sampling | Temperature 0.60; top- |
| Generation / input budget | 8,192 output tokens per turn / 96,000 input tokens per call |
| Actions and explicit thinking | Structured JSON actions; explicit thinking disabled |
| Output and history | Raw model outputs; linear interaction history |
| Context summarization | Proactive; triggered at 32,000 accumulated tokens |
Table 3: Final benchmark protocol, distinct from training monitoring. Context and interaction budgets are separate limits; benchmark-specific termination and time limits still apply.
Figure 13: RSR-first initialization remains stronger at the observed common local steps. The blue starting diamond marks RSR-1, filled blue circles mark intermediate RPR observations, and the blue star marks retained step 20; gray squares trace PPO from its own measured initialization. Every observed point has a numerical label. Lines join measured success rates without smoothing or imputation. Local steps align update counts rather than total compute because RSR-first refinement also incurs discovery and supervised-rewriting costs. Historical repeat protocols require reconciliation before treating the comparison as uniformly matched.
| Component | Comparison | Primary diagnostic |
|---|---|---|
| Credit assignment | Generated-token GAE; skip-only; TB-GAE, with matched rewards and terminal/chunk boundaries | Horizon-stratified advantage or gradient variance; strict success; boundary correctness |
| Reward evolution | FDR; saturation-only; success-offset-only; full SPR, with family calibration varied separately | Strict success; early failure; reward calibration |
| Critic handoff | Inherit/reset crossed with warmup under matched calibration budgets | Value calibration; update norms; strict success |
| RSR supervision | Cleaned native replay; uncleaned replay; fixed teacher, with matched supervision budgets | Replay validity; leakage; solution coverage |
| Recursive schedule | Continued RPR; rollback; alternating RSR and RPR, including discovery, SFT, and RL costs | Worst-suite retention under predefined validation rules and matched total compute |
Table 5: Component-specific controls are needed for mechanistic attribution. These are prospective experiments rather than measured effect estimates. Comparisons should match task streams, initialization, effective budgets, and repeated training seeds, while recording terminal reasons and strict success.
Figure 14: Five-stage auxiliary evaluation shows continued LHTB gains and an SWR100 plateau. Blue-shaded bars follow Base, RSR-1, RPR-1 at step 20, RSR-2, and RPR-2 at step 18. Dashed references mark the same final Terminal-RSI-27B checkpoint rather than per-suite maxima. LHTB bar heights encode process rewards of 0.21, 0.29, 0.30, 0.31, and 0.37, with separate strict-completion entries of 0/46, 0/46, 2/46, 4/46, and 5/46. SWR100 success is 3.00%, 5.00%, 6.00%, 6.00%, and 6.00%. All checkpoints share the 27B backbone; values come from the stage ledger without imputation. Appendix E distinguishes compact completion entries from run-level aggregation.
Figure 15: Separately recorded final-model executions pass the statistical checks. Five rows progress from Base to RPR-2 (Terminal-RSI-27B); percentages report mean checker completion over scored repeats. Each row links three selected action, implementation, and verification excerpts. RSR-2’s numerical probes differ from RPR-2’s six-module repair; both pass all 11 assertions in two trials. Green denotes passing evidence, not an error-free history; yellow denotes residual or mixed failures. Rows are ordered stages, not equal-duration executions.
Figure 16: Separately recorded final-stage trials satisfy the tested output contract. Five rows show separate stage evaluations; percentages average the wrapper and fuzz-equivalence gates. Base and RSR-1 retain a format mismatch; RPR-1 has mixed outcomes. RSR-2 and RPR-2 each pass both gates in two trials. The first final-model trial builds C/Rust components but delegates expression evaluation to Python. Green denotes tested output equivalence, not a wholly native AST path or memory safety; missing scores are not imputed.
More in this topic
Terminal Agents
Terminal agents use command-line interfaces and filesystems to carry out multi-step tasks in software and other digital environments.
See all 31 papers