Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.
Figures & tables
Figure 2: AutoEnvScaling automates the data flywheel. The proposer works inside a sandbox with a coding harness, solver rollouts, a workspace, and web access to construct Harbor training environments. It can launch child sandboxes to build and test tasks. The solver trains on accepted environments with verifier rewards; its new rollouts guide the next round of environment generation.
Proposer Source
Proposer Optimization
Method
Strategy
Data Sources
External
Self
Harness
Weights
Endless Term. [ 8 ]
Prompt + feedback
Human-defined
✓
×
×
×
TermiGen [ 67 ]
Prompt + feedback
Human-defined
✓
×
×
×
Table 1: Comparison of environment generation methods. ‘External’ uses a separate proposer model; ‘Self’ uses the trained solver as proposer. Proposer updates distinguish harness optimization from weight training. Entries describe the main settings, excluding ablations.
Figure 3: Environment design and solving as terminal tasks. Left : one shared policy acts as both proposer and solver. As proposer it designs environments and earns rP (Equation 2 ), which depends on whether each environment passes validation and calibration. As solver it attempts tasks and earns rS (Equation 1 ), the fraction of each task’s tests it passes. Right : an example of the proposer turning the failed PyTorch seed from Section 4.4 into a new environment. The trace follows its read, build, write, and test actions, and the shading shows where its runs spend their turns.
Figure 4: AutoEnvScaling closes the gap between weaker and stronger models. Valid-environment rate across eight proposers. Left: Prompt < Prompt + Env Feedback < AutoEnvScaling for every model. Right: Within GPT-5.6, where Sol is the strongest model and Luna the weakest, prompt-based rates fall as the model weakens, while ours stay between 84% and 92% .
Table 2: Valid-environment rate and cost across three generation methods. Costs are USD per valid environment. Appendix F examines reference execution during generation.
Figure 5: Environments generated by AutoEnvScaling are richer, longer-horizon, and harder to solve. Files per accepted environment, solver turns, and solver pass rate for each method.
Proposer
Model / Settings
GPT-5.6-sol
Claude Opus 5
Qwen3.5-9B
19.1 ± 2.3
+ Tmax
27.2 ± 1.4
29.2 ± 2.1
+ AutoEnvScaling
33.5 ± 1.9
36.6 ± 1.7
Qwen3.5-35B-A3B
26.7 ± 2.2
+ Tmax
34.6 ± 1.9
35.5 ± 1.6
Table 3: AutoEnvScaling versus baseline. Held-out Terminal-Bench 2.1 under matched training steps. ± gives the confidence interval.
Figure 6: AutoEnvScaling enables recursive self-improvement. Qwen3.6-35B-A3B performance on Terminal-Bench 2.1 (left) and 4.0 (right) during training. We compare AutoEnvScaling , AutoEnvScaling without proposer training, and static Tmax training. The run without proposer training ends earlier because training collapsed.
Figure 7: AutoEnvScaling retains more rollout groups and collects them faster. Qwen3.6-35B-A3B trained with AutoEnvScaling or the static Tmax baseline. Left: useful-group rate: groups retained for training as a percentage of all groups collected at each step; both methods retain 16 groups. Right: rollout-collection time per step. Faint lines show individual steps; solid lines show 3 -step (left) and 5 -step (right) trailing means.
Figure 8: Environment design from the same base-agent failure. The base agent timed out rebuilding a PyTorch model from its weights. Each line follows one proposer through Read, Build, Write, and Test actions. Numbered examples show the corresponding generation and validation actions. Additional cases and the annotation protocol are in Appendix I .
Text-to-SQL
Software engineering
Model
full_info
ask_human
Ask-F1
full_info
ask_human
Ask-F1
Qwen3.5-9B
13.6
0.3
0.1
21.1
0.0
0.0
Qwen3.6-35B-A3B
34.2
0.3
0.7
35.0
0.0
0.0
GPT-5.6-sol
51.2
0.7
1.9
42.7
2.7
0.0
Claude Opus 5
36.5
2.1
8.3
52.5
7.5
14.4
Table 4: HiL-Bench results for base and frontier models. full_info gives the complete instruction; ask_human gives an incomplete instruction and the ask_human tool. Ask-F1 is measured in the ask_human setting. All values are percentages.
Setting
Instances
Task success
Ask-F1
Ask rate
Qwen3.6-35B-A3B
–
0.0
0.0
1.8
+ Cold start
75
1.2
10.1
71.4
+ Cold start + official training set RL
100
6.8
29.6
84.2
+ Cold start + AutoEnvScaling
500
33.9
55.3
89.8
Table 5: Training Qwen3.6-35B-A3B to ask for help on HiL-Bench software engineering tasks. Instances are cold-start trajectories or RL training tasks. Task success and Ask-F1 are measured in the ask_human setting on held-out tasks. All values are percentages.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Blocker
Type
Withheld answer
amount_agreement_tolerance
Missing parameters
Amounts agree when the difference between the ledger amount and the billing total, rounded to two decimals, is at most 0.02 in either direction, inclusive.
invoice_status_scope
Business information
Only ISSUED and PAID invoices are in scope, compared case-insensitively. VOID and DRAFT invoices are dropped entirely, and ledger rows that point at them are not counted as ledger-only.
duplicate_invoice_row_precedence
Ambiguous requirements
For a repeated invoice_id , keep the last occurrence in file order and ignore earlier rows; do not sum them.
discrepancy_report_ordering
Question
Sort discrepancies.csv by the absolute difference, largest first, breaking ties by invoice_id ascending.
total_variance_definition
Schema
total_variance is the sum of absolute per-row differences over the rows in discrepancies.csv only, rounded to two decimals.
Appendix
Table 6: Details withheld from the task in Figure 9 . Each entry of the blocker registry names a gap in the instruction and the answer that ask_human returns. Types follow the HiL-Bench taxonomy.
Domain
Generated example
Environment
Verification
Terminal
Reconcile warehouse revenue
Files, manifests, and CLI tools
Parsed outputs, deduplication, corrections, and exact monetary totals
Software engineering
Repair telemetry time windows
Repository, issue, and terminal
Regression tests and private boundary cases
Tool calling
Apply cold-chain disposition rules
Seeded service and typed MCP calls
Service state, decision policy, and unaffected records
Browser use
Resolve an incident using a runbook
Local web application; clicks and text entry
Saved case fields, draft-to-resolve transition, and unaffected cases
Computer use
Complete a stock-audit workbook
LibreOffice desktop; mouse and keyboard
Saved formulas, cell values, formatting, and preserved inputs
GPU kernels
Fuse ragged residual RMSNorm
PyTorch/Triton on one T4
Numerical correctness and timed execution against an eager baseline
Appendix
Table 7: Generated environments across seven domains. The examples illustrate the task-specific state, interaction interface, and verification criteria.
Domain
Proposer
Acceptance (%)
Rounds
Terminal
GPT-6-astra
100.0
1.77
GPT-5.6-sol
100.0
1.30
Claude Opus 5
93.8
2.25
Software engineering
GPT-6-astra
100.0
1.41
GPT-5.6-sol
100.0
1.33
Claude Opus 5
91.2
2.35
Appendix
Table 8: Technical admission across seven domains. Acceptance is measured over completed, evaluable generation runs. Rounds include the initial attempt and subsequent revisions, averaged over accepted tasks (range 1–3). Bold marks the highest acceptance and lowest mean rounds within each domain, including ties.
Figure 10: Incident page before completing the resolution record. The symptom describes repeated brownout cycling after transfer to backup power. The solver consults the runbook library on a separate page before completing the resolution record.
Figure 11: Runbook lookup. The library maps repeated brownout cycling to RB-17, while the other entries address thermal-sensor drift and uplink failover. The reference then returns to QZ-418 to complete its resolution record.
Figure 12: Resolved case after saving the draft. The final page shows Mira Chen, RB-17, the resolution note, the completed review, and resolved status.
Figure 13: A GUI-only document transformation. The final brief contains bold “Approved for Q4 pilot,” italic “Morgan Lee,” and a centered “Decision Summary” heading. The panels show the document before and after editing. The heading remains selected in the final frame.
Figure 14: A GUI-only spreadsheet workflow. The reference enters =D2-E2 and corresponding formulas down the variance column, derives OK / CHECK status, and computes five discrepant lines and a total shortfall of 12.
Figure 15: Two linked professional deliverables. The Stress scenario falls to 18,796.86againsta30,000 cash floor and reaches the 70,000facilitylimit.Thememorecommendsdeferralbutretainsthe65,000 week-14 obligation.
Figure 16: A calculation-to-recommendation workflow. UB-A fails the deflection limit; UB-B and UB-C pass. The reference selects UB-B at GBP 108/m, with stress 76.52 MPa and deflection 5.64 mm. The memo states the limited scope of the check. Narrow columns in the generated workbook truncate some input labels.
Task
Shape / dtype
Eager (ms)
Kernel (ms)
Ratio
Ragged RMSNorm
4096×1025 / f32
1.6622
0.3274
0.1970
2048×2049 / f16
1.8906
0.2065
0.1092
8192×257 / f32
0.7781
0.1558
0.2002
Gated RMSNorm
512×256 / f16
0.1492
0.0503
0.3369
1024×512 / f16
0.1901
0.0535
0.2817
2048×1024 / f16
0.6586
0.1094
0.1661
Appendix
Table 9: Reference-kernel execution times on an NVIDIA T4. Values are mean milliseconds over 20 samples per implementation and shape after warmup. The ratio is kernel time divided by the task’s eager PyTorch baseline time.
Figure 18: Proposer workspace. A schematic layout of generation skills, tools, memory, and generated Harbor tasks. Colors distinguish editable components, retained experience, outputs, and fixed inputs. File names are illustrative.
Proposer
Solver
CPUs
16
4
Memory
32 GiB
8 GiB
Time budget per episode
2 hours
–
Turn limit
none
–
Coding agent
Claude Code, Codex, or Pi
Pi
Appendix
Table 10: Sandbox resources.
Figure 19: Image construction and runtime configuration. Simplified Dockerfile and TOML listings. Commands, paths, and configuration keys are shortened for presentation.
Proposer
Self-check disabled
Self-check enabled
Claude Opus 5
52.0
94.0
GPT-5.6-sol
19.5
84.8
GPT-5.6-terra
45.2
93.5
GPT-5.6-luna
40.1
85.3
DeepSeek-V4-Flash
19.4
73.0
Appendix
Table 11: Executing the reference solution improves valid-environment rate. Valid-environment rate (%) with the generation self-check disabled or enabled.
Setting
Value
Learning rate
10−6
Solver rollout batch
16 environments × 16 trajectories
Proposer rollout group
128 proposer trajectories
Proposer generation interval ( k )
16 solver updates
Active environment pool
256 tasks
Pool review interval
16 solver updates
Appendix
Table 12: Settings for shared-model recursive training.
Figure 20: Proposer training keeps the shared policy’s environments valid. Valid-environment rate of Qwen3.6-35B-A3B acting as proposer during shared-policy training. The open diamond marks the shared Cold-Start checkpoint. The run without proposer training ends at step 160 because training collapsed.
Figure 30
Model
Environment acceptance
TB2.1 avg@5
TB2.1 pass@5
Base
33.3
40.0
59.6
Cold-Start
76.7
41.1
66.3
Appendix
Table 13: Cold-Start improves environment generation without a detected change in solving performance. Environment acceptance uses the same 60 assignments for both models. Solving uses five attempts on each of 89 Terminal-Bench 2.1 tasks. All values are percentages.
Figure 23: fastText seed case. The solver trained a fastText classifier on 650 k Yelp reviews, measured 0.60 against a target of 0.62 , spent 19 of 35 commands chasing a non-existent option, and timed out without writing the model file. No Docker daemon was available to GPT-5.6-sol, so it ran static checks only. Panels follow Figure 8 .
Figure 24: Primer-design seed case. The hardest seed task: design Q5 mutagenesis primers under melting-temperature constraints, graded against primer3’s oligotm . The solver switched to Perl (no Python in the image), located the insertion, computed annealing temperatures, and timed out. GPT-5.6-sol drops primer design for a generic FASTA pipeline; Claude Opus 5 and Cold-Start Qwen3.6-35B-A3B both keep the biology. Panels follow Figure 8 .
Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interfaces (GUIs). Among these, the terminal command line provides a text-based, general-purpose interface, covering tasks from system operations to data science and machine learning. However, scaling terminal-agent training remains challenging, as it requires diverse and coherent task instructions, executable environments, and reliable verification, while lacking naturally grounded supervision data. In this work, we propose SETA, a scalable framework for generating verifiable terminal environments for reinforcement learning (RL). The framework consists of two pipelines sharing a unified verification mechanism: SETA-Synth converts diverse sources into standardized RL environments, and SETA-Evol further expands from existing environments with adaptive control of difficulty and diversity. Together, we construct and release SETA-Env, the largest open-source verifiable terminal RL dataset to date, containing over 4,500 environments. We evaluate our dataset by training Qwen3-8B with GRPO on SETA-Env, achieving 12% pass rate on Terminal-Bench 2.0, the best reported result for an RL-trained model at the 8B scale. We further observe gains on DeepSeek-V4-Flash under the same terminal agent harness, with pass@1 on Terminal-Bench 2.0 improving from 40% to 43% and pass@5 improving from 54% to 58%. These results demonstrate that SETA- Env provides high-quality training environments for terminal agents and serves as a valuable resource for advancing research on terminal-based agent learning.
Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru +19
Imperial College London · University College London · SambaNova +5
Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.
Terminal agents extend Large Language Models with the ability to execute tasks directly in command-line environments, but their progress is bottlenecked by the scarcity of high-quality training data. Existing approaches bootstrap from partial sources such as human-defined seeds or GitHub repositories to instantiate one component and then complete the rest, producing tasks confined to narrow seed distributions, environments misaligned with task semantics, and inefficient trajectories from unguided exploration. To address these limitations, we introduce Terminal-World, a fully automated pipeline that uses agent skills as the central synthesis primitive, which jointly encode what to accomplish, when to apply (preconditions and environment state), and how to execute, enabling task instructions, environments, and teacher trajectories to be co-derived. To further broaden the synthesis space, Terminal-World composes skills into skill teams and skill graphs for multi-role and cross-domain task synthesis. Using this pipeline, we construct 5,723 training environments and train Terminal-World-8B/14B/32B, evaluated across 6 benchmarks where the Terminal-World series consistently outperforms terminal-agent baselines. Notably, using the same teacher model and only 1.2% of the training data, Terminal-World-32B surpasses Nemotron-Terminal-32B on Terminal-Bench 2.0 by +4.5 Pass@1 (31.5) and achieves 43.8 Pass@3.
Zihao Cheng, Hongru Wang, Zeming Liu +6
School of Computer Science and Engineering, Beihang University, Beijing, China · Independent Researcher · 3Beijing Institute of Technology +2