AutoEnvScaling: Automating the Data Flywheel with Terminal Agents

Simon Yu; Baolin Peng; Bo Liu; Zhengyan Shi; Zhaoyang Wang; Hao Cheng; Qianhui Wu; Wenlin Yao

cs.SEOct 9, 2026

AutoEnvScaling: Automating the Data Flywheel with Terminal Agents

Authors: Simon Yu, Baolin Peng, Bo Liu, Zhengyan Shi, Zhaoyang Wang, Hao Cheng, Qianhui Wu, Wenlin Yao,

Organizations: Northeastern University · Microsoft · Stanford University · University of Washington ·

Abstract

Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.

Figures & tables

Figure 2: AutoEnvScaling automates the data flywheel. The proposer works inside a sandbox with a coding harness, solver rollouts, a workspace, and web access to construct Harbor training environments. It can launch child sandboxes to build and test tasks. The solver trains on accepted environments with verifier rewards; its new rollouts guide the next round of environment generation.

			Proposer Source		Proposer Optimization



Method	Strategy	Data Sources	External	Self	Harness	Weights
Endless Term. [ 8 ]	Prompt + feedback	Human-defined	$\boldsymbol{\checkmark}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$
TermiGen [ 67 ]	Prompt + feedback	Human-defined	$\boldsymbol{\checkmark}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$

Table 1: Comparison of environment generation methods. ‘External’ uses a separate proposer model; ‘Self’ uses the trained solver as proposer. Proposer updates distinguish harness optimization from weight training. Entries describe the main settings, excluding ablations.

Figure 3: Environment design and solving as terminal tasks. Left : one shared policy acts as both proposer and solver. As proposer it designs environments and earns

r_{P}

(Equation 2 ), which depends on whether each environment passes validation and calibration. As solver it attempts tasks and earns

r_{S}

(Equation 1 ), the fraction of each task’s tests it passes. Right : an example of the proposer turning the failed PyTorch seed from Section 4.4 into a new environment. The trace follows its read, build, write, and test actions, and the shading shows where its runs spend their turns.

Figure 4: AutoEnvScaling closes the gap between weaker and stronger models. Valid-environment rate across eight proposers. Left: Prompt

<

Prompt + Env Feedback

<

AutoEnvScaling for every model. Right: Within GPT-5.6, where Sol is the strongest model and Luna the weakest, prompt-based rates fall as the model weakens, while ours stay between

84\%

and

92\%

	Valid-env. rate (%)			Cost per valid env. (USD)
Proposer	Prompt	Prompt + Env Feedback	AutoEnvScaling	Prompt	Prompt + Env Feedback	AutoEnvScaling
Claude Opus 5	74.6 [70.0, 78.7]	89.5 [86.0, 92.2]	94.0 [91.2, 95.9]	0.72	0.70	1.18 ${}_{{\color[rgb]{0.6914,0.1406,0.0938}1.6\times\,\uparrow}}$
GPT-6-astra	67.0 [61.2, 72.4]	83.0 [78.0, 87.1]	94.0 [88.9, 96.8]	1.79	1.76	2.75 ${}_{{\color[rgb]{0.6914,0.1406,0.0938}1.5\times\,\uparrow}}$
GPT-5.6-sol	41.1 [36.4, 46.0]	65.5 [60.7, 70.1]	86.3 [82.5, 89.3]	0.90	0.80	0.59 ${}_{{\color[rgb]{0.1055,0.4961,0.293}0.7\times\,\downarrow}}$
GPT-5.6-terra	40.0 [35.3, 44.9]	53.2 [48.2, 58.1]	92.5 [88.0, 95.4]	0.13	0.15	0.09 ${}_{{\color[rgb]{0.1055,0.4961,0.293}0.7\times\,\downarrow}}$
Gemini-3.8-Flash	26.2 [22.1, 30.8]	59.6 [54.5, 64.4]	64.8 [57.2, 71.7]	0.28	0.20	0.16 ${}_{{\color[rgb]{0.1055,0.4961,0.293}0.6\times\,\downarrow}}$

Table 2: Valid-environment rate and cost across three generation methods. Costs are USD per valid environment. Appendix F examines reference execution during generation.

Figure 5: Environments generated by AutoEnvScaling are richer, longer-horizon, and harder to solve. Files per accepted environment, solver turns, and solver pass rate for each method.

	Proposer
Model / Settings	GPT-5.6-sol	Claude Opus 5
Qwen3.5-9B	19.1 $\pm$ 2.3
+ Tmax	27.2 $\pm$ 1.4	29.2 $\pm$ 2.1
+ AutoEnvScaling	33.5 $\pm$ 1.9	36.6 $\pm$ 1.7
Qwen3.5-35B-A3B	26.7 $\pm$ 2.2
+ Tmax	34.6 $\pm$ 1.9	35.5 $\pm$ 1.6

Table 3: AutoEnvScaling versus baseline. Held-out Terminal-Bench 2.1 under matched training steps.

\pm

gives the confidence interval.

Figure 6: AutoEnvScaling enables recursive self-improvement. Qwen3.6-35B-A3B performance on Terminal-Bench 2.1 (left) and 4.0 (right) during training. We compare AutoEnvScaling , AutoEnvScaling without proposer training, and static Tmax training. The run without proposer training ends earlier because training collapsed.

Figure 7: AutoEnvScaling retains more rollout groups and collects them faster. Qwen3.6-35B-A3B trained with AutoEnvScaling or the static Tmax baseline. Left: useful-group rate: groups retained for training as a percentage of all groups collected at each step; both methods retain

16

groups. Right: rollout-collection time per step. Faint lines show individual steps; solid lines show

3

-step (left) and

5

-step (right) trailing means.

Figure 8: Environment design from the same base-agent failure. The base agent timed out rebuilding a PyTorch model from its weights. Each line follows one proposer through Read, Build, Write, and Test actions. Numbered examples show the corresponding generation and validation actions. Additional cases and the annotation protocol are in Appendix I .

	Text-to-SQL			Software engineering
Model	full_info	ask_human	Ask-F1	full_info	ask_human	Ask-F1
Qwen3.5-9B	13.6	0.3	0.1	21.1	0.0	0.0
Qwen3.6-35B-A3B	34.2	0.3	0.7	35.0	0.0	0.0
GPT-5.6-sol	51.2	0.7	1.9	42.7	2.7	0.0
Claude Opus 5	36.5	2.1	8.3	52.5	7.5	14.4

Table 4: HiL-Bench results for base and frontier models. full_info gives the complete instruction; ask_human gives an incomplete instruction and the ask_human tool. Ask-F1 is measured in the ask_human setting. All values are percentages.

Setting	Instances	Task success	Ask-F1	Ask rate
Qwen3.6-35B-A3B	–	0.0	0.0	1.8
+ Cold start	75	1.2	10.1	71.4
+ Cold start + official training set RL	100	6.8	29.6	84.2
+ Cold start + AutoEnvScaling	500	33.9	55.3	89.8

Table 5: Training Qwen3.6-35B-A3B to ask for help on HiL-Bench software engineering tasks. Instances are cold-start trajectories or RL training tasks. Task success and Ask-F1 are measured in the ask_human setting on held-out tasks. All values are percentages.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Blocker	Type	Withheld answer
amount_agreement_tolerance	Missing parameters	Amounts agree when the difference between the ledger amount and the billing total, rounded to two decimals, is at most $0.02$ in either direction, inclusive.
invoice_status_scope	Business information	Only ISSUED and PAID invoices are in scope, compared case-insensitively. VOID and DRAFT invoices are dropped entirely, and ledger rows that point at them are not counted as ledger-only.
duplicate_invoice_row_precedence	Ambiguous requirements	For a repeated invoice_id , keep the last occurrence in file order and ignore earlier rows; do not sum them.
discrepancy_report_ordering	Question	Sort discrepancies.csv by the absolute difference, largest first, breaking ties by invoice_id ascending.
total_variance_definition	Schema	total_variance is the sum of absolute per-row differences over the rows in discrepancies.csv only, rounded to two decimals.

Appendix

Table 6: Details withheld from the task in Figure 9 . Each entry of the blocker registry names a gap in the instruction and the answer that ask_human returns. Types follow the HiL-Bench taxonomy.

Domain	Generated example	Environment	Verification
Terminal	Reconcile warehouse revenue	Files, manifests, and CLI tools	Parsed outputs, deduplication, corrections, and exact monetary totals
Software engineering	Repair telemetry time windows	Repository, issue, and terminal	Regression tests and private boundary cases
Tool calling	Apply cold-chain disposition rules	Seeded service and typed MCP calls	Service state, decision policy, and unaffected records
Browser use	Resolve an incident using a runbook	Local web application; clicks and text entry	Saved case fields, draft-to-resolve transition, and unaffected cases
Computer use	Complete a stock-audit workbook	LibreOffice desktop; mouse and keyboard	Saved formulas, cell values, formatting, and preserved inputs
GPU kernels	Fuse ragged residual RMSNorm	PyTorch/Triton on one T4	Numerical correctness and timed execution against an eager baseline

Appendix

Table 7: Generated environments across seven domains. The examples illustrate the task-specific state, interaction interface, and verification criteria.

Domain	Proposer	Acceptance (%)	Rounds
Terminal	GPT-6-astra	100.0	1.77
	GPT-5.6-sol	100.0	1.30
	Claude Opus 5	93.8	2.25
Software engineering	GPT-6-astra	100.0	1.41
	GPT-5.6-sol	100.0	1.33
	Claude Opus 5	91.2	2.35

Appendix

Table 8: Technical admission across seven domains. Acceptance is measured over completed, evaluable generation runs. Rounds include the initial attempt and subsequent revisions, averaged over accepted tasks (range 1–3). Bold marks the highest acceptance and lowest mean rounds within each domain, including ties.

Figure 10: Incident page before completing the resolution record. The symptom describes repeated brownout cycling after transfer to backup power. The solver consults the runbook library on a separate page before completing the resolution record.

Figure 11: Runbook lookup. The library maps repeated brownout cycling to RB-17, while the other entries address thermal-sensor drift and uplink failover. The reference then returns to QZ-418 to complete its resolution record.

Figure 12: Resolved case after saving the draft. The final page shows Mira Chen, RB-17, the resolution note, the completed review, and resolved status.

Panel of Figure 13 — Figure 13: A GUI-only document transformation. The final brief contains bold “Approved for Q4 pilot,” italic “Morgan Lee,” and a centered “Decision Summary” heading. The panels show the document before and after editing. The heading remains selected in the final frame.

Panel of Figure 14 — Figure 14: A GUI-only spreadsheet workflow. The reference enters =D2-E2 and corresponding formulas down the variance column, derives OK / CHECK status, and computes five discrepant lines and a total shortfall of 12.

Figure 15: Two linked professional deliverables. The Stress scenario falls to

18,796.86 against a

30,000 cash floor and reaches the

70,000 facility limit. The memo recommends deferral but retains the

65,000 week-14 obligation.

Figure 16: A calculation-to-recommendation workflow. UB-A fails the deflection limit; UB-B and UB-C pass. The reference selects UB-B at GBP 108/m, with stress 76.52 MPa and deflection 5.64 mm. The memo states the limited scope of the check. Narrow columns in the generated workbook truncate some input labels.

Task	Shape / dtype	Eager (ms)	Kernel (ms)	Ratio
Ragged RMSNorm	$4096\!\times\!1025$ / f32	1.6622	0.3274	0.1970
	$2048\!\times\!2049$ / f16	1.8906	0.2065	0.1092
	$8192\!\times\!257$ / f32	0.7781	0.1558	0.2002
Gated RMSNorm	$512\!\times\!256$ / f16	0.1492	0.0503	0.3369
	$1024\!\times\!512$ / f16	0.1901	0.0535	0.2817
	$2048\!\times\!1024$ / f16	0.6586	0.1094	0.1661

Appendix

Table 9: Reference-kernel execution times on an NVIDIA T4. Values are mean milliseconds over 20 samples per implementation and shape after warmup. The ratio is kernel time divided by the task’s eager PyTorch baseline time.

Figure 18: Proposer workspace. A schematic layout of generation skills, tools, memory, and generated Harbor tasks. Colors distinguish editable components, retained experience, outputs, and fixed inputs. File names are illustrative.

	Proposer	Solver
CPUs	16	4
Memory	32 GiB	8 GiB
Time budget per episode	2 hours	–
Turn limit	none	–
Coding agent	Claude Code, Codex, or Pi	Pi

Appendix

Table 10: Sandbox resources.

Figure 19: Image construction and runtime configuration. Simplified Dockerfile and TOML listings. Commands, paths, and configuration keys are shortened for presentation.

Proposer	Self-check disabled	Self-check enabled
Claude Opus 5	52.0	94.0
GPT-5.6-sol	19.5	84.8
GPT-5.6-terra	45.2	93.5
GPT-5.6-luna	40.1	85.3
DeepSeek-V4-Flash	19.4	73.0

Appendix

Table 11: Executing the reference solution improves valid-environment rate. Valid-environment rate (%) with the generation self-check disabled or enabled.

Setting	Value
Learning rate	$10^{-6}$
Solver rollout batch	16 environments $\times$ 16 trajectories
Proposer rollout group	128 proposer trajectories
Proposer generation interval ( $k$ )	16 solver updates
Active environment pool	256 tasks
Pool review interval	16 solver updates

Appendix

Table 12: Settings for shared-model recursive training.

Figure 20: Proposer training keeps the shared policy’s environments valid. Valid-environment rate of Qwen3.6-35B-A3B acting as proposer during shared-policy training. The open diamond marks the shared Cold-Start checkpoint. The run without proposer training ends at step 160 because training collapsed.

Model	Environment acceptance	TB2.1 avg@5	TB2.1 pass@5
Base	33.3	40.0	59.6
Cold-Start	76.7	41.1	66.3

Appendix

Table 13: Cold-Start improves environment generation without a detected change in solving performance. Environment acceptance uses the same 60 assignments for both models. Solving uses five attempts on each of 89 Terminal-Bench 2.1 tasks. All values are percentages.

Figure 23: fastText seed case. The solver trained a fastText classifier on $650$ k Yelp reviews, measured $0.60$ against a target of $0.62$ , spent $19$ of $35$ commands chasing a non-existent option, and timed out without writing the model file. No Docker daemon was available to GPT-5.6-sol, so it ran static checks only. Panels follow Figure 8 .

Figure 24: Primer-design seed case. The hardest seed task: design Q5 mutagenesis primers under melting-temperature constraints, graded against primer3’s oligotm . The solver switched to Perl (no Python in the image), located the insertion, computed annealing temperatures, and timed out. GPT-5.6-sol drops primer design for a generic FASTA pipeline; Claude Opus 5 and Cold-Start Qwen3.6-35B-A3B both keep the biology. Panels follow Figure 8 .

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 586 papers

SETA: Scaling Environments for Terminal Agents
Jul 12, 2026Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru +19Reinforcement Learning Terminal Agents
Imperial College London · University College London · SambaNova +5
Environment Evolution for Terminal Agents
Sep 3, 2026Zhiyuan Fan, Tinghao Yu, Yuanjun Cai +9Terminal Agents Curriculum RL
Hunyuan Team, Tencent
Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
May 20, 2026Zihao Cheng, Hongru Wang, Zeming Liu +6Tool-Augmented Language Model Agents Terminal Agents
School of Computer Science and Engineering, Beihang University, Beijing, China · Independent Researcher · 3Beijing Institute of Technology +2

Jul 12, 2026J/K move · Enter open · S save

SETA: Scaling Environments for Terminal Agents

Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru, Aznaur Aliev+18

Imperial College London · University College London · SambaNova+5

Reinforcement Learning Terminal Agents Reinforcement Learning with Verifiable Rewards RL for LLM Agents

Open paper AskPDF arXiv

Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interfaces (GUIs). Among these, the terminal command line provides a text-based, general-purpose interface, covering tasks from system operations to data science and machine learning. However, scaling terminal-agent training remains challenging, as it requires diverse and coherent task instructions, executable environments, and reliable verification, while lacking naturally grounded supervision data. In this work, we propose SETA, a scalable framework for generating verifiable terminal environments for reinforcement learning (RL). The framework consists of two pipelines sharing a unified verification mechanism: SETA-Synth converts diverse sources into standardized RL environments, and SETA-Evol further expands from existing environments with adaptive control of difficulty and diversity. Together, we construct and release SETA-Env, the largest open-source verifiable terminal RL dataset to date, containing over 4,500 environments. We evaluate our dataset by training Qwen3-8B with GRPO on SETA-Env, achieving 12% pass rate on Terminal-Bench 2.0, the best reported result for an RL-trained model at the 8B scale. We further observe gains on DeepSeek-V4-Flash under the same terminal agent harness, with pass@1 on Terminal-Bench 2.0 improving from 40% to 43% and pass@5 improving from 54% to 58%. These results demonstrate that SETA- Env provides high-quality training environments for terminal agents and serves as a valuable resource for advancing research on terminal-based agent learning.

Abstract

LLM Agent Training

Explore similar work

SETA: Scaling Environments for Terminal Agents

Environment Evolution for Terminal Agents

Terminal-World: Scaling Terminal-Agent Environments via Agent Skills