Self-Supervised Scaling of Terminal Environments for Scientific Domains

Zhongzhi Li; Yucheng Shi; Zongxia Li; Junyao Yang; Ruhan Wang; Yu Wang; Jingyuan Huang; Jichao Yu

cs.SEOct 2, 2026

Self-Supervised Scaling of Terminal Environments for Scientific Domains

Authors: Zhongzhi Li, Yucheng Shi, Zongxia Li, Junyao Yang, Ruhan Wang, Yu Wang, Jingyuan Huang, Jichao Yu,

Organizations: Tencent HY LLM Frontier · University of Georgia · University of Maryland, College Park · National University of Singapore ·

Abstract

Terminal agents are increasingly deployed beyond software engineering in science and other specialized domains. Constructing training environments requires executable reference behavior and a domain-specific verifier that distinguishes semantic correctness from superficially plausible artifacts. Authoring these components for each task requires repeated engineering and limits reuse. We introduce software-in-the-loop reconstruction, a self-supervised framework that obtains reference outputs and verification targets from existing software workflows, executable programs mapping structured inputs to outputs. For each workflow, we execute multiple input configurations and partition cases into public observations and hidden evaluations. Given the instruction, input schema, and public input--output observations, an agent constructs an editable program without access to the source workflow. The candidate is evaluated on hidden configurations against workflow outputs. A hierarchical verifier combines domain-specific semantic comparison, structural validity, and anti-shortcut checks, while public feedback supports iterative revision. The construction admits additional workflows and configurations without authoring a reference solution for each task. We instantiate SWR with 500 workflows and 46 software families across six domains. Across three attempts per task, Qwen3.8-Max solves 838 tasks and produces 1,422 verified trajectories, which we oversample to 3,000 reconstruction-only training examples. Supervised fine-tuning of Qwen3.8-27B improves mean Terminal-Bench 2 performance from 47.94% to 53.56% across three seeds and achieves the highest mean among four matched-token corpus controls on all four reported evaluations. These results indicate that existing scientific software can provide scalable, behaviorally verified supervision for terminal agents.

Figures & tables

Figure 2: Software-in-the-loop reconstruction. Public observations are the input–output examples released to the agent. They guide construction of a reusable program. Withheld inputs supply the reference outputs used for verification.

Figure 3: Comparison of verification standards. Structural checks can accept an invalid constant output, while byte-identical comparison can reject a behaviorally equivalent output. Semantic verification evaluates task-relevant behavior and is designed to reduce both failure modes.

Panel of Figure 4 — Figure 4: Composition of SWR. Panel (a) is the task distribution across software domains. Panel (b) is the artifact types. Panel (c) is the scenario parameter types and dimensions.

Dataset	Tasks	Families	Workflows	Public scenarios	Hidden scenarios
SWR	3,000	46	500	15,600	8,400

Table 1: Composition of the evaluated SWR release. A workflow is an existing program that maps an input to an output. The columns count tasks, software families, workflows, and public and hidden input configurations.

Figure 5: Sensitivity of SWR outputs to input changes. The four metrics record whether a pair of configurations changes any output, how many semantic fields vary, how many fields change in an average pair, and how many declared parameters affect the output.

Scheduled	Records	Strict pass	Pass@1	Record rate	Pass/record
3,000	2,978	685	22.8%	99.3%	23.0%

Table 2: Single-attempt evaluation of Qwen3.8-Max on SWR with the finalized verifier. Pass@1 uses all 3,000 scheduled tasks. Missing records count as failures.

Panel of Figure 6 — Figure 6: Eight-model comparison on SWR100. Panel (a) is overall strict success. Panel (b) is success by domain.

Figure 7: Eight-model outputs for an OpenSCAD snap-fit task with a 16 mm public example and a 21 mm hidden reference. GPT-5.6 Sol preserves separate body and lid components, whereas the other seven outputs collapse to plate-like meshes at wall-thickness height.

Result	Verified tasks/trajectories	Rate
Pass@3 / task union	838	27.9%
All verified trajectories	1,422	–

Table 3: Three-attempt result of Qwen3.8-Max on SWR. The columns give verified tasks or trajectories and the corresponding rate.

Training entries	TB2	TB4	LHTB reward	SWR100 success
Base	$47.94\pm 1.30$	$0.51\pm 0.87$	$20.67\pm 1.15$	$1.33\pm 0.58$
SFT-750	$50.19\pm 0.65$	$1.01\pm 0.87$	$22.33\pm 0.58$	$1.33\pm 0.58$
SFT-1500	$50.56\pm 1.12$	$2.02\pm 0.87$	$24.33\pm 1.53$	$2.67\pm 0.58$
SFT-3000	$53.56\pm 0.65$	$3.54\pm 0.87$	$27.67\pm 0.58$	$3.33\pm 0.58$

Table 4: Training-entry budgets for Qwen3.8-27B reconstruction SFT, including repeated samples. Entries are means and standard deviations over seeds 42, 43, and 44. Each seed is trained and tested separately. All entries are percentages.

Training corpus	Loss tokens	TB2	TB4	LHTB	SWR100
Base (no SFT)	–	$47.94\pm 1.30$	$0.51\pm 0.87$	$20.67\pm 1.15$	$1.33\pm 0.58$
OpenThoughts-Agent-v1	88.48M	$49.06\pm 1.30$	$0.00\pm 0.00$	$22.67\pm 1.53$	$1.67\pm 0.58$
TMax	88.48M	$49.44\pm 2.25$	$1.01\pm 0.87$	$20.67\pm 1.53$	$1.33\pm 0.58$
LiteCoder-Terminal	88.48M	$48.69\pm 1.72$	$0.51\pm 0.87$	$23.00\pm 1.00$	$1.00\pm 1.00$
Nemotron-Terminal	88.48M	$47.19\pm 1.12$	$0.51\pm 0.87$	$23.67\pm 1.15$	$1.33\pm 1.53$
Reconstruction (ours)	88.48M	$\mathbf{53.56\pm 0.65}$	$\mathbf{3.54\pm 0.87}$	$\mathbf{27.67\pm 0.58}$	$\mathbf{3.33\pm 0.58}$

Table 5: Matched-token comparison of Qwen3.8-27B training corpora. Entries are means and standard deviations over three seeds, reported as percentages. Every fine-tuned model receives 88.48M assistant-loss tokens. The reconstruction corpus contains 3,000 sampled entries, including repeats.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Tasks	Runs/model	Turn limit	Wall time	Retained outcomes
Full-suite evaluation	3,000	1	100	1 hour	Success and failure
Three-attempt Pass@3	3,000	3	–	–	Final task union
SWR100 comparison	100	1	500	3 hours	Success and failure
SFT scaling	4 suites	3	–	–	Train-and-test seeds

Appendix

Table 6: Protocols for the reported evaluations. Columns give the task count, runs per model, turn limit, wall-clock limit, and which outcomes are retained.

Figure 8: Output types of the 46 software families. Families are grouped by domain. A filled mark means every task in the family produces that output type. An open mark means only some tasks do. In CMake builds, source code appears in 36 of 60 tasks and configuration files appear in 54 of 60. The bars give the median number of checkable output fields on a logarithmic scale.

Figure 9: Released public artifacts from the six domains. Each row is one domain and contains three examples taken from the files released with the task. The software panels redraw a released JSON schema, a released parse tree, and the RTL module names in a released file list. The other panels are released waveforms, source-program meshes, a map boundary, a molecule, biological sequences, a query result, a musical score, glyph positions, a well log, and two views of one seismic trace.

Figure 10: Domain and software-family composition of SWR. Families are grouped into six domains. No family exceeds 2.4% of the tasks.

Figure 11: Number of checkable output fields per task in SWR. Digest-like profiling fields are excluded.

Control	Hidden-field agreement	Strict success
Oracle	100.0%	100.0%
Oracle-selected best-public replay	44.3%	0.0%
Nearest public scenario	38.0%	0.0%
Constant public output	34.3%	0.0%

Appendix

Table 7: Public-output replay audit on SWR. Hidden-field agreement is the share of hidden fields matched by a replay control. Strict success requires a complete hidden pass.

Panel of Figure 12 — Figure 12: Interaction cost in the single-attempt SWR evaluation. Panel (a) is the distribution of trajectory length. Panel (b) is family pass rate against median turns.

Model	Seed	TB2 (%)	TB4 (%)	LHTB	SWR100
Base	42	47.19	0.00	22.00	1.00
Base	43	49.44	1.52	20.00	1.00
Base	44	47.19	0.00	20.00	2.00
SFT-750	42	50.56	1.52	23.00	1.00
SFT-750	43	49.44	1.52	22.00	2.00
SFT-750	44	50.56	0.00	22.00	1.00

Appendix

Table 8: Per-seed training budgets for Qwen3.8-27B. These rows underlie Table 4 . LHTB and SWR100 entries are percentages.

Training corpus	Seed	TB2	TB4	LHTB	SWR100
OpenThoughts-Agent-v1	42	48.31	0.00	21.00	1.00
OpenThoughts-Agent-v1	43	50.56	0.00	23.00	2.00
OpenThoughts-Agent-v1	44	48.31	0.00	24.00	2.00
TMax	42	49.44	0.00	21.00	2.00
TMax	43	47.19	1.52	19.00	1.00
TMax	44	51.69	1.52	22.00	1.00

Appendix

Table 9: Per-seed scores for the four matched-token control corpora. Every evaluation entry is a percentage.

Panel of Figure 13 — Figure 13: SWR100 trajectory profiles under a three-hour limit per task. Panel (a) is agent turns. Panel (b) is prompt and output tokens. Points are medians. Horizontal segments are interquartile ranges.

Model	Records	Scored	Pass	Turns	Tokens (M)
GPT-5.6 Sol	99	89	20	191.0	10.12
Kimi-K3	98	98	6	131.5	6.94
Grok 4.6	100	99	4	98.0	5.03
DeepSeek-V4-Pro	100	99	3	187.5	10.00
Qwen3.8-Max	100	100	3	356.5	18.59
GLM-5.3	100	100	3	69.5	4.49

Appendix

Table 10: Per-model results on SWR100. Pass@1 uses all 100 scheduled tasks for each model. Turns and tokens are medians.

Mutation class	Trials	Accepted	Rejected
Valid near-boundary controls	7,065	7,065	0
Single deep-leaf change	2,940	0	2,940
Correlated numeric shift	2,868	0	2,868
Wrong hidden scenario	2,939	0	2,939
Ignored input axis	9,901	0	9,901
Numeric perturbation $>1\times$ tolerance	9,260	0	9,260

Appendix

Table 11: Stress test of the enhanced verifier on the full SWR release. Acceptance counts as correct for a valid control and as incorrect for every other mutation class.

Panel of Figure 14 — Figure 14: Stress tests of the enhanced verifier. Panel (a) uses 2,315 outputs with finite floating-point values at each perturbation size. Panel (b) reports detection rates with Wilson 95% lower bounds. Labels give rejected counts and totals.

Audit stratum	Trials	False accept	False reject	Correct	95% CI
Structured valid controls	376	0	0	100.00%	[98.99, 100.00]
Geometry valid controls	24	0	6	75.00%	[55.10, 88.00]
Structured invalid mutations	752	2	0	99.73%	[99.04, 99.93]
Geometry invalid mutations	48	1	0	97.92%	[89.10, 99.63]
All held-out trials	1,200	3	6	99.25%	[98.58, 99.60]

Appendix

Table 12: Verifier audit after freezing and before human adjudication. Intervals are Wilson 95% intervals for the correct-decision rate.

Audit stratum	Tasks	Bundles	Label validity	Verifier agreement	Annotator agreement
Family-stratified core	92	184	98.91% (182/184)	98.37% (181/184)	96.74% (178/184)
Risk-enriched	8	16	87.50% (14/16)	93.75% (15/16)	87.50% (14/16)
All	100	200	98.00% (196/200)	98.00% (196/200)	96.00% (192/200)

Appendix

Table 13: Blinded human audit of the held-out verifier set. Label validity and verifier agreement use adjudicated judgments. Annotator agreement is measured before adjudication.

Human-adjudicated label	Verifier accepts	Verifier rejects	Total
Valid / equivalent	98	2	100
Invalid / non-equivalent	2	98	100
Total	100	100	200

Appendix

Table 14: Verifier decisions after adjudication on 200 blinded bundles. Human-adjudicated semantic validity is the reference label.

Figure 15: Primary outcomes on SWR100, grouped by the order of evaluation checks. Every model receives the same 100 tasks. Rare geometry errors, missing records, and residual outcomes are grouped as Other.

Task mechanism	Passing trajectory	Failing trajectory	Decisive evidence and interpretation
Color-transform numerics	GPT-5.6 Sol, 213 turns, 41 candidate tests	Qwen3.8-Max, 376 turns, 25 candidate tests	The hidden mean is 0.291619 rather than 0.267706, and every reported XYZ component exceeds tolerance after 45 public-feedback calls.
Container SBOM completion	GPT-5.6 Sol, 110 turns, 11 edits, 17 tests	Grok 4.6, 149 turns, one edit, one test	No executable candidate remains, so semantic grading never begins. The final plan still defers implementation until after another search.
Robotics hidden token	GPT-5.6 Sol, 31 turns, 0.97M tokens	Qwen3.8-Max, 368 turns, 20.60M tokens	After six public passes, an unseen categorical value is cast to an integer and causes hidden execution to fail. Public completion conceals a parser that cannot process the new value.
Video policy category	Kimi-K3, 64 turns, 14 tests	DeepSeek-V4-Pro, 204 turns, seven tests	A closed vocabulary rejects an unseen audio-layout category. The passing implementation maps unseen categorical values deterministically.

Appendix

Table 15: Four matched pairs of a success and a failure on the same task. Turns and tests summarize interaction effort. The last column states the evidence tied to each outcome.

Figure 16: Same-task numeric generalization. GPT-5.6 Sol passes after 213 turns and 41 candidate tests. Qwen3.8-Max uses 376 turns and 45 feedback calls but only 25 candidate tests. Its hidden mean is 0.291619 rather than 0.267706, and every reported XYZ component is outside tolerance. The failure is joint transform recovery, not a lack of public checks.

Figure 17: Same-task artifact completion. GPT-5.6 Sol turns 110 turns into 11 edits and 17 candidate tests and leaves a passing executable fallback. Grok 4.6 spends 149 turns and 7.54M tokens on evidence search but makes one edit and one test. Its final plan still defers implementation. The workspace contains no executable candidate.

Figure 18: Same-task hidden-input robustness. GPT-5.6 Sol passes in 31 turns with a representation that accepts unseen categorical values. Qwen3.8-Max uses 20.60M tokens and declares completion after all six public scenarios pass, but an invalid integer conversion then prevents execution on a hidden input. The contrast separates hidden-input robustness from completion on public feedback.

Figure 19: Same-task policy generalization. Kimi-K3 maps unseen categorical values deterministically and passes after 64 turns and 14 candidate tests. DeepSeek-V4-Pro reads seven times as much public evidence but runs only seven candidate tests. Its closed vocabulary rejects an unseen audio-layout category. The trajectory then returns to external search instead of adding a fallback for unknown categories.

Terminal Agents

Terminal agents use command-line interfaces and filesystems to carry out multi-step tasks in software and other digital environments.

See all 31 papers

Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification
Sep 30, 2026Yingfeng Luo, Shaowei Wei, Daixin Wang +7LLM Agent Verification LLM Agent Reliability
Northeastern University · Inclusion AI, Ant Group · University of Maryland, College Park
AutoEnvScaling: Automating the Data Flywheel with Terminal Agents
Oct 9, 2026Simon Yu, Baolin Peng, Bo Liu +8LLM Agent Training Self-Evolving Agents
Northeastern University · Microsoft · Stanford University +2
SWE-Universe: Scale Real-World Verifiable Environments to Millions
Feb 2, 2026Mouxiang Chen, Lei Zhang, Yunlong Feng +16Software Engineering Benchmark Construction
Qwen Team, Alibaba Group · Zhejiang University

Sep 30, 2026J/K move · Enter open · S save

Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification

Yingfeng Luo, Shaowei Wei, Daixin Wang, Dingyang Lin+6

Northeastern University · Inclusion AI, Ant Group · University of Maryland, College Park

LLM Agent Verification LLM Agent Reliability

Open paper AskPDF arXiv

Terminal agents rely on self-verification to assess and correct their solutions as they solve tasks through interaction with command-line environments. Yet how trustworthy such self-verification is remains poorly understood. To investigate this question systematically, we introduce a diagnostic framework that identifies the first complete solution in each trajectory, determines whether it is objectively correct, and uses this ground truth to quantify the agent's subsequent verification and recovery behavior. Applying it to ten terminal agents on TerminalBench2.1, we find that verification is nearly universal after a complete candidate is formed, yet only 61.43% of incorrect candidates are detected and only 49.36% of detected errors are successfully repaired. These results show that the main weakness in self-verification lies not in initiating verification, but in detecting and repairing errors. Motivated by these findings, we propose Student-Conditioned Verification Distillation (SCVD), which lets the student first produce a candidate solution and distills a stronger teacher's subsequent verification and recovery from the same interaction context. Across three Qwen3.5 backbones, SCVD improves Pass@1 on TerminalBench2.1 by 9.74--16.85 percentage points over the corresponding base models and by 4.49--8.61 points over the standard full-trajectory distillation, while avoiding the pronounced out-of-distribution degradation of full-trajectory distillation on SWE-bench Verified.

Abstract

Terminal Agents

Explore similar work

Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification

AutoEnvScaling: Automating the Data Flywheel with Terminal Agents

SWE-Universe: Scale Real-World Verifiable Environments to Millions