When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenge

Xi Qin; Isabel Kurth; Xin Cui; Elin Park; Alexander Schaefer; Yaad Oren

cs.AIOct 1, 2026

When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenge

Authors: Xi Qin, Isabel Kurth, Xin Cui, Elin Park, Alexander Schaefer, Yaad Oren

Organizations: SAP Lab Palo Alto, CA 94304

Abstract

Using a frontier model like Claude Opus as a meta-agent to generate terminal tasks and verifiers for RL training is increasingly common. Yet a runnable Docker image and executable test suite do not guarantee a faithful end-to-end pipeline for terminal agent training. We present a meta-agent pipeline motivated by this gap, diagnosing three classes of failure: benchmark invalidity, harness brittleness, and reward misalignment. Prompt redesign and context extension raise baseline solvability 5.6 times, but a 9B model saturates at 81.3% mean pass@2 within 20 steps on Claude Opus-generated tasks. Adding hard tasks reduces mean pass@2 to 20.6% without changing the training configuration, a strong evidence that the solvability band is model-specific. These findings demonstrate that meta-agent reliability requires solvability-band calibration, verifier audits, and infrastructure error accounting as first-class evaluation criteria, not post-hoc diagnost.

Figures & tables

Panel of Figure 1 — Figure 1: Overall pipeline. Stages 1–5 generate Harbor task directories. Pre-agent validation filters structural failures before rollout. Solution sampling assigns empirical solvability labels, which control RL training signal.

Dataset	Model	Context	Total tasks	Pass@1	Category Strategy
$V1$ (Dataset-4k)	Opus 4.6	4096	5038	$\sim$ 10%	Round-robin
$V2$ (Dataset-8k)	Opus 4.6	8192	4929	56.5%	Difficulty and category coupling
$V3$ (Dataset-5k)	Opus 4.8	8192	8218	39.6%	Difficulty first, category next

Table 1: Dataset overview. For solvability measured by pass@1, Claude 4.6 Sonnet filters for

V1

and

V2

, and Claude 4.7 Opus for

V3

specifically.

Figure 2: Left: task categories in all three datasets; no single category dominates. Right: instruction length distributions for all datasets.

Figure 3: Assertion semantic types across

V1

V2

, and

V3

datasets.

Exp.	Model	Dataset	Key result
$E1$	Qwen2.5-3B	$V1$	7.8% base = best pass@1
$E2$	Qwen3.5-9B	$V1$ + $V2$	81.3% mean pass@2
$E3$	Qwen3.5-9B	$V1$ + $V2$ + $V3$ hard	20.6% mean pass@2

Table 2: Training evidence for dataset calibration.

Training set	Steps	Mean pass@2	Mean reward std.
$V1$ + $V2$	20	81.3% -> 100%	0.42
$V1$ + $V2$ + $V3$ hard	190	9.8% -> 10.1%	0.37

Table 3: Changing the training distribution changes rollout difficulty.

Panel of Figure 4 — Figure 4: Failure modes enter at generation, environment execution, and evaluation-operation stages; they require separate structural, solvability, and error-rate checks.

Failure mode	Observed scope	Implication
Hard-task under-representation	7.0% of a 3,310 task source batch	Insufficient hard cases can leave capable policies near reward saturation.
Dockerfile parse failures	$\sim$ 74% heredocs parsing errors	Generation defects can dominate apparent agent failure.
Docker network-pool exhaustion	$\sim$ 9.1% in solution generation	Infrastructure errors bias aggregate evaluation until controlled.
Docker build failures	$\sim$ 9.7% in solution generation	Fragile setup steps consume trials without producing trajectories. This defect remains to be fixed.

Table 4: Observed failure modes from distinct runs.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Chain complexity distribution across

V1

V2

, and

V3

datasets.

Figure 6: Tool breadth distribution across

V1

V2

, and

V3

datasets.

Figure 7: Assertion count distribution across

V1

V2

, and

V3

datasets.

Figure 8:

V2

dataset deep-dive. Left: pass@k on

V2

tasks evaluated by Claude 4.6 Sonnet (pass@1 = 60.2%, plateaus near 63%). Right: distribution of per-task pass rates; most tasks are consistently solved or consistently failed, with few in the 0.2–0.8 middle band.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 590 papers

Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training
Jul 30, 2026Zhihong Pan, Jiyuan He, Kai Zhang +5Synthetic Data Generation Terminal Agents
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Meituan
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Jun 22, 2026Zhanbo Hua, Yifan Yao, Weihao Xie +14LLM Agent Trajectory Synthesis LLM Agents
Nanjing University · StepFun · ZODA +1

Jul 30, 2026J/K move · Enter open · S save

Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training

Zhihong Pan, Jiyuan He, Kai Zhang, Yupeng Han+4

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Meituan

Synthetic Data Generation Terminal Agents LLM Agent Trajectory Synthesis LLM Agent Training

Open paper AskPDF arXiv

Training terminal agents at scale requires diverse, verifiable terminal tasks and high-quality interaction trajectories, yet acquiring such data remains a significant challenge. Existing synthesis methods face two key limitations: (1) weak reliability caused by the disconnect between task generation and real execution, and (2) limited diversity and scalability due to dependence on existing repositories. We propose Meta-Task, a framework that redefines terminal task synthesis as a Terminal-Bench-format task itself: an agent operates within a real container environment to iteratively generate, execute, and verify tasks, so that synthesized components are checked for internal consistency and executability within the generation loop itself. Building upon this, we decouple the target task requirements along multiple dimensions, introduce a multi-phase mechanism that dynamically designs novel task specifications before producing the actual tasks, and incorporate optional external material support to enhance diversity and realism. We additionally apply LLM-as-Judge filtering to ensure the quality of the final training data. Experiments on Terminal-Bench 2.0 show that fine-tuning on only 3,221 Meta-Task synthesized trajectories achieves 22.5% and 31.8% Avg Pass@1 for Qwen3-14B and Qwen3-32B respectively, outperforming concurrent approaches with significantly less training data.

Abstract

LLM Agent Training

Explore similar work

Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training

What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design

CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents