On Data Engineering for Scaling LLM Terminal Capabilities

Renjie Pi; Grace Lam; Mohammad Shoeybi; Pooya Jannaty; Bryan Catanzaro; Wei Ping

cs.CLFeb 24, 2026

On Data Engineering for Scaling LLM Terminal Capabilities

Authors: Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, Wei Ping

Abstract

Despite rapid recent progress in the terminal capabilities of large language models, the training data strategies behind state-of-the-art terminal agents remain largely undisclosed. We address this gap through a systematic study of data engineering practices for terminal agents, making two key contributions: (1) Terminal-Task-Gen, a lightweight synthetic task generation pipeline that supports seed-based and skill-based task construction, and (2) a comprehensive analysis of data and training strategies, including filtering, curriculum learning, long context training, and scaling behavior. Our pipeline yields Terminal-Corpus, a large-scale open-source dataset for terminal tasks. Using this dataset, we train Nemotron-Terminal, a family of models initialized from Qwen3(8B, 14B, 32B) that achieve substantial gains on Terminal-Bench 2.0: Nemotron-Terminal-8B improves from 2.5% to 13.0% Nemotron-Terminal-14B improves from 4.0% to 20.2%, and Nemotron-Terminal-32B improves from 3.4% to 27.4%, matching the performance of significantly larger models. To accelerate research in this domain, we open-source our model checkpoints and most of our synthetic datasets at https://huggingface.co/collections/nvidia/nemotron-terminal.

Figures & tables

Figure 1: Overview of Terminal-Task-Gen. Our framework combines Dataset Adaptation, which transforms existing benchmarks into terminal prompts, with Synthetic Task Generation, which uses seed data and a Skill Taxonomy to construct targeted scenarios. The tasks from both streams are utilized during Trajectory Generation phase, where agents interact with Dockerized environments to produce solution traces, followed by Post-Processing (decontamination and filtering) to yield the final SFT dataset.

Closed Source Models
Model	GPT-5-Nano	GPT-5-Mini	Grok Code Fast 1	Grok 4	Gemini 2.5 Flash	Qwen3-Max-Thinking
Accuracy	7.90	24.0	14.2	23.1	16.9	22.5
Open Source Models
Model	Qwen3-32B	Qwen3-Coder	GPT-OSS (high) 20B	GPT-OSS (high) 120B	Nemotron-T-14B	Nemotron-T-32B
Size	32B	480B	20B	120B	14B	32B
Accuracy	3.37	23.9	3.10	18.7	20.2	27.4

Table 1: Model performance comparison on Terminal-Bench 2.0.

Figure 3

Benchmark (pass@1)	DeepSeek-V3.2
AIME 2024, AIME 2025	93.33
LiveCodeBench v6	67.20
SWE-bench Verified	52.40

Table 2: DeepSeek-V3.2 performance on adapted math, coding, and SWE benchmarks. Using the Terminus 2 agent, we evaluate DeepSeek-V3.2 on AIME, LiveCodeBench, and SWE-bench Verified adapted to Terminal-Bench format, and we find that it performs reasonably well even under this terminal-based setting.

Model	Size	TB2.0
Closed Source Models
GPT-5-Nano	–	7.90 $\pm$ 1.9
GPT-5-Mini	–	24.0 $\pm$ 2.5
GPT-5	–	35.2 $\pm$ 3.1
GPT-5.1	–	47.6 $\pm$ 2.8
GPT-5.2	–	54.0 $\pm$ 2.9

Table 3: Terminal-Bench 2.0 (TB2.0) results with Terminus 2 agent.

Category	Qwen3			Nemotron-Terminal
	8B	14B	32B	8B	14B	32B
Software & System
Software Engineering (24)	1.70	6.70	5.00	9.20	18.3	31.7
System Administration (9)	13.3	6.70	6.70	22.2	28.9	31.1
Debugging (3)	0.00	0.00	0.00	20.0	40.0	33.3
Security (8)	0.00	0.00	2.50	12.5	17.5	27.5

Table 4: Performance of models by Terminal-Bench category. TB2.0 scores for Qwen3 and Nemotron-Terminal models, broken down by category, showing where Nemotron-Terminal most improves over the Qwen3 baselines.

Table 7

Table 8

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Skill Types	Example Skill
Security	Systems, Data Processing, Web Security, Algorithmic, Testing	Craft exploit payloads to bypass authentication and identify vulnerabilities
Software Engineering	Algorithmic, Systems, Data Processing, Web Security, Testing, Mathematical	Implement graph traversal (BFS/DFS) for dependency resolution
File Operations	File I/O, Navigation, Data Parsing, Transformation, Archives, Resources, Network	Parse structured formats (JSON/XML/CSV) with encoding and validation
Data Querying	Query Construction, Data Comprehension, Graph Processing, Result Processing	Writing queries using the formal syntax of declarative query languages for structured data
Data Science	Systems, Data Processing, Algorithmic, Mathematical, Testing, Web Security	Load and transform tabular data with groupby, filtering, and aggregation
Debugging	Systems, Debugging, Testing, Algorithmic, Data Processing, Mathematical	Resolve package dependency conflicts through constraint analysis

Appendix

Table 10: Skills Summary by Domain

Figure 5: Distribution of # tokens in the generated trajectories.

Figure 6: Distribution of # turns in the generated trajectories.

Panel of Figure 7 — Figure 7: System prompt template for Terminus 2.

Panel of Figure 8 — Figure 8: Instruction template for math dataset adapter.

Panel of Figure 10 — Figure 10: Instruction template for SWE dataset adapter.

Panel of Figure 11 — Figure 11: Prompt template used for all skill-based generation. Domain-specific modules are inserted into section 2.

Panel of Figure 12 — Figure 12: Module for Data Processing tasks.

Panel of Figure 13 — Figure 13: Module for Data Querying tasks.

Panel of Figure 14 — Figure 14: Module for Data Science tasks.

Panel of Figure 15 — Figure 15: Module for Debugging tasks.

Panel of Figure 16 — Figure 16: Module for File Operations tasks.

Panel of Figure 17 — Figure 17: Module for Scientific Computing tasks.

Panel of Figure 18 — Figure 18: Module for Security tasks.

Panel of Figure 19 — Figure 19: Module for Software Engineering tasks.

Panel of Figure 20 — Figure 20: Module for System Administration tasks.

Terminal Agents

Terminal agents use command-line interfaces and filesystems to carry out multi-step tasks in software and other digital environments.

See all 31 papers

Tmax: A simple recipe for terminal agents
Jun 22, 2026Hamish Ivison, Junjie Oscar Yin, Rulin Shao +3Synthetic Data Generation Terminal Agents
α Allen Institute for AI · ω University of Washington
Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
May 20, 2026Zihao Cheng, Hongru Wang, Zeming Liu +6Tool-Augmented Language Model Agents Terminal Agents
School of Computer Science and Engineering, Beihang University, Beijing, China · Independent Researcher · 3Beijing Institute of Technology +2
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Jun 22, 2026Zhanbo Hua, Yifan Yao, Weihao Xie +14LLM Agent Trajectory Synthesis LLM Agents
Nanjing University · StepFun · ZODA +1

Jun 22, 2026J/K move · Enter open · S save

Tmax: A simple recipe for terminal agents

Hamish Ivison, Junjie Oscar Yin, Rulin Shao, Teng Xiao+2

α Allen Institute for AI · ω University of Washington

Synthetic Data Generation Terminal Agents Synthetic Data Generation for LLM Agents LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

Terminal-using agents have quickly become the most popular downstream application of language models (LMs). Despite their prevalence, relatively little academic work has examined RL-based training of these models, likely due to difficult benchmarks, a lack of data, and a lack of simple baseline recipes. We present Tmax, the strongest open RL recipe for terminal agents to date, bringing open data recipes closer to the frontier. While simple, our recipe achieves 27% on Terminal-Bench 2.0 with only 9B parameters, outperforming much larger models from prior work. Concretely, we generate data using a novel taxonomy, combining difficulty control, personas, and verifier diversification, which allows us to cheaply generate large amounts of terminal environments for RL and SFT training. We open-source our terminal dataset, which is over 2.5x larger than previously released terminal-agent datasets. We then train open-weight models using RL with our data, using a simple, outcome-only recipe. We release our data, models, and code as a strong baseline for future open academic work on terminal agents at https://github.com/hamishivi/tmax.

Abstract

Terminal Agents

Explore similar work

Tmax: A simple recipe for terminal agents

Terminal-World: Scaling Terminal-Agent Environments via Agent Skills

CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents