Terminal Agents: A Survey of AI Agents in Command-Line Environments
Authors: Yi Bin, Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, Wenqi Shao, Chen Qian, Wei Ye, Yujuan Ding, Zheng Wang, Pengpeng Zeng, Jingkuan Song, Heng Tao Shen
Organizations: Tongji University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · The Hong Kong Polytechnic University, Hong Kong, China
Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research. We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Using terminal-mediated execution as an organizing lens, this survey establishes workload-level boundaries and connects system architecture, competence acquisition, and evaluation through a seven-dimensional terminal competence profile. Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment. Executable trajectories ground learning in action consequences, verification, and recovery, whereas prevailing evaluations emphasize final outcomes and expose process quality, recovery, and governance unevenly. Bounded fixed-condition diagnostics illustrate two implications: benchmark families expose different process signals, and matched system comparisons reveal benchmark-dependent performance and limits of component attribution. These findings motivate explicit reporting of system and runtime conditions, supported by replayable traces and process-level evidence. The framework provides a unified basis for studying terminal-mediated agency across software engineering and emerging application domains.
Figures & tables
Figure 1: Terminal agents and related concepts. An agent controller, terminal substrate, and mutable environment state form a progress-bearing interaction loop. Systems are in scope when command execution drives progress, textual feedback guides later actions, and stateful environment interaction is central; surface-level CLI access and incidental command use remain adjacent.
Figure 2: Analytical storyline of the survey. The survey organizes the study of terminal agents around operational foundation, system architecture, competence acquisition, evaluation and diagnostics, and the resulting research agenda. Expanded branches summarize the principal analytical content of Sections II–VII. The column-aligned reference strip provides representative literature anchors and boundary comparators rather than one-to-one mappings to individual subtopics.
Figure 3: Operational scope of terminal agents, linking terminal-mediated interaction to workload-level boundary tests, scope categories, and competence dimensions.
Figure 4: Shifts in design emphasis toward terminal-mediated agency, from tool-augmented prompting to runtime- and harness-centered systems, with interfaces, workspaces, recovery, governance, and context management becoming increasingly explicit.
Figure 5: Layered architecture of terminal-agent systems. The central interaction loop is shaped by interface, runtime, verification, recovery, governance, and harness-level context mechanisms.
Figure 6: Terminal competence acquisition ecosystem. Data sources, interaction traces, runtime and governance mechanisms, and learning and adaptation approaches jointly shape the multidimensional competence profile.
Approach
Primary signal
Capability target
Blind spot
Key risk
SFT on successful traces
Successful interaction traces
Common commands, repository navigation, and standard workflows
Recovery, rollback, and diagnosis after wrong assumptions
Clean-trace overfit; absent failure paths
RL with environment rewards
Executable success, verifier reward, and test outcome
Outcome-driven exploration and completion
Process quality and safe intermediate behavior
Sparse-reward hacking; benchmark overfit
Process-aware / verifier-guided
Step judgments and verifier rankings
Diagnosis, action selection, and recovery decisions
Open-domain transfer
Verifier bias and shifted failures
Synthetic task generation
Generated tasks with controlled verification
Rare commands, setup patterns, and targeted subskills
Live realism and environment drift
Synthetic heuristics; weak transfer
Failure-conditioned training
Failed or repaired traces; hindsight relabeling
Recovery, diagnosis, and early failure detection
Generalization across inconsistent failure taxonomies
Noisy labels; repair-loop overfit
Runtime memory / context adaptation
Session history, summaries, and retrieved experience
Long-horizon persistence and workflow reuse
Correction under incorrect memory
Compression loss; memory contamination
Table 1: Complementary approaches to terminal competence acquisition and adaptation.
Emphasis
Primary evaluation focus
Representative examples
Principal blind spot
Repository repair
Issue resolution under tests
SWE-bench and SWE-PolyBench [ 67 , 119 ]
Conflates repository repair with terminal competence; command use is not directly measured
CLI/terminal-centered
Commands, output interpretation, and CLI workflows
Immature protocols; task success may conceal harmful actions
Table 2: Benchmark design emphases by primary evaluation focus and principal blind spot.
Benchmark
Terminal-competence dimensions
Eval. validity
Action form.
Feedback interp.
Runtime mgmt.
State tracking
Verify
Recover
Govern
Fresh.
SWE-bench / SWE-PolyBench [ 67 , 119 ]
I
T
I
T
T
I
–
–
Claw-SWE-Bench [ 183 ]
I
T
I
T
T
I
–
I
Terminal-Bench / TerminalWorld [ 101 , 26 ]
E
E
T
T
T
T
–
I
SetupBench [ 7 ]
E
E
E
T
T
T
–
–
LongCLI-Bench / GitTaskBench [ 39 , 105 ]
E
E
I
E
T
I
–
–
Table 3: Qualitative coding of terminal-competence observability and evaluation freshness. E=Explicit, T=Trace-visible, I=Incidental, and –=Not observable. Freshness concerns evaluation validity rather than competence.
Benchmark
Tasks
Outcome
P1 Rule inv. fail.
P2 Feedback
P3 Env. exit
P4 State err.
P5 Final verify
P6 Recovery
P7 Gov. trigger
Terminal-Bench 2.1 [ 101 ]
241
52.6%
0.0%
82.1%
12.5%
14.7%
29.5%
79.9%
0.5%
SetupBench [ 7 ]
93
59.1%
0.0%
80.2%
9.0%
19.5%
36.6%
85.5%
1.4%
LongCLI-Bench [ 39 ]
21
23.8%
0.0%
76.4%
6.8%
7.7%
23.8%
67.8%
4.5%
BashArena [ 70 ]
640
41.8%
0.0%
76.3%
13.7%
23.3%
66.2%
67.0%
3.0%
Table 4: Benchmark-exposure profile under mini-SWE-agent with DeepSeek-V4-Flash. Outcome is the official benchmark score. P1–P7 are bounded trace-derived process indicators. P1 and P3 record execution-friction signals; P2, P4, and P6 are auxiliary judge-assisted task-level rates; P5 records a detected final-window check; and P7 flags actions requiring contextual governance review.
System
F
P
Δ
Time F/P
Claw-SWE-Bench Lite
mini-SWE-agent
58.75%
56.25%
−2.50
476/476
SWE-agent [ 164 ]
76.25%
77.50%
+1.25
675/712
OpenHands [ 150 ]
68.75%
67.50%
−1.25
600/653
SWE-bench Lite
mini-SWE-agent
55.67%
55.67%
0.00
294/389
Table 5: Matched single-run results by benchmark and system. F/P denote DeepSeek-V4-Flash/Pro; entries report resolved rate, Pro-minus-Flash difference, and mean task time in seconds.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Description
Count
Corpus status
Research corpus
Research works retained for the survey synthesis.
186
Engineering practice, boundary, and disclosure references
Five engineering-practice or boundary entries and one disclosure citation.
6
Total cited set
Distinct cited entries after work-level version consolidation.
192
Release year of research entries
2022–2023
Earliest retained foundations for tool use and interactive execution.
4
Appendix
Table 6: Composition of the current review corpus. Release-year counts cover the 186 research entries; analytical-use counts are non-exclusive.
Stage
Procedure
Recorded decision
Role in synthesis
Scope freeze
Fix the 2022 to August 2026 interval, terminal-mediated research object, adjacent comparators, and workload-level boundary tests
Date range, source classes, and inclusion boundary
Prevents surface-level CLI mentions from entering as core evidence
Retrieval
Apply keyword families, venue-oriented search, and backward and forward reference chaining
Source, execution date, query form when retained, and work identity
Combines named-field retrieval with architecture, acquisition, evaluation, and governance routes
Work reconciliation
Group versions of the same contribution and retain the most complete research account
Preferred version and version relationship
Avoids double counting preprint, conference, and journal releases
Screening
Apply the five questions below and assign a corpus disposition
Research, engineering-practice, boundary, or exclusion disposition
Separates progress-bearing terminal evidence from surface-level or adjacent evidence
Analytical coding
Assign manuscript roles to every retained entry and apply architecture, acquisition, or evaluation fields to the relevant subsets
Entry-level roles and subset-specific analytical codes
Connects retained evidence to the architecture, acquisition, and evaluation synthesis
Claim calibration
Assess convergence, directness, domain breadth, and evidence maturity at the passage level
Bounded claim strength and scope
Keeps synthesis claims proportional to the available evidence
Appendix
Table 7: Corpus-construction workflow. Each stage produces the input required by the next stage and constrains how evidence enters the synthesis.
ID
Purpose
Boolean template
Q1
Named field
("terminal agent" OR "terminal agents" OR "CLI agent" OR "command-line agent")
Q2
Execution loop
("LLM agent" OR "language model agent") AND (terminal OR shell OR CLI OR "command line") AND (execution OR runtime OR feedback)
Q3
SWE and harnesses
("software engineering agent" OR "coding agent") AND (terminal OR shell OR command) AND (benchmark OR harness OR environment)
Q4
Acquisition
("agent trajectory" OR "executable trajectory" OR "environment interaction") AND (training OR post-training OR "reinforcement learning") AND (terminal OR CLI OR repository)
Q5
Process evaluation
("terminal benchmark" OR "CLI benchmark" OR "agent benchmark") AND (trace OR process OR recovery OR verification OR state)
Q6
Governance
("LLM agent" OR "coding agent" OR "computer-use agent") AND (shell OR terminal OR "code execution") AND (sandbox OR permission OR safety OR governance)
Appendix
Table 8: Boolean query families used for subsequent corpus updates. Each source-specific execution is logged with its date and hit count.
Field group
Coded fields
Unit
Analytical use
Corpus identity
Bibliographic metadata, corpus status, and unique work identity
Work
Supports corpus identity, release-year counts, and work-level deduplication
Analytical role
Scope, architecture, acquisition, evaluation, empirical setting, or implications
Work, non-exclusive
Connects retained entries to the survey passages they support
Architecture
Interface granularity, operational coupling, autonomy regime, recovery and control style, planning strategy
System or artifact
Supports the layered architecture and recurring-pattern synthesis
Acquisition
Data source, trajectory source, supervision signal, filtering or relabeling, failure-data treatment, capability target
Dataset, environment, or pipeline
Supports comparison of executable data, learning, and adaptation routes
Supports benchmark-family and competence-observability comparisons
Evidence calibration
Convergence, directness, domain breadth, and maturity of evidence for the passage claim
Synthesis passage
Bounds the strength and generality of each synthesized claim
Appendix
Table 9: Level-specific coding schema used in corpus construction and synthesis. Entry-level fields cover every retained reference; specialized fields apply to relevant evidence subsets or synthesis passages.
Field
Value
Field
Value
Judge model
DeepSeek-V4-Flash
Prompt design
Separate prompts for feedback use, state tracking, and recovery
Temperature
0.0
Top- p
1.0
Initial maximum output
4,096 tokens
Local seed
42, passed when supported
Response mode
One non-streaming JSON object
Retry policy
At most three retries; output cap doubles on reasoning-only truncation, up to 32,768 tokens
Eligibility filter
Confidence ≥0.7 and not uncertain
Aggregation
Task-level semantic rates over eligible evidence
Appendix
Table 10: Configuration of the semantic episode judge used for formal P2, P4, and P6 labels.
Figure 7: Matched-system outcomes for all 12 benchmark–system–model cells. Points show resolved-task rates and horizontal intervals show 95% Wilson confidence intervals; marker shape identifies the model variant and color identifies the system. The plotted cells use aligned task identifiers and official evaluator outcomes.
Benchmark
System
Both fail
Pro only
Flash only
Both pass
P − F (pp)
95% CI (pp)
Exact p
Holm p
Claw Lite
mini-SWE-agent
28
5
7
40
−2.50
[ −11.25 , 6.25]
0.774
1.000
Claw Lite
SWE-agent
16
3
2
59
+1.25
[ −3.75 , 6.25]
1.000
1.000
Claw Lite
OpenHands
22
3
4
51
−1.25
[ −7.50 , 5.00]
1.000
1.000
SWE-bench Lite
mini-SWE-agent
120
13
13
154
0.00
[ −3.33 , 3.33]
1.000
1.000
SWE-bench Lite
SWE-agent
112
12
13
163
−0.33
[ −3.67 , 3.00]
1.000
1.000
SWE-bench Lite
OpenHands
133
12
12
143
0.00
[ −3.00 , 3.33]
1.000
1.000
Appendix
Table 11: Task-paired outcome transitions between Flash and Pro. “Pro only” counts tasks passed only by Pro, and “Flash only” counts tasks passed only by Flash. Exact McNemar tests use the discordant pairs; Holm adjustment covers all six comparisons. Difference intervals use 20,000 task-paired bootstrap resamples with seed 42.
Benchmark
Model
System A
System B
A only
B only
A − B (pp)
Exact p
Holm p
Claw Lite
Flash
mini-SWE-agent
SWE-agent
3
17
−17.50
0.0026
0.0232
Claw Lite
Flash
mini-SWE-agent
OpenHands
8
16
−10.00
0.1516
0.4033
Claw Lite
Flash
SWE-agent
OpenHands
9
3
+7.50
0.1460
0.4033
Claw Lite
Pro
mini-SWE-agent
SWE-agent
2
19
−21.25
0.0002
0.0022
Claw Lite
Pro
mini-SWE-agent
OpenHands
5
14
−11.25
0.0636
0.4033
Claw Lite
Pro
SWE-agent
OpenHands
12
4
+10.00
0.0768
0.4033
Appendix
Table 12: Task-paired cross-system contrasts. “A only” and “B only” count discordant tasks resolved by one system. Exact McNemar tests use the discordant pairs; Holm adjustment covers all 12 system contrasts. The difference is A minus B in percentage points.
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 representative, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve. Data and code are available at https://github.com/EuniAI/TerminalWorld.
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
University College London · Nanjing University · Tencent
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.
Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interfaces (GUIs). Among these, the terminal command line provides a text-based, general-purpose interface, covering tasks from system operations to data science and machine learning. However, scaling terminal-agent training remains challenging, as it requires diverse and coherent task instructions, executable environments, and reliable verification, while lacking naturally grounded supervision data. In this work, we propose SETA, a scalable framework for generating verifiable terminal environments for reinforcement learning (RL). The framework consists of two pipelines sharing a unified verification mechanism: SETA-Synth converts diverse sources into standardized RL environments, and SETA-Evol further expands from existing environments with adaptive control of difficulty and diversity. Together, we construct and release SETA-Env, the largest open-source verifiable terminal RL dataset to date, containing over 4,500 environments. We evaluate our dataset by training Qwen3-8B with GRPO on SETA-Env, achieving 12% pass rate on Terminal-Bench 2.0, the best reported result for an RL-trained model at the 8B scale. We further observe gains on DeepSeek-V4-Flash under the same terminal agent harness, with pass@1 on Terminal-Bench 2.0 improving from 40% to 43% and pass@5 improving from 54% to 58%. These results demonstrate that SETA- Env provides high-quality training environments for terminal agents and serves as a valuable resource for advancing research on terminal-based agent learning.
Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru +19
Imperial College London · University College London · SambaNova +5