cs.AIAug 20, 2026
SaveTerminal Agents: A Survey of AI Agents in Command-Line Environments
Organizations: Tongji University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · The Hong Kong Polytechnic University, Hong Kong, China
Abstract
Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research. We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Using terminal-mediated execution as an organizing lens, this survey establishes workload-level boundaries and connects system architecture, competence acquisition, and evaluation through a seven-dimensional terminal competence profile. Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment. Executable trajectories ground learning in action consequences, verification, and recovery, whereas prevailing evaluations emphasize final outcomes and expose process quality, recovery, and governance unevenly. Bounded fixed-condition diagnostics illustrate two implications: benchmark families expose different process signals, and matched system comparisons reveal benchmark-dependent performance and limits of component attribution. These findings motivate explicit reporting of system and runtime conditions, supported by replayable traces and process-level evidence. The framework provides a unified basis for studying terminal-mediated agency across software engineering and emerging application domains.
Figures & tables
Figure 1: Terminal agents and related concepts. An agent controller, terminal substrate, and mutable environment state form a progress-bearing interaction loop. Systems are in scope when command execution drives progress, textual feedback guides later actions, and stateful environment interaction is central; surface-level CLI access and incidental command use remain adjacent.
Figure 2: Analytical storyline of the survey. The survey organizes the study of terminal agents around operational foundation, system architecture, competence acquisition, evaluation and diagnostics, and the resulting research agenda. Expanded branches summarize the principal analytical content of Sections II–VII. The column-aligned reference strip provides representative literature anchors and boundary comparators rather than one-to-one mappings to individual subtopics.
Figure 3: Operational scope of terminal agents, linking terminal-mediated interaction to workload-level boundary tests, scope categories, and competence dimensions.
Figure 4: Shifts in design emphasis toward terminal-mediated agency, from tool-augmented prompting to runtime- and harness-centered systems, with interfaces, workspaces, recovery, governance, and context management becoming increasingly explicit.
Figure 5: Layered architecture of terminal-agent systems. The central interaction loop is shaped by interface, runtime, verification, recovery, governance, and harness-level context mechanisms.
Figure 6: Terminal competence acquisition ecosystem. Data sources, interaction traces, runtime and governance mechanisms, and learning and adaptation approaches jointly shape the multidimensional competence profile.
| Approach | Primary signal | Capability target | Blind spot | Key risk |
| SFT on successful traces | Successful interaction traces | Common commands, repository navigation, and standard workflows | Recovery, rollback, and diagnosis after wrong assumptions | Clean-trace overfit; absent failure paths |
| RL with environment rewards | Executable success, verifier reward, and test outcome | Outcome-driven exploration and completion | Process quality and safe intermediate behavior | Sparse-reward hacking; benchmark overfit |
| Process-aware / verifier-guided | Step judgments and verifier rankings | Diagnosis, action selection, and recovery decisions | Open-domain transfer | Verifier bias and shifted failures |
| Synthetic task generation | Generated tasks with controlled verification | Rare commands, setup patterns, and targeted subskills | Live realism and environment drift | Synthetic heuristics; weak transfer |
| Failure-conditioned training | Failed or repaired traces; hindsight relabeling | Recovery, diagnosis, and early failure detection | Generalization across inconsistent failure taxonomies | Noisy labels; repair-loop overfit |
| Runtime memory / context adaptation | Session history, summaries, and retrieved experience | Long-horizon persistence and workflow reuse | Correction under incorrect memory | Compression loss; memory contamination |
Table 1: Complementary approaches to terminal competence acquisition and adaptation.
| Emphasis | Primary evaluation focus | Representative examples | Principal blind spot |
| Repository repair | Issue resolution under tests | SWE-bench and SWE-PolyBench [ 67 , 119 ] | Conflates repository repair with terminal competence; command use is not directly measured |
| CLI/terminal-centered | Commands, output interpretation, and CLI workflows | Terminal-Bench , TerminalWorld , and LongCLI-Bench [ 101 , 26 , 39 ] | Mixes terminal-native and repository-mediated tasks, leaving transfer unclear |
| Setup | Environment setup and dependency resolution | SetupBench [ 7 ] | Often isolated from end-to-end workflows |
| Process | Intermediate behavior and scaffold compliance | OctoBench , ProcBench , and AppWorld [ 31 , 57 , 143 ] | Lacks standardized process scoring |
| Long horizon | Persistence, drift, and temporal coherence | SWE-Bench Pro , LoCoEval , and LifelongAgentBench [ 30 , 91 , 182 ] | Costly and difficult to reproduce at scale |
| Safety/governance | Permissions, containment, and privileged commands | BashArena , ClawSafety , and AgentHazard [ 70 , 152 , 40 ] | Immature protocols; task success may conceal harmful actions |
Table 2: Benchmark design emphases by primary evaluation focus and principal blind spot.
| Benchmark | Terminal-competence dimensions | Eval. validity | ||||||
| Action form. | Feedback interp. | Runtime mgmt. | State tracking | Verify | Recover | Govern | Fresh. | |
| SWE-bench / SWE-PolyBench [ 67 , 119 ] | I | T | I | T | T | I | – | – |
| Claw-SWE-Bench [ 183 ] | I | T | I | T | T | I | – | I |
| Terminal-Bench / TerminalWorld [ 101 , 26 ] | E | E | T | T | T | T | – | I |
| SetupBench [ 7 ] | E | E | E | T | T | T | – | – |
| LongCLI-Bench / GitTaskBench [ 39 , 105 ] | E | E | I | E | T | I | – | – |
Table 3: Qualitative coding of terminal-competence observability and evaluation freshness. E=Explicit, T=Trace-visible, I=Incidental, and –=Not observable. Freshness concerns evaluation validity rather than competence.
| Benchmark | Tasks | Outcome | P1 Rule inv. fail. | P2 Feedback | P3 Env. exit | P4 State err. | P5 Final verify | P6 Recovery | P7 Gov. trigger |
| Terminal-Bench 2.1 [ 101 ] | 241 | 52.6% | 0.0% | 82.1% | 12.5% | 14.7% | 29.5% | 79.9% | 0.5% |
| SetupBench [ 7 ] | 93 | 59.1% | 0.0% | 80.2% | 9.0% | 19.5% | 36.6% | 85.5% | 1.4% |
| LongCLI-Bench [ 39 ] | 21 | 23.8% | 0.0% | 76.4% | 6.8% | 7.7% | 23.8% | 67.8% | 4.5% |
| BashArena [ 70 ] | 640 | 41.8% | 0.0% | 76.3% | 13.7% | 23.3% | 66.2% | 67.0% | 3.0% |
Table 4: Benchmark-exposure profile under mini-SWE-agent with DeepSeek-V4-Flash. Outcome is the official benchmark score. P1–P7 are bounded trace-derived process indicators. P1 and P3 record execution-friction signals; P2, P4, and P6 are auxiliary judge-assisted task-level rates; P5 records a detected final-window check; and P7 flags actions requiring contextual governance review.
| System | F | P | Time F/P | |
| Claw-SWE-Bench Lite | ||||
| mini-SWE-agent | 58.75% | 56.25% | 476/476 | |
| SWE-agent [ 164 ] | 76.25% | 77.50% | 675/712 | |
| OpenHands [ 150 ] | 68.75% | 67.50% | 600/653 | |
| SWE-bench Lite | ||||
| mini-SWE-agent | 55.67% | 55.67% | 294/389 | |
Table 5: Matched single-run results by benchmark and system. F/P denote DeepSeek-V4-Flash/Pro; entries report resolved rate, Pro-minus-Flash difference, and mean task time in seconds.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Description | Count |
| Corpus status | ||
| Research corpus | Research works retained for the survey synthesis. | 186 |
| Engineering practice, boundary, and disclosure references | Five engineering-practice or boundary entries and one disclosure citation. | 6 |
| Total cited set | Distinct cited entries after work-level version consolidation. | 192 |
| Release year of research entries | ||
| 2022–2023 | Earliest retained foundations for tool use and interactive execution. | 4 |
Table 6: Composition of the current review corpus. Release-year counts cover the 186 research entries; analytical-use counts are non-exclusive.
| Stage | Procedure | Recorded decision | Role in synthesis |
| Scope freeze | Fix the 2022 to August 2026 interval, terminal-mediated research object, adjacent comparators, and workload-level boundary tests | Date range, source classes, and inclusion boundary | Prevents surface-level CLI mentions from entering as core evidence |
| Retrieval | Apply keyword families, venue-oriented search, and backward and forward reference chaining | Source, execution date, query form when retained, and work identity | Combines named-field retrieval with architecture, acquisition, evaluation, and governance routes |
| Work reconciliation | Group versions of the same contribution and retain the most complete research account | Preferred version and version relationship | Avoids double counting preprint, conference, and journal releases |
| Screening | Apply the five questions below and assign a corpus disposition | Research, engineering-practice, boundary, or exclusion disposition | Separates progress-bearing terminal evidence from surface-level or adjacent evidence |
| Analytical coding | Assign manuscript roles to every retained entry and apply architecture, acquisition, or evaluation fields to the relevant subsets | Entry-level roles and subset-specific analytical codes | Connects retained evidence to the architecture, acquisition, and evaluation synthesis |
| Claim calibration | Assess convergence, directness, domain breadth, and evidence maturity at the passage level | Bounded claim strength and scope | Keeps synthesis claims proportional to the available evidence |
Table 7: Corpus-construction workflow. Each stage produces the input required by the next stage and constrains how evidence enters the synthesis.
| ID | Purpose | Boolean template |
| Q1 | Named field | ("terminal agent" OR "terminal agents" OR "CLI agent" OR "command-line agent") |
| Q2 | Execution loop | ("LLM agent" OR "language model agent") AND (terminal OR shell OR CLI OR "command line") AND (execution OR runtime OR feedback) |
| Q3 | SWE and harnesses | ("software engineering agent" OR "coding agent") AND (terminal OR shell OR command) AND (benchmark OR harness OR environment) |
| Q4 | Acquisition | ("agent trajectory" OR "executable trajectory" OR "environment interaction") AND (training OR post-training OR "reinforcement learning") AND (terminal OR CLI OR repository) |
| Q5 | Process evaluation | ("terminal benchmark" OR "CLI benchmark" OR "agent benchmark") AND (trace OR process OR recovery OR verification OR state) |
| Q6 | Governance | ("LLM agent" OR "coding agent" OR "computer-use agent") AND (shell OR terminal OR "code execution") AND (sandbox OR permission OR safety OR governance) |
Table 8: Boolean query families used for subsequent corpus updates. Each source-specific execution is logged with its date and hit count.
| Field group | Coded fields | Unit | Analytical use |
| Corpus identity | Bibliographic metadata, corpus status, and unique work identity | Work | Supports corpus identity, release-year counts, and work-level deduplication |
| Analytical role | Scope, architecture, acquisition, evaluation, empirical setting, or implications | Work, non-exclusive | Connects retained entries to the survey passages they support |
| Architecture | Interface granularity, operational coupling, autonomy regime, recovery and control style, planning strategy | System or artifact | Supports the layered architecture and recurring-pattern synthesis |
| Acquisition | Data source, trajectory source, supervision signal, filtering or relabeling, failure-data treatment, capability target | Dataset, environment, or pipeline | Supports comparison of executable data, learning, and adaptation routes |
| Evaluation | Task regime, harness assumptions, verification type, trace visibility, artifact availability, dimension coverage | Benchmark or protocol | Supports benchmark-family and competence-observability comparisons |
| Evidence calibration | Convergence, directness, domain breadth, and maturity of evidence for the passage claim | Synthesis passage | Bounds the strength and generality of each synthesized claim |
Table 9: Level-specific coding schema used in corpus construction and synthesis. Entry-level fields cover every retained reference; specialized fields apply to relevant evidence subsets or synthesis passages.
| Field | Value | Field | Value |
| Judge model | DeepSeek-V4-Flash | Prompt design | Separate prompts for feedback use, state tracking, and recovery |
| Temperature | 0.0 | Top- | 1.0 |
| Initial maximum output | 4,096 tokens | Local seed | 42, passed when supported |
| Response mode | One non-streaming JSON object | Retry policy | At most three retries; output cap doubles on reasoning-only truncation, up to 32,768 tokens |
| Eligibility filter | Confidence and not uncertain | Aggregation | Task-level semantic rates over eligible evidence |
Table 10: Configuration of the semantic episode judge used for formal P2, P4, and P6 labels.
Figure 7: Matched-system outcomes for all 12 benchmark–system–model cells. Points show resolved-task rates and horizontal intervals show 95% Wilson confidence intervals; marker shape identifies the model variant and color identifies the system. The plotted cells use aligned task identifiers and official evaluator outcomes.
| Benchmark | System | Both fail | Pro only | Flash only | Both pass | P F (pp) | 95% CI (pp) | Exact | Holm |
| Claw Lite | mini-SWE-agent | 28 | 5 | 7 | 40 | [ , 6.25] | 0.774 | 1.000 | |
| Claw Lite | SWE-agent | 16 | 3 | 2 | 59 | [ , 6.25] | 1.000 | 1.000 | |
| Claw Lite | OpenHands | 22 | 3 | 4 | 51 | [ , 5.00] | 1.000 | 1.000 | |
| SWE-bench Lite | mini-SWE-agent | 120 | 13 | 13 | 154 | [ , 3.33] | 1.000 | 1.000 | |
| SWE-bench Lite | SWE-agent | 112 | 12 | 13 | 163 | [ , 3.00] | 1.000 | 1.000 | |
| SWE-bench Lite | OpenHands | 133 | 12 | 12 | 143 | [ , 3.33] | 1.000 | 1.000 |
Table 11: Task-paired outcome transitions between Flash and Pro. “Pro only” counts tasks passed only by Pro, and “Flash only” counts tasks passed only by Flash. Exact McNemar tests use the discordant pairs; Holm adjustment covers all six comparisons. Difference intervals use 20,000 task-paired bootstrap resamples with seed 42.
| Benchmark | Model | System A | System B | A only | B only | A B (pp) | Exact | Holm |
| Claw Lite | Flash | mini-SWE-agent | SWE-agent | 3 | 17 | 0.0026 | 0.0232 | |
| Claw Lite | Flash | mini-SWE-agent | OpenHands | 8 | 16 | 0.1516 | 0.4033 | |
| Claw Lite | Flash | SWE-agent | OpenHands | 9 | 3 | 0.1460 | 0.4033 | |
| Claw Lite | Pro | mini-SWE-agent | SWE-agent | 2 | 19 | 0.0002 | 0.0022 | |
| Claw Lite | Pro | mini-SWE-agent | OpenHands | 5 | 14 | 0.0636 | 0.4033 | |
| Claw Lite | Pro | SWE-agent | OpenHands | 12 | 4 | 0.0768 | 0.4033 |
Table 12: Task-paired cross-system contrasts. “A only” and “B only” count discordant tasks resolved by one system. Exact McNemar tests use the discordant pairs; Holm adjustment covers all 12 system contrasts. The difference is A minus B in percentage points.
More in this topic
Terminal Agents
Terminal agents use command-line interfaces and filesystems to carry out multi-step tasks in software and other digital environments.
See all 31 papers