Terminal Agents Suffice for Enterprise Automation

Patrice Bechard; Orlando Marquez Ayala; Emily Chen; Jordan Skelton; Sagar Davasam; Srinivas Sunkara; Vikas Yadav; Sai Rajeswar

cs.SEMar 31, 2026

Terminal Agents Suffice for Enterprise Automation

Authors: Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, Sai Rajeswar

Organizations: ServiceNow · Mila – Quebec AI Institute · Université de Montréal

Abstract

There has been growing interest in building agents that can interact with digital platforms to execute meaningful enterprise tasks autonomously. Among the approaches explored are tool-augmented agents built on abstractions such as Model Context Protocol (MCP) and web agents that operate through graphical interfaces. Yet, it remains unclear whether such complex agentic systems are necessary given their cost and operational overhead. We argue that a coding agent equipped only with a terminal and a filesystem can solve many enterprise tasks more effectively by interacting directly with platform APIs. We evaluate this hypothesis across diverse real-world systems and show that these low-level terminal agents match or outperform more complex agent architectures at a fraction of the cost. Our findings suggest that simple, flexible programmatic interfaces combined with strong foundation models should be the backbone of enterprise automation.

Figures & tables

Figure 1: Execution traces of agents ordering an iPad Pro. (Top-left) The MCP agent identifies the catalog item but cannot proceed without an ordering tool. It falls back to creating a support ticket and fails. (Top-right) The web agent reaches the catalog page but becomes confused within the iframe-based UI, leading to a long, costly trajectory that fails to complete the order. (Bottom) The terminal agent encounters JSON quoting errors when constructing the request payload and 404 responses from incorrect API endpoints, but recovers by writing the payload to a temporary file and exploring alternative endpoints. It completes the task at an order of magnitude lower cost than the web agent, demonstrating the flexibility, resilience, and efficiency of direct terminal-based interaction.

Figure 2: Overview of StarShell: a minimal terminal agent for enterprise automation. The agent operates through a terminal and filesystem, optionally using documentation and persistent skills to discover and invoke APIs directly on enterprise platforms (e.g., GitLab, ServiceNow, ERPNext), without relying on GUI interaction or pre-defined tool registries.

	ServiceNow	GitLab	ERPNext
Samples	330	192	207
MCP Tools	83	81	7
Doc. Pages	61k	2.65k	5.41k

Table 1: Summary statistics of the evaluation benchmark across three enterprise platforms.

	ServiceNow		GitLab		ERPNext		Overall
Agent	(330)		(192)		(207)		(729)
	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)
Claude Sonnet 4.6
Tool-use	11.5	0.76	45.2	0.48	55.6	0.14	32.9	0.51
Web	72.4	4.49	82.9	0.88	61.8	3.63	72.2	3.29
Terminal	73.6	0.78	76.5	0.28	67.6	0.46	72.7	0.56

Table 2: Main results across agent interaction paradigms . Success rate (SR,

\uparrow

) and average cost per task (

\downarrow

) for tool-use, web, and terminal agents on three enterprise platforms with four backbone LLMs. Bold indicates the best SR and lowest cost within each platform–model group, along with any score within one standard error of the best.

Figure 3: Skills accumulation over sequential tasks. Top: cumulative number of successful tasks. Middle: cumulative cost ($USD). Bottom: skills directory size (KB). The agent with memory (blue) accumulates reusable procedures; the baseline (black) starts fresh every time.

	ServiceNow		GitLab		ERPNext		Overall
Agent	(330)		(192)		(207)		(729)
	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)
Claude Sonnet 4.6
Web	72.4	4.49	82.9	0.88	61.8	3.63	72.2	3.29
Terminal	73.6	0.78	76.5	0.28	67.6	0.46	72.7	0.56
Hybrid	72.1	3.02	77.7	0.64	66.7	1.44	72.0	1.94

Table 3: Hybrid agent comparison. Success rate and cost per task for web, terminal, and hybrid agents on three enterprise platforms. The hybrid agent has access to both a terminal and a Playwright-controlled browser. Bold indicates the best SR and lowest cost within each platform–model group, along with any score within one standard error of the best.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

	ServiceNow		GitLab		ERPNext		Overall
Agent	(120)		(127)		(197)		(444)
	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)
Claude Sonnet 4.6
Tool-use	24.2	1.30	66.9	0.52	58.4	0.14	51.6	0.56
Web	66.7	4.57	86.4	0.91	60.7	3.82	69.6	3.20
Terminal	75.8	0.49	80.2	0.44	66.0	0.48	72.7	0.47

Appendix

Table 4: Results on the feasible task subset for tool-use agents. Success rate (SR,

\uparrow

) and average cost per task (

\downarrow

) restricted to task types that are feasible for all three agent paradigms. Bold indicates the best SR and lowest cost within each platform–model group, along with any score within one standard error of the best. ServiceNow: 12 task types (120 instances), GitLab: 15 task types (127 instances), ERPNext: 78 task types (197 instances).

	ServiceNow		GitLab		ERPNext		Overall
Agent	(330)		(192)		(207)		(729)
	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)
Claude Sonnet 4.6
Tool-use	11.5	0.76	45.2	0.48	55.6	0.14	32.9	0.51
Web	72.4	4.49	82.9	0.88	61.8	3.63	72.2	3.29
Terminal	73.6	0.78	76.5	0.28	67.6	0.46	72.7	0.56

Appendix

Table 5: Isolating tool granularity from terminal access. Success rate (SR,

\uparrow

) and average cost per task (

\downarrow

) for tool-use, web, terminal, and API-call agents. The API-call agent uses a single generic, authenticated HTTP tool with no filesystem or shell access. Tool-use, web, and terminal rows are reproduced from Table 2 . Bold indicates the best SR and lowest cost within each platform–model group, along with any score within one standard error of the best.

	ServiceNow		GitLab		ERPNext		Overall
Agent	(330)		(192)		(207)		(729)
	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)
Gemma-4-31B-it
Tool-use	9.4	0.005	39.9	0.012	30.4	0.007	23.4	0.007
Web	20.3	0.148	35.6	0.235	22.7	0.247	25.0	0.199
Terminal	48.5	0.024	52.1	0.016	36.7	0.091	46.1	0.041

Appendix

Table 6: Results with open-weight models. Success rate (SR,

\uparrow

) and average cost per task (

\downarrow

) for tool-use, web, and terminal agents using Gemma-4-31B-it and Qwen-3.6-27B, alongside Gemini 3.1 Pro (the most cost-efficient frontier model in Table 2 ) as a reference point. Overall results are weighted by the number of tasks in each platform. Costs for the two open-weight models are reported to three decimal places, since they are an order of magnitude smaller than frontier-model costs and two decimals would collapse distinct values; the Gemini 3.1 Pro reference rows keep the two-decimal precision of Table 2 . Bold indicates the best SR and lowest cost within each platform–model group, along with any score within one standard error of the best.

	ServiceNow		GitLab		ERPNext		Overall
Agent	(330)		(192)		(207)		(729)
	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)
Claude Sonnet 4.6
No Docs	73.6	0.78	76.5	0.28	67.6	0.46	72.7	0.56
With Docs	67.3	1.10	79.1	0.48	72.3	0.47	71.8	0.76
Claude Opus 4.6

Appendix

Table 7: Effect of documentation access on terminal agents. Success rate and cost per task for terminal agents with and without access to official documentation. Bold indicates the best SR and lowest cost within each platform–model group, along with any score within one standard error of the best.

	ServiceNow		GitLab		ERPNext		Overall
Agent	(330)		(192)		(207)		(729)
	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)	SR (%)	Cost ($)
Claude Sonnet 4.6
Single Agent	73.6	0.78	76.5	0.28	67.6	0.46	72.7	0.56
Planner/Executor	71.5	1.13	80.8	0.36	74.4	0.68	74.8	0.80
Claude Opus 4.6

Appendix

Table 8: Single agent vs. planner-executor multi-agent system. Success rate and cost per task for a single terminal agent compared to a planner-executor design where both agents share the same model and tool access. Bold indicates the best SR and lowest cost within each platform–model group, along with any score within one standard error of the best.

			Success rate (%)			Tokens per task (K)
Category	n	p	Tool-use	Web	Term.	Tool-use	Web	Term.
ServiceNow
Form Creation	50	$<$ 0.001	2.0	67.0	89.0	59	513	42
Dashboard	40	0.302	11.9	36.2	39.1	107	599	186
List Filtering	60	0.072	20.8	70.0	76.7	50	397	34
List Sorting	60	0.003	35.8	62.5	75.4	87	174	249

Appendix

Table 9: Success rate and token cost by task category. Unweighted means over the four backbones of Table 2 .

p

is a paired exact McNemar test of terminal against web over the tasks solved by exactly one of the two, and bold marks the categories it separates at

p<0.05

, together with the paradigm that wins there; every other row is a tie. Tokens are input plus output per task, in thousands, and include failed attempts and tasks that hit the turn limit, so a paradigm that fails expensively is charged for it.

	pass^ $k$ (all $k$ instances succeed)				pass@ $k$
Agent	$k{=}1$	$k{=}3$	$k{=}5$	$k{=}10$	$k{=}3$	$k{=}5$	$k{=}10$
Claude Sonnet 4.6
Terminal	73.3	43.9	27.6	9.1	94.8	96.7	97.0
Web	72.7	44.5	30.9	15.6	95.7	99.3	100.0
Tool-use	11.8	2.0	0.7	0.0	25.1	32.8	42.4
Claude Opus 4.6

Appendix

Table 10: Consistency across instances of a task type , averaged over the ServiceNow task types. pass^

k

is the probability that all

k

instances drawn from a task type succeed, pass@

k

that at least one does, and

k{=}1

is the success rate. Instances vary records and field values within a fixed operation, so these numbers measure robustness across instances rather than run-to-run determinism.

Category	Description	Example output
Success categories
Success	Valid JSON response with expected data.	{"result": {"sys_id": "a1b2...", "number": "INC001"}}
Success (trunc.)	Valid response cut off by output length limit.	{"result": [{"sys_id": ...} ...] [OUTPUT TRUNCATED]
Non-JSON success	Output piped through jq , python3 , or awk ; non-JSON but correct.	number: INC0000039 state: 6
Failure categories
API error	Server returned a JSON error response (invalid payload, bad field, wrong HTTP method).	{"error": {"detail": "The payload is not valid JSON."}, "status": "failure"}

Appendix

Table 11: Classification of tool call outcomes observed in evaluation traces. The agent interacts with web platforms through curl commands, whose outputs are categorized into success or failure types based on the response content.

	ServiceNow				GitLab				ERPNext				Overall
Agent	(330)				(192)				(207)				(729)
	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost
Claude Sonnet 4.6
Tool-use	11.5	4.3	35	0.76	45.2	4.0	22	0.48	55.6	5.7	17	0.14	32.9	4.6	26	0.51
Web	72.4	26.0	153	4.49	82.9	10.4	55	0.88	61.8	19.9	138	3.63	72.2	20.2	123	3.29
Terminal	73.6	10.5	67	0.78	76.5	8.8	42	0.28	67.6	13.0	42	0.46	72.7	10.8	53	0.56

Appendix

Table 12: Experiment results of StarShell on enterprise benchmarks. Success rate (SR

\uparrow

), average tool calls (Tools), wall-clock time in seconds (Time), and cost per task in USD (Cost

\downarrow

) are reported for tool-use, web, and terminal agents on 3 platforms for 4 backbone LLMs. Bold indicates the best value for each metric within a model group, along with any SR within one standard error of the best.

	ServiceNow				GitLab				ERPNext				Overall
Agent	(330)				(192)				(207)				(729)
	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost
Claude Sonnet 4.6
No Docs	73.6	10.5	67	0.78	76.5	8.8	42	0.28	67.6	13.0	42	0.46	72.7	10.8	53	0.56
With Docs	67.3	18.0	94	1.10	79.1	12.2	45	0.48	72.3	15.3	52	0.47	71.8	15.7	69	0.76
Claude Opus 4.6

Appendix

Table 13: Effect of documentation access on terminal agents. Success rate (SR

\uparrow

), average tool calls (Tools), wall-clock time in seconds (Time), and cost per task in USD (Cost

\downarrow

) are reported for terminal agents with and without access to official platform documentation. Bold indicates the best value for each metric within a model group, along with any SR within one standard error of the best.

	ServiceNow				GitLab				ERPNext				Overall
Agent	(330)				(192)				(207)				(729)
	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost
Claude Sonnet 4.6
No Skills	73.6	10.5	67	0.78	76.5	8.8	42	0.28	67.6	13.0	42	0.46	72.7	10.8	53	0.56
With Skills	77.0	8.7	67	0.44	78.1	9.9	50	0.31	72.9	11.7	62	0.35	76.1	9.9	61	0.38

Appendix

Table 14: Effect of the ability to create and reuse skills on terminal agents. Success rate (SR

\uparrow

), average tool calls (Tools), wall-clock time in seconds (Time), and cost per task in USD (Cost

\downarrow

) are reported for terminal agents with and without access to self-generated skills. Bold indicates the best value for each metric within a model group, along with any SR within one standard error of the best.

	ServiceNow				GitLab				ERPNext				Overall
Agent	(330)				(192)				(207)				(729)
	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost
Claude Sonnet 4.6
Single Agent	73.6	10.5	67	0.78	76.5	8.8	42	0.28	67.6	13.0	42	0.46	72.7	10.8	53	0.56
Planner/Executor	71.5	16.6	103	1.13	80.8	10.9	54	0.36	74.4	20.9	109	0.68	74.8	16.3	92	0.80
Claude Opus 4.6

Appendix

Table 15: Single agent vs. planner-executor multi-agent system. Success rate (SR

\uparrow

), average tool calls (Tools), wall-clock time in seconds (Time), and cost per task in USD (Cost

\downarrow

) are reported for a single terminal agent compared to a planner-executor design where both agents share the same model and tool access. Bold indicates the best value for each metric within a model group, along with any SR within one standard error of the best.

	ServiceNow				GitLab				ERPNext				Overall
Agent	(330)				(192)				(207)				(729)
	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost	SR	Tools	Time	Cost
Claude Sonnet 4.6
Web	72.4	26.0	153	4.49	82.9	10.4	55	0.88	61.8	19.9	138	3.63	72.2	20.2	123	3.29
Terminal	73.6	10.5	67	0.78	76.5	8.8	42	0.28	67.6	13.0	42	0.46	72.7	10.8	53	0.56
Hybrid	72.1	21.0	127	3.02	77.7	10.5	49	0.64	66.7	16.5	89	1.44	72.0	17.0	96	1.94

Appendix

Table 16: Hybrid agent comparison. Success rate (SR

\uparrow

), average tool calls (Tools), wall-clock time in seconds (Time), and cost per task in USD (Cost

\downarrow

) are reported for a hybrid agent using the Playwright MCP and a terminal, compared against a simple terminal agent and an agent equipped solely with the Playwright MCP tools. Bold indicates the best value for each metric within a model group, along with any SR within one standard error of the best.

Category	Task Name	# Tasks
Form Creation	create-change-request	10
	create-hardware-asset	10
	create-incident	10
	create-problem	10
	create-user	10
Dashboard	single-chart-value-retrieval	10

Appendix

Table 17: ServiceNow tasks

Category	Task Name	# Tasks
User Management	assign-role	2
	create-user-email-first-last	2
	create-user-email-first-last-username-phone	3
	create-user-group-with-new-users	3
	create-user-with-email-account-dependency	3
	disable-user-account	2

Appendix

Table 19: ERPNext Tasks

Category	Tool Name		# Tools
Incident Management	create_incident	resolve_incident	6
	update_incident	list_incidents
	add_comment	get_incident_by_number
Service Catalog	list_catalog_items	move_catalog_items	11
	get_catalog_item	get_optimization_recommendations
	update_catalog_item	create_catalog_item_variable

Appendix

Table 20: Tools advertised by the ServiceNow MCP server (83 tools). The server’s tool_packages.yaml lists 93 tools in its default full package, and its startup log reports that figure, but 11 of the 93 have no implementation and are silently omitted from the ListTools response; list_tool_packages is added by the server itself. We report the 83 the model can actually invoke.

Category	Tool Name		# Tools
Authentication	authenticate_erpnext		1
Document Management	get_documents	update_document	3
Document Management	create_document		3
Schema	get_doctypes	get_doctype_fields	2
Reporting	run_report		1
Total			7

Appendix

Table 21: Overview of tools available in the ERPNext MCP server.

Category	Tool Name		# Tools
Merge Requests	create_merge_request	update_merge_request_note	31
	get_merge_request	delete_merge_request_note
	update_merge_request	create_merge_request_thread
	merge_merge_request	resolve_merge_request_thread
	list_merge_requests	mr_discussions
	get_merge_request_diffs	create_merge_request_discussion_note

Appendix

Table 22: Tools advertised by the GitLab MCP server (81 tools). The server additionally implements wiki, milestone and pipeline tool groups, which it exposes only when the USE_GITLAB_WIKI , USE_MILESTONE and USE_PIPELINE environment variables are set; we leave them at their defaults, so those 26 tools were unavailable to the agent and appear in no recorded trajectory.

Category	Tool Name	# Tools
Core Automation	browser_click	20
	browser_close
	browser_console_messages
	browser_drag
	browser_evaluate
	browser_file_upload

Appendix

Table 23: Tools advertised by the Playwright MCP server 6 6 6 https://github.com/microsoft/playwright-mcp , v0.0.68, the release current throughout our evaluation window. in the configuration used by our web and hybrid agents (21 tools). The server additionally implements vision (coordinate-level mouse), PDF, and tracing tool groups, which are gated behind opt-in capability flags that we leave disabled; browser_install is advertised but blocked by our harness, since browsers are pre-installed in the container. None of these tools were available to the agent, and none appear in any recorded trajectory.

Figure 6: User interface used to inspect agent traces.

Terminal Agents

Terminal agents use command-line interfaces and filesystems to carry out multi-step tasks in software and other digital environments.

See all 31 papers

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Computer-Use Agent Benchmarks Terminal-Agent Benchmarks
University College London · Nanjing University · Tencent
Applying Anthropic Primitives at Large Enterprises: Harness Paradigm for Knowledge Work
Aug 20, 2026George Juraj SalapaLLM Agent Harnesses LLM Agent Orchestration
G.S. s.r.o., PwC Austria
Terminal Agents: A Survey of AI Agents in Command-Line Environments
Aug 20, 2026Yi Bin, Xiaoyang Yuan, Haoxi Zeng +9Terminal Agents Tool-Using Agents
Tongji University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China +1

May 21, 2026J/K move · Enter open · S save

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks

Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou+7

University College London · Nanjing University · Tencent

Computer-Use Agent Benchmarks Terminal-Agent Benchmarks Terminal Agents LLM Agent Evaluation Long-Horizon Agent Evaluation AI Agent Benchmarks+2

Open paper AskPDF arXiv

We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 representative, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson r=0.20). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve. Data and code are available at https://github.com/EuniAI/TerminalWorld.

Abstract

Terminal Agents

Explore similar work

TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks

Applying Anthropic Primitives at Large Enterprises: Harness Paradigm for Knowledge Work

Terminal Agents: A Survey of AI Agents in Command-Line Environments