RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Yunqi Liu; Peiyu Wang; Bingli Wang; Leo Luo; Xuanwu Song; Xudong Ma; Hongxin Li; Yaxin Du

cs.CLOct 15, 2026

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Authors: Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du,

Organizations: IQuest Research · Beihang University · Renmin University

Abstract

Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score ( $J_{\mathrm{collab}}$ ), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.

Figures & tables

Figure 1: A COLLAB episode in REALCLI. Top: a user script releases the specification in slices and schedules probes. Middle: the agent guards a planted secret but drops a required feature on an unverified claim. Bottom: the verifier scores task outcome; checklists score collaboration.

Benchmark	Ver.	User	Pair	Und	Ste	Obs	Tru	Saf
SWE-bench ( Jimenez et al., 2023 )	✓	✗	✗	✗	✗	✗	✗	✗
Ambig-SWE ( Vijayvargiya et al., 2025 )	✓	✓	✓	✗	✓	✗	✗	✗
Terminal-Bench ( Merrill et al., 2026 )	✓	✗	✗	✗	✗	(✓)	(✓)	✗
LongCLI-Bench ( Feng et al., 2026 )	✓	(✓)	(✓)	✗	(✓)	(✓)	✗	✗
Dialogue SWE-Bench ( King and Flanigan, 2026 )	✓	✓	✗	(✓)	✓	✗	✗	✗
SWE-Together ( Wu et al., 2026 )	✓	✓	✗	✗	(✓)	✗	✗	✗

Table 1: Comparison of RealCLI with representative CLI-agent benchmarks across environment-level verification, runtime user participation, paired evaluation, and five collaboration dimensions. Marks reflect each benchmark’s stated evaluation scope. Ver. : environment-level verification. User : runtime user participation. Pair : paired evaluation of autonomous and user-interactive modes on matched task instances. Und : understandability. Ste : steerability. Obs : observation and recovery. Tru : trust calibration. Saf : safety boundary management. Legend: ✓ explicitly evaluated, (✓) partially or secondarily evaluated, and ✗ not evaluated.

Dimension	Evaluation focus	Sub. / Tpl.
U nderstandability (U)	Legibility of plans, execution state, and outcomes	4 / 12
S teerability (S)	User intervention, redirection, and constraint persistence	5 / 11
O bservation & R ecovery (OR)	State verification and recovery from failures	3 / 9
T rust C alibration (TC)	Accurate disclosure of risks, limits, and task status	5 / 12
S afety & R isk Awareness (SR)	User control over high-impact or unauthorized actions	5 / 13

Table 2: Overview of collaboration dimensions (Sub.: subdimensions, Tpl.: scenario templates).

Figure 2: Construction pipeline of RealCLI .

Figure 3: Statistics dashboard of RealCLI . Fig. (a) shows the task categories and their subdomains. Fig. (b), (c) show the distributions of instruction length and of itemized requirements per task, and Fig. (d) the number of distinct file paths cited per instruction. Fig. (e) shows how the task verifiers are realized. Fig. (f) shows the task coverage of each subdimension of our collaboration taxonomy, and Fig. (g) the distribution of tasks over the scenarios. Fig. (h) shows the evidence source of the checklist items, and Fig. (i) the per-task spread of the four interaction item kinds.

Closed-source models
Model	Claude Code			Terminus-2			Codex			Avg.
	$R_{\mathrm{exec}}$	$R_{\mathrm{collab}}$	$J_{\mathrm{collab}}$	$R_{\mathrm{exec}}$	$R_{\mathrm{collab}}$	$J_{\mathrm{collab}}$	$R_{\mathrm{exec}}$	$R_{\mathrm{collab}}$	$J_{\mathrm{collab}}$
Claude Opus 5	58.2	58.4 +0.2	27.4 -30.8	65.4	62.4 -3.0	37.6 -27.9	46.5	55.5 +9.0	31.5 -15.0	49.2
Claude Opus 4.8	54.5	53.5 -1.0	18.3 -36.3	52.5	53.5 +1.0	21.1 -31.4	40.6	50.0 +9.4	23.0 -17.6	40.8
GPT-5.6 Sol	44.6	45.5 +0.9	6.7 -37.9	51.5	47.5 -4.0	9.1 -42.4	58.5	49.5 -9.0	8.5 -50.0	35.7
Open-source models

Table 3: RealCLI main results on the held-out split across three agent scaffolds. Subscripts on

R_{collab}

and

J_{collab}

indicate performance changes relative to the

R_{exec}

baseline (green positive, red negative). Bold and underline denote the highest and second-highest scores, respectively.

Closed-source models
Model	Claude Code			Terminus-2			Codex			Avg.
	$S_{\mathrm{col.}}$	$J_{\mathrm{col.}}$	U $\cdot$ S $\cdot$ OR $\cdot$ TC $\cdot$ SR	$S_{\mathrm{col.}}$	$J_{\mathrm{col.}}$	U $\cdot$ S $\cdot$ OR $\cdot$ TC $\cdot$ SR	$S_{\mathrm{col.}}$	$J_{\mathrm{col.}}$	U $\cdot$ S $\cdot$ OR $\cdot$ TC $\cdot$ SR
Claude Opus 5	46.9	27.4	50 $\cdot$ 20 $\cdot$ 78 $\cdot$ 43 $\cdot$ 43	60.2	37.6	85 $\cdot$ 40 $\cdot$ 89 $\cdot$ 43 $\cdot$ 43	56.8	31.5	60 $\cdot$ 30 $\cdot$ 100 $\cdot$ 33 $\cdot$ 60	32.1
Claude Opus 4.8	34.1	18.3	56 $\cdot$ 10 $\cdot$ 44 $\cdot$ 23 $\cdot$ 37	39.4	21.1	42 $\cdot$ 30 $\cdot$ 56 $\cdot$ 23 $\cdot$ 47	43.4	23.0	63 $\cdot$ 30 $\cdot$ 44 $\cdot$ 43 $\cdot$ 37	20.8
GPT-5.6 Sol	14.7	6.7	29 $\cdot$ 10 $\cdot$ 11 $\cdot$ 13 $\cdot$ 10	19.1	9.1	48 $\cdot$ 30 $\cdot$ 11 $\cdot$ 7 $\cdot$ 0	17.1	8.5	48 $\cdot$ 20 $\cdot$ 11 $\cdot$ 7 $\cdot$ 0	8.1
Open-source models

Table 4: RealCLI -COLLAB multi-dimensional results on the held-out split across three agent scaffolds. Scores are reported per scaffold in U

\cdot

\cdot

\cdot

\cdot

SR order (darker = higher). Bold and underline denote the highest and second-highest scores, respectively.

Figure 4: Pass rate changes by category. Hollow/filled: non-interactive/interactive. Blue/orange: gains/losses (

\Delta\mathrm{pp}

, percentage points).

Figure 5: Collaboration across five dimensions and 22 subdimensions. Colors indicate scenario excellence rates.

Figure 6: Failure patterns. (a) Execution issues among failed runs (categories overlap). (b) Collaboration failure rates by task outcome over triggered, applicable scenarios.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Subdimension	Scenarios	Tasks
Understandability	Plan transparency	2	25
	Feedback granularity and timing	3	34
	Error presentation	2	35
	Operation economy	1	17
Steerability	Interruptibility	2	19
	Clarification behavior	2	22

Appendix

Table 5: Collaboration dimensions, subdimensions, evaluation scenarios, and task coverage.

Type	Scenario Condition & Expected Behavior
Dimension 1: Understandability
1.1 Plan transparency
Probe	Hidden complexity: User understates task complexity. Expected: Agent discovers hidden couplings and explicitly updates its plan before acting.
Probe	Execution deviation: Plan fails due to errors/limits. Expected: Agent updates the plan and informs the user.
Passive	Trivial task: The task is simple and unambiguous. Expected: Agent avoids generating a lengthy, disproportionate plan.
1.2 Feedback granularity and timing

Appendix

Table 6: Detailed taxonomy of dimensions, subdimensions, and scenarios.

Category	Sub.	Representative Task Scope	Subdomain	Tasks
System Operations	9	Files, processes and services, environments and dependencies, containers, networks, cloud resources, permissions, cleanup, and scheduling	Cloud Resource Management	1
			Container Orchestration	2
			Environment & Dependencies	2
			File Management	2
			Network Service Development	1
			Permission Management	1

Appendix

Table 7: Task taxonomy and distribution across categories and subdomains.

Terminal-Agent Benchmarks

See all 10 papers

AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents
Jun 19, 2026Han Chi, Jiaxin Qi, Yan Cui +2LLM Agent Evaluation AI Agent Benchmarks
Computer Network Information Center, Chinese Academy of Sciences · Hangzhou Institute for Advanced Study, Chinese Academy of Sciences
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14LLM Agent Harnesses AI Agent Benchmarks
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8
CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks
Apr 20, 2026Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3AI Agent Evaluation Human-in-the-Loop Evaluation
Stanford University · Carnegie Mellon University · University of Rochester +1

Jun 19, 2026J/K move · Enter open · S save

AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents

Han Chi, Jiaxin Qi, Yan Cui, Baisheng Lai+1

Computer Network Information Center, Chinese Academy of Sciences · Hangzhou Institute for Advanced Study, Chinese Academy of Sciences

LLM Agent Evaluation AI Agent Benchmarks Tool-Using Agents LLM Agent Harnesses

Open paper AskPDF arXiv

LLM agents increasingly solve local tasks through command-line and CLI-based harness interfaces, including code editing, repository inspection, data analysis, and file workflows. Existing evaluations often emphasize task success, but deployed local agents are not models alone: the CLI mediates prompts, context replay, tool outputs, file access, terminal observations, and stopping behavior. As a result, the same model can produce different success, token, and cost profiles under different CLIs. We introduce AGENTMETER, a benchmark for evaluating model-CLI matching in CLI-mediated local task-solving agents, together with AgentMeter Score (AMS), a success-anchored, cost-aware metric over calibrated task-effort tiers. AgentMeter uses Benchmark90 as the full validation set and Core30 as a lower-cost subset for expanded comparison across 24 complete model-CLI configurations. On Core30, common deployment criteria select different configurations: highest Pass/30 selects GLM-5.1 with qwen-coder, lowest Tok./Pass selects GPT-5.3-Codex with kimi-cli, lowest billable USD/Pass selects Qwen3.6+ with Codex, while highest AMS selects Qwen3.6+ with kimi-cli. Benchmark90 validation preserves the Top-1 configuration and Top-3 set, with Spearman correlation 0.765, Kendall correlation 0.567, and AMS MAE 0.0383. These results show that model choice and CLI choice should not be decoupled, and that model-CLI configurations should be evaluated as the deployed unit.

Abstract

Terminal-Agent Benchmarks

Explore similar work

AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks