Terminal-Agent Benchmarks

Latest papers

10 papers2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
Oct 15, 2026Yunqi Liu, Peiyu Wang, Bingli Wang +21Terminal-Agent Benchmarks LLM Agent Evaluation
IQuest Research · Beihang University · Renmin University
Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress
Oct 9, 2026Zongxia Li, Zhongzhi Li, Yucheng Shi +14Process-Based Agent Evaluation Terminal-Agent Benchmarks
Tencent HY LLM Frontier · University of Maryland, College Park · University of Georgia +7
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Jul 9, 2026Zongxia Li, Zhongzhi Li, Yucheng Shi +12Terminal-Agent Benchmarks Long-Horizon Agent Evaluation
Tencent HY LLM Frontier · University of Maryland, College Park · University of Georgia +5
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
Jun 26, 2026Shoufa Chen, Luyuan Wang, Xuan Yang +7Computer-Use Agent Benchmarks Computer-Use Agents
Meta AI · Duke University · Stanford University
unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning
May 27, 2026Geoffrey Bradway, Roger Creus Castanyer, Lorenz Wolf +3Terminal-Agent Benchmarks AI Agent Benchmarks
Vmax
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Computer-Use Agent Benchmarks Terminal-Agent Benchmarks
University College London · Nanjing University · Tencent
ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
May 13, 2026Yuxiang Lai, Peng Xia, Haonian Ji +8Computer-Use Agent Benchmarks Terminal-Agent Benchmarks
University of North Carolina at Chapel Hill · Stanford University · University of Southern California +2
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14LLM Agent Harnesses AI Agent Benchmarks
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8
Learning CLI Agents with Structured Action Credit under Selective Observation
May 8, 2026Haoyang Su, Ying WenTerminal-Agent Benchmarks Computer-Use Agents
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking

Oct 15, 2026J/K move · Enter open · S save

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo+20

IQuest Research · Beihang University · Renmin University

Terminal-Agent Benchmarks LLM Agent Evaluation AI Agent Benchmarks Human-AI Collaboration Agent Benchmarks Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score (

J_{\mathrm{collab}}

), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.