AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

96 papers in the last four weeks, up 30% on the four weeks before. 1.2% of all new papers.

Jul 6Week of Sep 21

Latest papers 1,013

All topics

TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
Jun 26, 2026Shoufa Chen, Luyuan Wang, Xuan Yang +7Computer-Use Agent Benchmarks Computer-Use Agents
Meta AI · Duke University · Stanford University
Triadic Werewolf: A Jester Role for Multi-Hop Theory of Mind in LLMs
Jun 26, 2026Avni MittalGame-Playing Agents Social Deduction Games
NormAct: A Benchmark for Hidden Social Norm Compliance in Embodied Planning
Jun 26, 2026Shiyun Zhao, Xinwei Song, Tianyu Guo +7AI Agent Benchmarks Multimodal Agents
1State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI) · 3ShanghaiTech University · 2China Academy of Information and Communications Technology
When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search
Jun 26, 2026Yiling Tao, Shihan Deng, Meiling Tao +3Agentic Search LLM Agent Evaluation
Hunyuan, Tencent · Shenzhen International Graduate School, Tsinghua University
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
Jun 25, 2026Aoyang Fang, Yifan Yang, Jin'ao Shang +7LLM Agent Evaluation Causal Attribution
The Chinese University of Hong Kong, Shenzhen · Lero · The Chinese University of Hong Kong
Qwen-Image-Agent: Bridging the Context Gap in Real-World Image Generation
Jun 25, 2026Zekai Zhang, Jiahao Li, Jie Zhang +18Conditional Image Generation T2I Generation
How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?
Jun 24, 2026David Akinpelu, Akintonde Abbas, Rereloluwa Alimi +1LLM Agent Evaluation AI Agent Benchmarks
Independent Researcher · Tume AI
RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments
Jun 24, 2026Babak Rahmani, Sebastian Dziadzio, Joschka Strüber +2Opponent Modeling LLM-Based Program Synthesis
Tübingen AI Center
Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
Jun 24, 2026Yang Tian, Zhengpeng Shi, Yu Zhou +1AI Agent Reliability AI Agent Benchmarks
School of AI, Shanghai Jiao Tong University1 · School of Cyber Science and Engineering, Southeast University2
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Jun 24, 2026Divake Kumar, Sina Tayebati, Devashri Naik +5Computer-Use Agent Benchmarks Uncertainty Quantification
University of Illinois Chicago · Intel Labs · Capital One AI Labs
SidConArena: An Environment Evaluating Agents in Open-Ended,Positive-Sum Bargaining Game
Jun 24, 2026Yeqi Feng, Yuxin Chen, Tianxing HeMulti-Agent Negotiation Game Theory
Institute for Interdisciplinary Information Sciences, Tsinghua University
BrainAgent: A Large Language Model-Driven Multi-Agent Framework for Autonomous Brain Signal Understanding
Jun 24, 2026Yangxuan Zhou, Sha Zhao, Jiquan Wang +2Multi-Agent LLM Systems Brain-Computer Interfaces
State Key Laboratory of Brain-machine Intelligence, Zhejiang University, Hangzhou, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · MOE Frontier Science Center for Brain Science and Brain-machine Integration, Zhejiang University, Hangzhou, China
EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?
Jun 23, 2026Zhiyuan Peng, Xin Yin, Chenhao Ying +5Computer-Use Agent Benchmarks LLM Agent Self-Improvement
1Shanghai Jiao Tong University · 2Zhejiang University · 3Hithink Research
MemAudit: Auditing Long-Term Agent Memory via Hidden User-State Recovery
Jun 23, 2026Enze Ma, Yufan Zhou, Wei-Chieh Huang +7Agent Memory AI Agent Benchmarks
University of Illinois Chicago · KU Leuven · UC San Diego
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Jun 23, 2026Yuru Wang, Lejun Cheng, Yuxin Zuo +14AI for Science Coding Agents
Horizon Research, Frontis.AI Tsinghua University
Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War
Jun 23, 2026Arnaud RicciStrategic Reasoning Risks in Language Models AI Agent Benchmarks
Independent researcher, Switzerland.
SP-Mind: An Autonomous Reasoning Agent for Spatial Proteomics Analysis
Jun 23, 2026Yucheng Yuan, Yuanfeng Ji, Zhongxiao Li +1AI Agent Benchmarks Tool-Using Agents
Department of Computer Science, Stanford University, Stanford, USA · Department of Radiation Oncology, Stanford University, Stanford, USA
MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models
Jun 23, 2026Jinru Ding, Chuchu Jiang, Lu Lu +12AI Agent Benchmarks Medical VLMs
1Shanghai Artificial Intelligence Laboratory
When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents
Jun 22, 2026Tianyu Ding, Juan Pablo De la Cruz WeinsteinTool-Use Evaluation Retrieval-Augmented Classification
Amazon Web Services
MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?
Jun 22, 2026Juyang Bai, Laixi ShiMulti-Agent LLM Systems Multi-Agent System Optimization
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA.
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jun 22, 2026Jincheng Zhong, Weizhi Wang, Che Jiang +5Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents
Horizon Research, Frontis.AI
EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning
Jun 22, 2026Yitong Qiao, Lei Liu, Yue Shen +4AI Agent Benchmarks Clinical Reasoning
1Zhejiang University · †Work done during an internship at Ant Group. · 2Ant Group
Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation
Jun 22, 2026Julia Belikova, Rauf Parchiev, Evgeny Egorov +4LLM Agent Memory LLM Agent Evaluation
IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO
Jun 22, 2026Mostapha BenhendaIn-Context Retrieval LLM-as-a-Judge
StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs
Jun 22, 2026Youxin Zhu, Yixuan Ding, Peng Lai +3LLM Evaluation AI Agent Benchmarks
Southern University of Science and Technology · The University of Hong Kong · Alibaba Group +1
ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents
Jun 22, 2026Yincheng Zhou, Athena Zhuoming Zhong, Shijie Zhang +3Distribution Shift Robustness Computer-Use Agent Benchmarks
University of Pennsylvania · 2Peking University
Agent-as-a-Router: Agentic Model Routing for Coding Tasks
Jun 22, 2026Pengfei Zhou, Zhiwei Tang, Yixing Ma +8AI Coding Agents AI Agent Benchmarks
1National University of Singapore · 2DAMO Academy, Alibaba Group · 3Hupan Lab +3
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
Jun 22, 2026Jeffrey FlyntAI Agent Reliability Tool-Use Evaluation
Independent Researcher
RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents
Jun 21, 2026Meher Bhaskar Madiraju, Meher Sai Preetam MadirajuAI Agent Reliability Coding Agents
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Jun 21, 2026Yikun Fu, Bowen Fu, Zhenyu Wu +10Computer-Use Agent Benchmarks Computer-Use Agents
1Shanghai Jiao Tong University · 2Shanghai AI Laboratory · 3Xi’an Jiaotong University +5

Jun 26, 2026, cs.SEJ/K move · Enter open · S save

TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents

Shoufa Chen, Luyuan Wang, Xuan Yang, Zhiheng Liu+6

Meta AI · Duke University · Stanford University

Computer-Use Agent Benchmarks Computer-Use Agents Terminal Agents LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.

Momentum

Related topics

Latest papers 1,013