AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

146 papers in the last four weeks, up 161% on the four weeks before. 1.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
Jun 2, 2026Joel Sol, Homayoun NajjaranMulti-Agent Coordination LLM Agent Evaluation
Faculty of Engineering and Computer Science University of Victoria Victoria, BC, Canada
Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning
Jun 2, 2026Eric Cho, Shawn Huang, Alice Lu +1Quantitative Finance AI Agent Evaluation
1Trata · 2Brigham Young University · 3Osmosis
The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol
Jun 2, 2026Jai Lal Lulla, Matthias Galster, Jie M. Zhang +2Software Engineering AI Coding Agents
Singapore Management University, Singapore · University of Bamberg, Germany · King’s College London, United Kingdom +1
RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
Jun 2, 2026Zongwei Lv, Zhewen Tan, Yaoming Li +7Computer-Use Agent Benchmarks AI Coding Agents
1Peking University · 2Qiyuan Tech · 2Qiyuan Tech*
Multi $^2$ : Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments
Jun 2, 2026Sangeun Park, Minhae KwonMulti-Agent LLM Systems Hierarchical RL
Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea.
SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystems
Jun 2, 2026Linyue Pan, Yaoming Zhu, Lin Qiu +2Multi-Agent Collaboration AI Agent Benchmarks
Tsinghua University, China · Meituan, China
What Makes Interaction Trajectories Effective for Training Terminal Agents?
Jun 2, 2026Sidi Yang, Chaofan Tao, Jierun Chen +11AI Agent Benchmarks LLM Agent Training
1The University of Hong Kong · 2Huawei Technologies · 3Nanyang Technological University
MeDxAgent: Multi-Agent Consultation for Interactive Medical Diagnosis
Jun 2, 2026Akshat Sanghvi, Naren Akash, Raza Imam +2Multi-Agent LLM Systems Medical Diagnosis
Microsoft Research India · Institute of AI and Innovation, AIG Hospitals · Mohamed bin Zayed University of Artificial Intelligence
MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents
Jun 2, 2026Jia Yu, Zilong Wang, Xinyang Jiang +2Computer-Use Agent Benchmarks Computer-Use Agents
Microsoft Research Asia, Shanghai, China · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China · Shanghai Key Laboratory of MICCAI, Shanghai, China
Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigation
Jun 2, 2026Xunyi Zhao, Sihao Lin, Gengze Zhou +5Embodied Navigation AI Agent Benchmarks
1Adelaide University · 2Responsible AI Research Centre, Australian Institute for Machine Learning · Institute for Infocomm Research (I2R), A*STAR +2
DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration
Jun 2, 2026Wenkai Wang, Tao Xiong, Jingchen Ni +6Computer-Use Agent Benchmarks GUI Agents
1Zhejiang University · 2Tsinghua University · 3Tencent +1
What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents
Jun 1, 2026Victor Ojewale, Suresh VenkatasubramanianAgent Evaluation AI Agent Safety
Brown University Providence, Rhode Island, USA
$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues
Jun 1, 2026Peixuan Han, Hongyi Du, Jiayu Liu +3LLM Personalization LLM Agent Evaluation
University of Illinois Urbana-Champaign
ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents
Jun 1, 2026Yuxing Lu, Yushuhong Lin, Wenqi Shi +4Long-Horizon Agent Evaluation Clinical Decision-Making
1Georgia Institute of Technology · 2Peking University · University of Texas Southwestern Medical Center +1
HERO'S JOURNEY: Testing Complex Rule Induction with Text Games
Jun 1, 2026Anshun Asher Zheng, Kanishka Misra, David I. Beaver +1LLM Agent Evaluation AI Agent Benchmarks
Department of Linguistics The University of Texas at Austin
MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation
Jun 1, 2026Wenhao Wang, Peizhi Niu, Gongyi Zou +9LLM Agent Evaluation AI Agent Benchmarks
Multi-Agent Governance & Intelligence Crew (MAGIC), Shanghai Jiao Tong University, Shanghai, China · Department of Computer Science and Technology, Zhejiang University, Hangzhou, China · University of Illinois at Urbana-Champaign, Illinois, USA +2
HLL: Can Agents Cross Humanity's Last Line of Verification?
Jun 1, 2026Xinhao Song, Su Su, Sirui Song +6GUI Agents AI Agent Evaluation
1Shanghai Jiao Tong University · 2Shandong University · 3Tongji University
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Jun 1, 2026Nahyun Lee, Dongkeun Yoon, Guijin Son +12Computer-Use Agent Benchmarks LLM Evaluation
Chung-Ang University1 · KAIST2 · Seoul National University3 +3
SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
Jun 1, 2026Yuyan Bu, Haowei Li, Qirui Zheng +7LLM Agent Evaluation AI Agent Safety
Beijing Academy of Artificial Intelligence · University of Science and Technology of China · Peking University +2
TVIR: Building Deep Research Agents Towards Text-Visual Interleaved Report Generation
Jun 1, 2026Xinkai Ma, Zhiqi Bai, Dingling Zhang +21Deep Research Agents AI Agent Benchmarks
Nanjing University · Alibaba Group
Cross-Environment Neural Reranking for Sample-Efficient Action Selection in Text-Based Agents
Jun 1, 2026Kan ShaoAI Agent Benchmarks
Jinglue Technology Development (Nanjing) Co., Ltd. Nanjing, China
Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories
Jun 1, 2026Jiaming Wang, Ziteng Feng, Jiangtao Wu +8Agent Failure Analysis Deep Research Agents
NJU-LINK Team, Nanjing University · JIUTIAN Research
MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?
Jun 1, 2026Xinyu Che, Junqi Xiong, Yunfei Ge +10Continual Learning for LLM Agents LLM Agent Skill Learning
Nanjing University · Kuaishou Technology
AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
Jun 1, 2026Junqi Liu, Selena Song, Yuhan Wang +12AI Agent Evaluation AI Agent Benchmarks
University of California, Santa Cruz · 2NVIDIA
SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes
Jun 1, 2026Kuan Li, Shuo Zhang, Huacan Wang +12LLM Agent Evaluation Internet of Things
Midea Group · Beijing University of Posts and Telecommunications · Donghua University +2
RescueBench: Can Embodied Agents Save Lives in the Wild ?
Jun 1, 2026Kui Wu, Beiyu Guo, Hao Chen +6Agent Memory Embodied Navigation
1Beihang University · 5ATEC2025 Challenge Committee · 4City University of Macau +2
CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
Jun 1, 2026Danqing Wang, Akshay Sivaraman, Lei LiHuman Behavior Simulation LLM Agent Evaluation
Carnegie Mellon University
Planning Takes More Than Token Prediction: Causal Plan for Benchmarking and Building Physically Grounded Embodied Reasoners
Jun 1, 2026Zheng Lu, Mingqi Gao, Qinlei Xie +7Physical Reasoning AI Agent Benchmarks
Tsinghua University · Microsoft Research Asia · MBZUAI
Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation
Jun 1, 2026Donghwan Kim, Prakhar Singh, Younghoon Min +3LLM Agent Evaluation AI Agent Benchmarks
The Pennsylvania State University · SK Hynix · KAIST
RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agents
Jun 1, 2026Huayi Lai, Shichao Song, Simin Niu +5Moral Reasoning in Language Models AI Agent Benchmarks
School of Information, Renmin University of China, Beijing, China

Jun 2, 2026, cs.AIJ/K move · Enter open · S save

SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models

Joel Sol, Homayoun Najjaran

Faculty of Engineering and Computer Science University of Victoria Victoria, BC, Canada

Multi-Agent Coordination LLM Agent Evaluation AI Agent Benchmarks Multi-Agent Communication LLM Agent Reliability

Open paper AskPDF arXiv

As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty. We introduce SMAC-Talk, a natural language extension of the StarCraft Multi-Agent Challenge for evaluating LLM-based agents in cooperative multi-agent environments. The environment has several key features such as decentralized control, partial observability and long-horizon decision making. SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust. We use this communication channel to construct different evaluation scenarios, including settings with an embedded deceptive communicator that tries to disrupt and deceive allies through communication alone. We provide three agents for benchmarking using 4 models from the Qwen3.5 family and study how reasoning structure, memory and model scale affect coordination between agents. We release SMAC-Talk as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent settings.

Momentum

Related topics

Latest papers 1,013