AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

146 papers in the last four weeks, up 161% on the four weeks before. 1.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Jun 15, 2026Anzhe Xie, Weihang Su, Yujia Zhou +2AI-Assisted Scientific Research LLM Agent Evaluation
Tsinghua University Beijing, China
LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control
Jun 15, 2026Anqi Zou, Han Deng, Chengyu Zhang +9Computer-Use Agent Benchmarks Computer-Use Agents
Shenzhen Loop Area Institute · Dalian University of Technology · The Chinese University of Hong Kong +1
MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents
Jun 15, 2026Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh +1Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents
Carnegie Mellon University
AgentFairBench: Do LLM Agents Discriminate When They Act?
Jun 15, 2026Triveni Morla, Rohith Reddy Bellibaltu, Manpreet Singh +1Algorithmic Fairness LLM Agent Evaluation
Florida International University, Miami, FL 33199 USA · Boston University, Boston, MA 02215 USA · Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, Bihar, India
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
Jun 15, 2026Issa Sugiura, Daichi Hattori, Kazuo Araragi +5Long-Horizon Agent Evaluation AI Agent Benchmarks
Sakana AI · KPMG AZSA LLC
How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks
Jun 15, 2026Alibek T Kaliyev, Artem MaryanskyyMulti-Agent Coordination Pairwise Comparison
The University of Texas at Austin · Uber
VisualClaw: A Real-Time, Personalized Agent for the Physical World
Jun 15, 2026Haoqin Tu, Jianwen Chen, Zijun Wang +14Continual Learning for LLM Agents AI Agent Benchmarks
1UC Santa Cruz · 2UNC-Chapel Hill · 3Google +1
UXBench: Measuring the Actionability of LLM-Generated UX Critiques
Jun 15, 2026Wenjie Wang, Yue Huang, Zipeng Ling +11LLM-as-a-Judge LLM Agent Evaluation
University of Notre Dame · University of Pennsylvania · University of Rochester +4
RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
Jun 14, 2026Linghua Zhang, Jun Wang, Jingtong Wu +1Long-Horizon Agent Evaluation AI Agent Benchmarks
Ant Group · City University of Hong Kong
Multi-agent Framework for Time-Sensitive Complementary Collaboration in Minecraft
Jun 14, 2026Juheon Yi, Jinglu Wang, Xiaoyi Zhang +1Multi-Agent Coordination Multi-Agent Collaboration
Microsoft Research Asia
ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
Jun 13, 2026Rahul Suresh Babu, Laxmipriya Ganesh IyerAI Agent Reliability Tool-Use Evaluation
Independent Researcher United States of America
LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure
Jun 13, 2026Daksh Mittal, Tommaso Castellani, Thomson Yen +7Continual Learning for LLM Agents Lifelong Learning Agents
1Columbia University · 2Oumi
CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?
Jun 13, 2026Yuxin Zhang, Ju Fan, Meihao Fan +2AI Coding Agents Software Engineering Benchmarks
Renmin University of China
Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation
Jun 13, 2026Shijun Wan, Xuehai Wu, Jiwen Zhang +2LLM Agent Evaluation AI Agent Benchmarks
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute
Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoning
Jun 13, 2026Sanhorn Chen, Xiaoyang Chen, Boyu Liu +1Irregular Time-Series Modeling LLM Evaluation
University of Illinois Urbana Champaign
StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance
Jun 12, 2026Guanming Liu, Yuqi Ren, Hansu Gu +5Agent Memory Long-Horizon Agent Evaluation
1Fudan University · 2Amazon
Running the Gauntlet: Hard Agentic Tasks
Jun 12, 2026Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel +203D Spatial Reasoning Web Agent Benchmarks
SoftServe · Ukrainian Catholic University · University of Oxford +3
OdysSim: Building Foundation Models for Human Behavior Simulation
Jun 12, 2026Xuhui Zhou, Weiwei Sun, Weihua Du +6Human Behavior Simulation AI Agent Benchmarks
Carnegie Mellon University, Language Technologies Institute
LessonBench-V1: A Benchmark Dataset for Evaluating AI Lesson Generation Agents
Jun 12, 2026Ravidu Suien Rammuni Silva, Ahmad Lotfi, Isibor Kennedy Ihianle +2Educational Assessment Generative AI in Education
Department of Computer Science, Nottingham Trent University, Nottingham, UK · Department of Physics and Mathematics, Nottingham Trent University, Nottingham, UK
Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
Jun 12, 2026Brendan King, Jeffrey FlaniganCoding Agents LLM Agent Evaluation
University of California, Santa Cruz
SANA: What Matters for QA Agents over Massive Data Lakes?
Jun 11, 2026Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang +1LLM Agent Evaluation Question Answering
Columbia University New York, USA
Poker Arena: Multi-Axis Profiling of Strategic Reasoning and Memory in LLMs
Jun 11, 2026Pratham Singla, Shivank Garg, Vihan SinghGame-Playing Agents Strategic Reasoning Risks in Language Models
Indian Institute of Technology Roorkee · Raeth AI
EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments
Jun 11, 2026Jundong Xu, Qingchuan Li, Jiaying Wu +11Benchmark Design Long-Horizon Agent Evaluation
National University of Singapore · Singapore Management University · University of Washington +5
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
Jun 11, 2026Xiaoyuan Liu, Jianhong Tu, Yuqi Chen +26Software Engineering Agents AI Agent Evaluation
UC Berkeley · University of California, Santa Cruz · EPFL +12
EpiBench: Verifiable Evaluation of AI Agents on Epigenomics Analysis
Jun 11, 2026Harihara Muralidharan, Reema Baskar, Soo Hee Lee +2AI Agents for Scientific Discovery AI Agent Evaluation
LatchBio, San Francisco, CA, USA
RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue
Jun 11, 2026Sara Candussio, Emanuele Ballarin, Lorenzo Bonin +2AI Agent Reliability Language Model Safety Evaluation
1AILab, MIGe, University of Trieste, 34127 Trieste, IT · 2Computational Statistics and Machine Learning, Istituto Italiano di Tecnologia, Genova, IT · 3DIA, University of Trieste, 34127 Trieste, IT
ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm
Jun 11, 2026Jiaxin Ai, Tao Hu, Xuemeng Yang +11Computer-Use Agents Program Synthesis
Wuhan University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +4
TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?
Jun 11, 2026Dat Tien Nguyen, Thao Nguyen, Fadillah Adamsyah Maani +5AI Agents for Scientific Discovery Scientific Reasoning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · ADIA Lab, Abu Dhabi, UAE
EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge
Jun 11, 2026Yunhan Wang, Jiaan Wang, Lianzhe Huang +2Benchmark Contamination Web Search Agents
Northeastern University, China · Weixin AI, Tencent Inc, China
The Illusion of Multi-Agent Advantage
Jun 11, 2026Prathyusha Jwalapuram, Hehai Lin, Chuyuan Li +7AI Agent Evaluation Agent Evaluation
1Salesforce Research. · 2HKUST (Guangzhou). · University of British Columbia. +1

Jun 15, 2026, cs.CLJ/K move · Enter open · S save

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio

Anzhe Xie, Weihang Su, Yujia Zhou, Yiqun Liu+1

Tsinghua University Beijing, China

AI-Assisted Scientific Research LLM Agent Evaluation AI Agent Benchmarks Biomedical IR Evidence Retrieval

Open paper AskPDF arXiv

Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442 expert-curated meta-analyses from Nature Portfolio journals. Each entry pairs a research question with PI/ECO criteria, a retrieval corpus of 140k PubMed articles, verified positive studies, hard negatives that are topically similar but PI/ECO-ineligible, and complete search strategies and date bounds. Benchmarking twelve pipeline configurations (nine RAG variants and a protocol-driven agent) reveals a critical screening bottleneck: despite a retrieval ceiling of 90.9% recall at K=200, no system recovers more than 52.7% of ground-truth included literature. Current LLMs fail to reliably separate eligible studies from PI/ECO-failing distractors in pools of comparable topical relevance. Stage-attributed metrics capture where systems succeed and fail; a single end-to-end score does not.

Momentum

Related topics

Latest papers 1,013