AI Agent Benchmarks

Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 1,013

All topics
CardsList

  1. LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

    Sep 28, 2026Bingo Zhang, Haochuan Lu, Zongjie Li +3GUI AgentsLong-Horizon Agent Evaluation

    Vera Praxis · Tencent · The Hong Kong University of Science and Technology +2

  2. VehicleArena: A Realistic Urban Environment for Multi-Agent Driving

    Sep 28, 2026Jie Yang, Jiajun Chen, Jiazheng Zhou +4Autonomous Driving BenchmarksAutonomous Driving

    Fudan University · Shanghai Innovation Institute

  3. FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Management

    Sep 28, 2026Peiyu ZangLong-Horizon Agent EvaluationAI Agent Benchmarks

    School of Mathematical Sciences, Beijing Normal University · CoRe Lab, Institute for Artificial Intelligence, Peking University

  4. AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs

    Sep 28, 2026Cheuk Hang Lau, Zeyu Cao, Kevin Wong Cheuk Yin +6LLM Inference EfficiencyLLM Inference

    Imperial College London · University of Cambridge · University of Oxford

  5. The Marathon of Scientific Reasoning: Robustness of Scientific Agents to Perturbations in Multi-Turn Interactions

    Sep 28, 2026Xiaoting Lyu, Xinbo Ma, Yufei Han +5AI Agent BenchmarksScientific Reasoning in Language Models

    Xi’an Jiaotong University, China · Inria, France · Agency for Science, Technology and Research (A*STAR), Singapore +1

  6. MASTraceBench: Diagnosing Collaboration Gains through Proposal Trajectories in LLM-Based Multi-Agent Systems

    Sep 28, 2026Yapeng Li, Songze Li, Shuang Yu +4Multi-Agent LLM SystemsMulti-Agent Collaboration

    Harbin Institute of Technology, Harbin, China · Peking University, Beijing, China

  7. PowerBench: A Benchmark for Agentic Retrieval and Reasoning in Power Systems

    Sep 28, 2026Xijing Wang, Yinsheng Yao, Jinru Ding +5LLM Agent EvaluationAI Agent Benchmarks

    Tongji University · Johns Hopkins University · Shanghai Artificial Intelligence Laboratory +3

  8. AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering

    Sep 28, 2026Chanhee Park, Jeongho Yoon, Sungbin Han +2Multi-Hop QATool-Augmented Language Model Agents

    Korea University · Sookmyung Women’s University

  9. BIABench: Evaluating AI agents on real-world bioimage analysis tasks

    Sep 28, 2026Zixuan Pan, Davide Panzeri, Lukas Johanns +5AI Agent ReliabilityAI Agent Evaluation

    Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA · Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V., Dortmund, Germany · Institute of Computer Science, University of Tartu, Tartu, Estonia

  10. Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

    Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Reward HackingBenchmark Auditing

    Scale AI

  11. SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals

    Sep 28, 2026Jingyun Jia, Antoine Remond-Tiedrez, Aaron Alvarez +3LLM EvaluationAI Agent Benchmarks

    University of Wisconsin–Madison, Intelligible · Intelligible · University of Cincinnati

  12. Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures

    Sep 28, 2026Dong Xu, Zhangfan Yang, Jiantao Wu +5LLM Agent EvaluationAI Agent Benchmarks

    School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo

  13. RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

    Sep 28, 2026Haitong Ma, Chenxiao Gao, Rushi Qiang +2AI Agent BenchmarksRobot Learning

    Harvard University · Georgia Institute of Technology

  14. PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models

    Sep 28, 2026Shane K. A. Dalumura Hettige, Jonas OppenlaenderCreativity AssessmentAgentic Image Generation

    Computer Science and Engineering University of Oulu Oulu, Finland · Centre for Applied Computing University of Oulu Oulu, Finland

  15. DynGraphAgentBench: A Benchmark for Agentic Lifecycle Control in Dynamic Graph Anomaly Detection

    Sep 27, 2026Yuwei Han, Lingwei Wei, Wooseong Yang +4Online Anomaly DetectionAI Agent Benchmarks

  16. When Successful Strategies Fail: Adaptation to Environmental Novelty in Terminal Agents

    Sep 27, 2026Janvijay Singh, Vaishnavi Shrivastava, Dilek Hakkani-Tur +2Long-Horizon Agent EvaluationAI Agent Benchmarks

    University of Illinois Urbana-Champaign · Microsoft Research AI Frontiers

  17. Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models

    Sep 27, 2026Eduardo Ariño de la Rubia, Szilard PafkaAI Coding AgentsLLM Agent Evaluation

    Central European University · Epoch

  18. Self-Designed Evaluators and Warm Memory for Long-Horizon Agents

    Sep 27, 2026Saeid Asgari, Emre Kiciman, Leonardo de Oliveira Nunes +1Language Model Generation EvaluationLong-Horizon Agent Evaluation

    Microsoft

  19. Auditing Agent Actions through Query-Conditioned Attribution

    Sep 27, 2026Yifan Liu, Praveen Venkateswaran, Abdulhamid Adebayo +1Gradient-Based AttributionAI Agent Benchmarks

    University of Illinois Urbana-Champaign · IBM

  20. LiveOption: Evaluating LLM Agents in Structured Option Trading with Nonlinear Payoffs

    Sep 27, 2026Haochen Luo, Yifan Li, Binh Minh An +4Quantitative FinanceLLM Agent Evaluation

    City University of Hong Kong · The Chinese University of Hong Kong (Shenzhen) · Shanghai University of Finance and Economics

  21. DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

    Sep 27, 2026Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4AI Agent ReliabilityScientific Hypothesis Generation

    University of California San Diego · Independent Researcher

  22. Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks

    Sep 27, 2026Analog Design Bench TeamBenchmark DesignElectronic Design Automation

  23. TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces

    Sep 27, 2026Dehai Min, Daoan Zhang, Yiming Zeng +13Benchmark ConstructionLLM Agent Evaluation

    ByteDance Inc., USA · University of Illinois at Chicago

  24. Dude, Where's My State? Execution Information Requirements for Stateful Agents

    Sep 26, 2026Nikita Mehrotra, Ashish Tiwari, Priyanshu Gupta +1Software Engineering AgentsLong-Horizon Agent Evaluation

  25. ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

    Sep 24, 2026Ming Zhang, Zhenghao Xiang, Peizhong Gao +17AI Agent EvaluationAI Agent Benchmarks

    Fudan University · Hunyuan Team, Tencent

  26. Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge

    Sep 24, 2026Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3Tool-Augmented Language Model AgentsLLM Agent Evaluation

    Eon

  27. Policy as Code: A Coroutine-Bridge Harness for Fast-Reasoning Reliability on CAR-bench

    Sep 24, 2026Ivan MatveevAgent Harness OptimizationAI Agent Benchmarks

    Proxima Ultra

  28. RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

    Sep 23, 2026Mithil Salunkhe, Haochen Ding, Samridhi Verma +1LLM Agent EvaluationAI Agent Benchmarks

    University of Illinois Urbana-Champaign · National Center for Supercomputing Applications

  29. TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation

    Sep 23, 2026Geng Chen, Ruotong Pan, Zhirui Yang +9AI Agent BenchmarksUser Simulation

    Peking University · Kuaishou Technology · Xi’an Jiaotong University +3

  30. SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving

    Sep 23, 2026Zhilong Ge, Yuting Shao, Yutao Yang +6Tool-Augmented Language Model AgentsLLM Agent Skill Learning

    School of Computer Science and Technology, East China Normal University, Shanghai · Shanghai AI Laboratory