AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows
Jul 25, 2026Summer SunLLM Agent Evaluation AI Agent Benchmarks
Shaqiu Community
The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
Jul 24, 2026Darshan Tank, Baran NamaLLM Agent Evaluation LLM Agent Skill Learning
Sentient Labs
SceneActBench: Can Agents Act on the 3D Scenes They See?
Jul 24, 2026Yifei Zhao, Xiangxin Zhou, Wenhao Yang +11AI Agent Benchmarks Tool-Using Vision-Language Agents
1Tencent Hunyuan · 2THU · 3NJU +3
Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
Jul 24, 2026Jiaqi Shao, Hanck Chen, Wei Zhang +2Computer-Use Agent Benchmarks Reward Hacking
1Hunyuan Team, Tencent · 2The Hong Kong University of Science and Technology · 3Duke Kunshan University
DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents
Jul 24, 2026Junming Chen, Junyang Jiang, Xu Chen +2Benchmark Design AI Agent Benchmarks
University of Electronic Science and Technology of China Chengdu, China
Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents
Jul 24, 2026Suman Navaratnarajah, Taehyoung Kim, Jona Ruthardt +5Zero-Shot Learning Multimodal Large Language Models
University of Technology Nuremberg · Fraunhofer IVI · THWS +1
Agentic Evaluation of Copyright Law Compliance
Jul 23, 2026Zheng Hui, Doni Bloomfield, Noam KoltAI Agent Evaluation AI Agent Benchmarks
Language Technology Lab, University of Cambridge, Cambridge, United Kingdom · Law School, Fordham University, New York, United States · Faculty of Law and School of Computer Science and Engineering, Hebrew University, Jerusalem, Israel
Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks
Jul 23, 2026Michael Kouremetis, Ads Dawson, Raja Sekhar Rao Dheekonda +1LLM Auditing Deception in Language Models
dreadnode, Principal AI Research Engineer · dreadnode, StaffAI Security Researcher · dreadnode, Distinguished Engineer +1
Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
Jul 23, 2026Mack Nixon, Liam Wright, Yevgeniya Kovalchuk +4AI Agent Benchmarks Open-Weight Language Models
Centre for Advanced Research Computing, University College London, London, UK · Centre for Longitudinal Studies, University College London, London, UK · UK Longitudinal Linkage Collaboration, University of Bristol, Bristol, UK
AI Assistants Overassist
Jul 23, 2026Verona Teo, Raghav Jain, Tobias Gerstenberg +1Intelligent Tutoring Systems LLM Agent Evaluation
1Stanford University · University of California, San Diego · University of Washington
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
Jul 23, 2026Zhongyuan Peng, Dan Huang, Chuyu Zhang +8Coding Agents Software Engineering Benchmarks
Fudan University · Meituan Group · Singapore Management University +1
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
Jul 23, 2026Yinhao Tang, Youqing Fang, Yanan Sun +6AI Agent Benchmarks Information Retrieval
University of Science and Technology of China · Shanghai AI Laboratory
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
Jul 23, 2026Tencent WorkBuddy Bench Team, Siqi Cai, Shaopeng Chen +35Benchmark Contamination Software Engineering Benchmarks
Youtu Lab · Keen Security Lab · Workbuddy · Yunding Security Lab
ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management
Jul 22, 2026Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko +1LLM Agent Evaluation AI Agent Benchmarks
University of Toronto · Vector Institute
LLMs Get Lost in Evolving User Intent
Jul 22, 2026Jihoon Tack, Philippe Laban, Jennifer NevilleLLM Agent Evaluation AI Agent Benchmarks
Microsoft Research
Frontier Financial Judgement: Can agents tell what might move a stock?
Jul 22, 2026Joshua HarrisAI Agent Evaluation AI Agent Benchmarks
Sonto
KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?
Jul 22, 2026Peiyu Zang, Jian Tao, Jialing Zhang +4GPU Kernel Optimization Benchmark Design
Beijing Normal University · BAAI · Beijing Jiaotong University +2
Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning
Jul 22, 2026Mohit Jiwatode, Ronja Fuchs, Robin Schmöcker +2Game-Playing Agents LLM Agent Evaluation
Institute for Information Processing · SDU Metaverse Lab
DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations
Jul 22, 2026Jiazhen Jiang, Boxi Cao, Lingyong Yan +6Long-Horizon Agent Evaluation AI Agent Benchmarks
Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc.
Same Game, Different Story: A Minimal Conservative Strategic Robustness Benchmark for Large Language Model Agents
Jul 22, 2026Seyed Pouyan Mousavi Davoudi, Arshia Gharagozlou, Alireza Amiri-Margavi +2AI Agent Benchmarks Framing Effects in Language Models
Independent Researcher in AI and Statistics Tehran, Iran · Mathematics & Statistics Department University of Minnesota Duluth Duluth, MN, USA · Computational Modeling and Simulation University of Pittsburgh Pittsburgh, PA, USA +1
BioSecBench-Surveillance: A Verifiable Benchmark for AI Agents in Pathogen Genomic Surveillance
Jul 21, 2026Harmon Bhasin, Kevin Flyangolts, Dianzhuo Wang +9AI Agent Evaluation AI Agent Benchmarks
LatchBio, San Francisco, CA · Aclid, New York, NY
Do AI-Native Biotechs Need Departments? Benchmarking Company World Models for AI-Driven Drug Development
Jul 21, 2026Yinan WangWorld Models AI Agent Evaluation
Noah AI Research
AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research
Jul 20, 2026Marjan Moodi, Xuankang Zhu, Fernando De Mesentier Silva +2World Model Learning AI Agent Benchmarks
Electronic Arts · Simon Fraser University
Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions
Jul 20, 2026Pin Qian, Su Wang, Yihang Chen +5LLM Agent Evaluation AI Agent Benchmarks
Carnegie Mellon University Pittsburgh, USA · Georgia Institute of Technology Atlanta, USA · Cornell University Ithaca, USA +1
The Autonomous Agency Scale: A Behavioral Framework for Measuring Self-Directed Behavior in AI Systems
Jul 20, 2026Samuel PresgravesAI Agent Evaluation AI Agent Benchmarks
Independent researcher.
Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows
Jul 20, 2026Jinyuan Deng, Zhengrui Chen, Xufeng Wei +4Electronic Design Automation LLM Agent Evaluation
1Zhejiang University, China · 2ChipFlux, China
EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
Jul 19, 2026Yaohan Yang, Minglei Shi, Borui Zhang +2Computer-Use Agent Benchmarks GUI Agents
Department of Automation, Tsinghua University
Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent
Jul 19, 2026Arunabh DastidarAI Agent Reliability LLM Answer Verification
for the Leni Team · Leni Inc.
Lomekwi: Resource-Bounded Tool Discovery in LLM Agents
Jul 18, 2026Roshan Klein-Seetharaman, Daniel Wang, Andrew XuLanguage Model Scaling Laws LLM Agent Evaluation
Sea12 Technologies · Yale University
Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration
Jul 18, 2026Maksim Sheverev, David Finkelstein, Sergey NikolenkoLLM Agent Memory AI Agent Benchmarks
Quantellence Research · St. Petersburg Department of the Steklov Institute of Mathematics, St. Petersburg, Russia · St. Petersburg State University, St. Petersburg, Russia

Jul 25, 2026, cs.AIJ/K move · Enter open · S save

SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows

Summer Sun

Shaqiu Community

LLM Agent Evaluation AI Agent Benchmarks LLM Agent Reliability

Open paper AskPDF arXiv

Existing evaluations of large language models cover knowledge, reasoning, coding, and tool use, but they rarely treat a verifiable deliverable produced within a constrained workflow as the unit of evaluation. We introduce SQBench, a benchmark for evaluating production-oriented task delivery by language-model agents. SQBench v1.0 contains 220 standardized tasks organized into L1 atomic capabilities, L2 composite skills, and L3 business scenarios. Each task requires an agent to process input assets, use available tools, and produce an explicitly specified deliverable. The evaluation first computes functional Completion and then derives Risk Penalty and Performance from independently evidenced triggers in a 10D Risk Matrix. A Strict Pass requires Completion = 1 and Risk Penalty = 0. We evaluate 27 model configurations under a common protocol, with one run per configuration-task pair. The highest prespecified Weighted Pass@1 is 60.5%. Mean Strict Pass@1 on L3 is 18.5%, and every configuration performs worse on L3 than on both L1 and L2, indicating that delivery under domain constraints is a shared weakness within the current task set. Of 2,348 results with Completion = 1, 113 (4.8%) fail the Strict Pass criterion because of risks such as unverifiable citations, inappropriate resource use, or format violations. These results show that functional completion alone does not fully characterize delivery quality and that risk determinations should be reported separately.

Momentum

Related topics

Latest papers 1,013