LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

JobBench: Aligning Agent Work With Human Will
May 25, 2026Yuetai Li, Yichen Feng, Zhangchen Xu +21Computer-Use Agent Benchmarks LLM Agent Evaluation
University of Washington · 10Bake AI · University of California, Santa Barbara +10
CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
May 25, 2026Junlin Yang, Dylan Zhang, Xiangchen Song +7Structural Causal Models LLM Agent Evaluation
Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University +2
When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation
May 25, 2026Liyun Zhang, Jiayi GuoCoT Reasoning LLM Agent Evaluation
School of Information and Software Engineering, UESTC · Jacobs School of Engineering, UC San Diego
PolyGnosis 2.0: Enhancing LLM Reasoning via Agentic Harness Engineering for Polymarket and OSINT Insight Extraction
May 25, 2026Daren Wang, Hong Xu, Jiawen XianMulti-Agent LLM Systems LLM Agent Orchestration
The Chinese University of Hong Kong · Evolution AI Lab
GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning
May 24, 2026Xiang Cheng, Yulan Hu, Lulu Zheng +4AI-Assisted Decision Making LLM Planning
Gaoling School of Artificial Intelligence, Renmin University of China · AMAP, Alibaba Group
ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions
May 22, 2026Xianzhong Ding, Yangyang Yu, Changwei Liu +1Tool-Augmented Language Model Agents AI Coding Agents
Center for Advanced AI, Accenture
Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows
May 22, 2026Harshada Badave, Santosh Borse, Andrea Gomez +6AI Agent Reliability AI Agent Auditing
1IBM · 2Columbia University
SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills
May 22, 2026Yingtie Lei, Zhongwei Wan, Jiankun Zhang +13LLM Agent Evaluation LLM Agent Skill Learning
1The Ohio State University · 2The University of Chicago · University College London +4
From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills
May 22, 2026Zisu Huang, Jingwen Xu, Yifan Yang +13LLM Agent Evaluation LLM Agent Skill Learning
Fudan University · Microsoft Research · Shanghai Jiao Tong University
OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents
May 22, 2026Jiahao Ying, Boxian Ai, Wei Tang +2LLM Agent Evaluation AI Agent Benchmarks
Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University · Joy Future Academy, JD
More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries
May 21, 2026Hongwen Song, Song WeiLLM Agent Evaluation LLM Agent Skill Learning
Databricks Inc.
MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance
May 21, 2026Thomson Yen, Julian Poeltl, Harshith Srinivas Gear +9LLM Agent Evaluation AI Agent Benchmarks
Decision, Risk, and Operations Division, Columbia Business School · ESB Business School, Reutlingen University · Decision, Risk,and Operations Division, Columbia Business School
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
May 21, 2026Asaf Yehudai, Lilach Eden, Michal Shmueli-ScheuerLLM Agent Evaluation AI Agent Benchmarks
IIBM Research
SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations
May 21, 2026Shuaiqi Wang, Aadyaa Maddi, Zinan Lin +1Tool-Use Evaluation LLM Agent Evaluation
Carnegie Mellon University · Microsoft Research
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Computer-Use Agent Benchmarks Terminal Agents
University College London · Nanjing University · Tencent
Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions
May 21, 2026Jianan Ma, Xiaohu Du, Ruixiao Lin +8LLM Agent Evaluation LLM Agent Security
Hangzhou Dianzi University · Ant Group · Zhejiang University +1
Evaluating Large Language Models as Live Strategic Agents: Provider Performance, Hybrid Decomposition, and Operational Gaps in Timed Risk Play
May 21, 2026H. C. EkneLLM Agent Evaluation AI Agent Benchmarks
SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?
May 20, 2026Kevin Han, Renfei Zhang, Kathy Wei +3LLM Agent Evaluation AI Agent Benchmarks
Carnegie Mellon University · Stealth · Pennsylvania State University
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark Auditing LLM Auditing
BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
May 20, 2026Akshay Manglik, Vijay S. Kalmath, Jason Qin +6Multi-Agent LLM Systems LLM Agent Evaluation
Scale AI
MemGym: a Long-Horizon Memory Environment for LLM Agents
May 20, 2026Wujiang Xu, Yu Wang, Kai Mei +8LLM Agent Memory LLM Agent Evaluation
1Rutgers University · 2Capital One · 3Princeton University +1
AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
May 19, 2026Parsa Mazaheri, Kasra MazaheriAgent Failure Analysis Computer-Use Agent Benchmarks
University of California, Santa Cruz · Massachusetts Institute of Technology
Modeling Emotional Dynamics in Agent-to-Agent Interactions on Moltbook
May 19, 2026Syed Mhamudul Hasan, Abdur R. ShahidMulti-Agent LLM Systems LLM Agent Evaluation
Southern Illinois University Carbondale, IL, USA
Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving
May 19, 2026Oussama Zenkri, Oliver BrockLarge Language Model-Based Robot Planning LLM Agent Evaluation
Robotics and Biology Laboratory, Technische Universität Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Robotics Institute Germany (RIG)
When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity
May 19, 2026Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam +1LLM Agent Evaluation LLM Agent Skill Learning
Florida State University, Tallahassee, Florida, USA
EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design
May 19, 2026Gioele Molinari, Florian Felten, Soheyl Massoudi +1Multi-Agent LLM Systems LLM Agent Evaluation
IDEAL Chair of Artificial Intelligence in Engineering Design ETH Zurich Zurich, Switzerland · IDEAL, ETH Zurich, Zurich, Switzerland · Autom8.build, Brussels, Belgium
Toward User Comprehension Supports for LLM Agent Skill Specifications
May 19, 2026Zikai Alex WenLLM Agent Evaluation
University of Washington, Tacoma School of Engineering & Technology
PAVE: A Cognitive Architecture for Legitimate Violation in Generative Agent Societies
May 19, 2026Ahmad Yehia, Abduallah Mohamed, Kun Qian +4Cognitive Architectures for AI Agents LLM Agent Evaluation
The University of Texas at Austin · Meta Reality Labs · University of Calgary
Agentic Trading: When LLM Agents Meet Financial Markets
May 19, 2026Yihan Xia, Panpan You, Taotao Wang +4LLM Agent Evaluation Finance
College of Electronic and Information Engineering, Shenzhen University, Shenzhen, China
PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations
May 19, 2026Yingjie LeiAutomated Negotiation LLM Agent Evaluation
University of Aberdeen

May 25, 2026, cs.AIJ/K move · Enter open · S save

JobBench: Aligning Agent Work With Human Will

Yuetai Li, Yichen Feng, Zhangchen Xu, Zixian Ma+20

University of Washington · 10Bake AI · University of California, Santa Barbara+10

Computer-Use Agent Benchmarks LLM Agent Evaluation AI Agent Benchmarks Human-AI Collaboration AI and Labor

Open paper AskPDF arXiv

Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.

Momentum

Related topics

Latest papers 793