LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents
May 10, 2026Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen +2LLM Agent Evaluation AI Agent Benchmarks
Novo Nordisk London, UK
Causal state binding predicts action control in language agents
May 10, 2026Xiao JiaLanguage Model-Based Control LLM Agent Evaluation
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen
MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions
May 10, 2026Yaoning Yu, Ye Yu, Haojing Luo +1Human Behavior Simulation LLM Agent Evaluation
University of Illinois at Urbana-Champaign
ProactBench: Beyond What The User Asked For
May 9, 2026Sepehr Harfi, Ahmad Salimi, Dongming Shen +1LLM Agent Evaluation Proactive Assistance
Boson AI Toronto, ON, Canada · Boson AI Santa Clara, CA, USA
When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents
May 9, 2026Strick Sheng, Ziyue Wang, Liyi ZhouAI Agent Reliability LLM Agent Evaluation
The University of Sydney · Nanjing University
Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows
May 9, 2026Tao Yu, Hao Wang, Changyu Li +15LLM Agent Evaluation Business Process Automation
BAAI · CASIA · UCAS +2
NARRA-Gym for Evaluating Interactive Narrative Agents
May 8, 2026Yue Huang, Yuchen Ma, Jiayi Ye +14Language Model Generation Evaluation Interactive Storytelling
University of Notre Dame · 2LMU Munich · 11Munich Center for Machine Learning +8
Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
May 8, 2026Anmol Gulati, Hariom Gupta, Elias Lumer +2LLM Agent Evaluation Long-Horizon Agent Evaluation
PricewaterhouseCoopers U.S.
AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
May 8, 2026Zhengkang Guo, Yiyang Li, Lin Qiu +7LLM Agent Evaluation Long-Horizon Agent Evaluation
1Fudan University · 2Meituan Longcat Team
SCENE: Recognizing Social Norms and Sanctioning in Group Chats
May 8, 2026Mateusz Jacniacki, Maksymilian BilskiLLM Agent Evaluation AI Agent Benchmarks
Humalike
DRIP-R: A Benchmark for Decision-Making and Reasoning Under Real-World Policy Ambiguity in the Retail Domain
May 8, 2026Hsuvas Borkakoty, Sebastian Pohl, Cheng Wang +2LLM Agent Evaluation AI Agent Benchmarks
Interdisciplinary Transformation University, Austria · Amazon, Berlin, Germany
Business Utility of Large Language Models as Exploratory Data Analysis Agents
May 8, 2026Rafał Łabędzki, Patryk Miziuła, Hubert Rutkowski +5LLM Agent Evaluation AI Agent Benchmarks
SGH Warsaw School of Economics · Bydgoszcz University of Science and Technology · At the time of contribution aﬀiliated with deepsense.ai, present aﬀiliation is Google
Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning
May 8, 2026Yuyang Wu, Yue Huang, Shuaike Shen +8AI Agents for Scientific Discovery LLM Agent Evaluation
Carnegie Mellon University · University of Notre Dame · University of North Carolina, Chapel Hill +1
MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries
May 8, 2026Zixuan Xie, Xinyu Liu, Shangtong ZhangLLM Evaluation LLM Agent Evaluation
University of Virginia
TeamBench: Evaluating Agent Coordination under Enforced Role Separation
May 8, 2026Yubin Kim, Chanwoo Park, Taehan Kim +9Multi-Agent Coordination LLM Agent Evaluation
MIT · Google Research · Independent Researcher +1
SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair
May 7, 2026Ion George Dinu, Marian Cristian Mihăescu, Traian RebedeaAutomated Program Repair LLM Agent Evaluation
University of Craiova Craiova, Romania · NVIDIA University Politehnica of Bucharest Bucharest, Romania
Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility
May 7, 2026Nyx Iskandar, Perla GamezTool-Use Evaluation LLM Agent Evaluation
Foam
Bridging the Last Mile of Circuit Design: PostEDA-Bench, a Hierarchical Benchmark for PPA Convergence and DRC Fixing
May 7, 2026Pengju Liu, Nuo Xu, Jinwei Tang +2Automated Program Repair Electronic Design Automation
University of Minnesota
Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
May 7, 2026Hailey Onweller, Elias Lumer, Austin Huber +3LLM Evaluation Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
May 7, 2026Hanxiang Chao, Yihan Bai, Rui Sheng +2Agent Memory LLM Agent Evaluation
Wuhan University1, Wuhan, China1 · The Hong Kong University of Science and Technology3, Hong Kong, China23 · The Chinese University of Hong Kong2, Hong Kong, China23
Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems
May 7, 2026Donghao Huang, Joon Kiat Chua, Zhaoxia WangFinancial Services Multi-Agent LLM Systems
School of Computing and Information Systems, Singapore Management University, 80 Stamford Rd, Singapore 178902, Singapore · Research and Development, Mastercard, 4250 Fairfax Dr, Arlington, VA 22203, USA
Stop Comparing LLM Agents Without Disclosing the Harness
May 7, 2026Yunbei Zhang, Janet Wang, Yingqiang Ge +3LLM Agent Evaluation Long-Horizon Agent Evaluation
Tulane University · Rutgers University · Independent Researcher +1
BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
May 7, 2026Jhen-Ke LinBenchmark Construction LLM Agent Evaluation
National Yang Ming Chiao Tung University
More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffolding
May 7, 2026Ming LiuLLM Agent Orchestration LLM Agent Evaluation
Amazon
SODE: Analyzing Social Dynamics in LLM Agents
May 6, 2026Inseo Jung, Yoonseok Oh, Kyungryul Back +2Multi-Agent LLM Systems LLM Agent Evaluation
Department of Computer Science, Korea University · Kakao Mobility
Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games
May 5, 2026Connacher MurphyGame-Playing Agents Benchmark Design
Stanford University
SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment
May 5, 2026Joseph Breda, Fadi Yousif, Beszel Hawkins +30Healthcare Medical Diagnosis
Google Research · Work done while at Google Research · Google DeepMind
Evaluating Generative Models as Interactive Emergent Representations of Human-Like Collaborative Behavior
May 5, 2026Shinas Shaji, Teena Chakkalayil Hassan, Sebastian Houben +1Multi-Agent LLM Systems Theory of Mind
Institute of AI and Autonomous Systems (A2S), Hochschule Bonn-Rhein-Sieg, Grantham-Allee 20, Sankt Augustin, 53757, Germany. · 3Fraunhofer Institute for Intelligent Analysis and Information SystemsMay (IAIS), Schloss Birlinghoven, Sankt Augustin, 53757, Germany. · 2Division for Systems and Control, Chalmers University of Technology, Chalmersplatsen 1, Gothenburg, 41296, Sweden.
SOTOPIA-TOM: Evaluating Privacy and Information Management in Multi-Agent Interaction with Theory of Mind
May 4, 2026Yashwanth YS, Ruichen Wang, Shihua Zeng +4Multi-Agent Coordination Theory of Mind
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA · Fujitsu Research of America, Pittsburgh, USA
MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing
May 4, 2026Nishant Bhargava, Rodrigo Sobral BarrentoPersistent Memory for Language Models LLM Agent Evaluation
Purdue University

May 10, 2026, cs.AIJ/K move · Enter open · S save

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens+1

Novo Nordisk London, UK

LLM Agent Evaluation AI Agent Benchmarks LLM Agents Data Analysis Agents

Open paper AskPDF arXiv

As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but unintended task framings, producing clean, executable artifacts that hide their incorrect assessment of the task. Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified. We introduce Ambig-DS, two diagnostic suites: one for prediction-target ambiguity (Ambig-DS-Target, 51 tasks built on DSBench, a tabular modeling benchmark) and one for evaluation-objective ambiguity (Ambig-DS-Objective, 61 tasks built on MLE-bench, a Kaggle-style ML competition benchmark), constructed so that scoring uses each source benchmark's original evaluator. For every task we pair the original, fully specified version with an ambiguous variant produced by controlled edits; a human-and-LLM verification pipeline confirms each variant admits multiple plausible interpretations with decision-relevant consequences. The suites are analyzed independently and ambiguity lowers performance in both. Across five agents spanning efficient to frontier-class models, we find in our controlled diagnostic setting: (i) failures are silent commitments: wrong-target submissions on Target, wrong-metric or non-committal baseline submissions on Objective, rather than execution errors; (ii) allowing the agent to ask one clarifying question recovers much of the loss under idealized conditions, suggesting missing framing information drives a substantial part of the observed degradation; but (iii) agents cannot reliably tell when to use it: permissive prompts induce over-asking on clear tasks, while conservative prompts induce silent defaulting on ambiguous ones. Recognizing target and objective underspecification, not pipeline execution, is the bottleneck missing from standard DS-agent evaluations.

Momentum

Related topics

Latest papers 793