LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability
Jul 7, 2026Chenxu Wang, Yongkun Yang, Boyuan Du +2LLM Agent Evaluation Multi-Agent Collaboration
Department of Computer Science and Technology, Tsinghua University · Fuzhou University
AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
Jul 7, 2026Andrey Podivilov, Vadim Lomshakov, Sergey Savin +4Computer-Use Agent Benchmarks AI Coding Agents
1Explyt · 2St. Petersburg Department of the Steklov Institute of Mathematics · 3St. Petersburg State University
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
Jul 7, 2026Wei Dong, Tianyu Fu, Zhe Yu +9LLM-as-a-Judge Web Agent Benchmarks
Mininglamp Technology · School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences
Onnes: A Physics-Grounded Multi-Agent LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Jul 7, 2026Praneeth Narisetty, Uday Kumar Reddy Kattamanchi, Shiva Nagendra Babu KoreMulti-Agent LLM Systems LLM Agent Evaluation
Onnes Research | San Francisco, CA
Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
Jul 7, 2026Wael Albayaydh, Rui Zhao, Ivan FlechaisAI Agent Reliability LLM Agent Evaluation
University of Oxford
LLM-as-a-Verifier: A General-Purpose Verification Framework
Jul 6, 2026Jacky Kwok, Shulu Li, Pranav Atreya +6LLM Answer Verification LLM Agent Evaluation
Stanford University · UC Berkeley · NVIDIA Research
SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
Jul 6, 2026Dylan Zongmin LiuLLM Agent Evaluation AI Agent Benchmarks
Stanford University
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
Jul 6, 2026Zhiheng Xi, Dingwen Yang, Jiaqi Liu +22LLM Agent Evaluation AI Agent Benchmarks
Fudan University · Zhejiang University · Shanghaijiaotong University +1
When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games
Jul 6, 2026Jerick Shi, Terry Jingcheng Zhang, Bernhard Schölkopf +2Multi-Agent LLM Systems Game-Playing Agents
Carnegie Mellon University, Pittsburgh, USA · Jinesis AI Lab, Vector Institute and University of Toronto, Toronto, Canada · EuroSafeAI +1
Decision Protocols in Multi-Agent Large Language Model Conversations
Jul 6, 2026Lars Benedikt KaesbergMulti-Agent LLM Systems LLM Agent Evaluation
GippLab Georg August University of Göttingen Göttingen
Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents
Jul 5, 2026Haiwen Yi, Xinyuan SongSoftware Engineering Agents LLM Agent Evaluation
University of Toronto · Emory University
Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems
Jul 5, 2026Xinyu Lin, Yashar Deldjoo, Sunhao Dai +7LLM Agent Evaluation Recommender Systems
National University of Singapore · Polytechnic University of Bari · Renmin University of China +1
Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces
Jul 5, 2026Andrew Zhang, Chengzhan LiLLM Agent Evaluation AI Agent Evaluation
KTH Royal Institute of Technology · University of Electronic Science and Technology of China
HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation
Jul 5, 2026Yaozu Wu, Wei-Chieh Huang, Jizhou Guo +11LLM Agent Evaluation Human-in-the-Loop Evaluation
The University of Tokyo · University of Illinois Chicago · MBZUAI +2
The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Calling
Jul 4, 2026Stefan Broecker, Mason del Rosario, Boris Selitser +1AI Agent Reliability LLM Agent Evaluation
University of California, Davis, Department of Computer Science · Okareo, Inc. · University of California, Davis, Department of Mathematics
The Remarkable Effectiveness of Providing AI Agents with Natural Language Tools: A Replication Study Validating NLT Performance Across 14 Models
Jul 4, 2026Alexander Somma, Isabelle Plante, Fred PremjiAI Agent Reliability LLM Agent Evaluation
Sage.is AI-UI
ProACT: Towards Breakdown-Aware Proactive Agent in Multi-User Collaboration
Jul 4, 2026Shu Yang, Difei Xu, Jiaxin Pei +1LLM Agent Evaluation Proactive Assistance
1PRADA Lab, King Abdullah University of Science and Technology · 2Stanford University
Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality
Jul 4, 2026Oussama Ben Sghaier, Hao Li, Bram Adams +1Software Engineering Agents LLM Agent Orchestration
Queen’s University, Canada
When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions
Jul 3, 2026Peiying Zhu, Sidi ChangLLM Agent Evaluation Policy Evaluation
Blossom AI San Francisco, USA · Blossom AI Labs Tokyo, Japan
Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
Jul 3, 2026Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin +3LLM Agent Evaluation LLMs for Cybersecurity
School of Computer Science and Mathematics, Keele University, Newcastle-under-Lyme ST5 5AA, U.K. · Cybersecurity Institute, University of Liverpool, Liverpool L69 3BX, U.K. · Department of Applied Computing IICL, University of Wales Trinity Saint David, London E14 4HA, U.K. +2
APeB: Benchmarking Personalization Ability of Large Language Model Agents
Jul 3, 2026Garry Yang, Zizhe Chen, Xinru Chen +9LLM Personalization LLM Agent Evaluation
1The Chinese University of Hong Kong · 2ByteDance
Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita
Jul 3, 2026Dan C. Hsu, Luke LuLLM Agent Evaluation AI Agent Benchmarks
RedMind Research, San Francisco, CA, USA · National Taiwan University, Taipei, Taiwan
MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents
Jul 3, 2026Siran Zhao, Ruihui Hou, Ziyue Huai +2LLM Agent Evaluation LLM Tool Use
East China University of Science and Technology, Shanghai, China
Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI
Jul 3, 2026Romain Gerard, Assmaa Zeghaider, Yan GuoLLM Agent Orchestration LLM Agent Evaluation
University of Science and Technology of China, Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China
What LLM Agents Say When No One Is Watching: Social Structure and Latent Objective Emergence in Multi-Agent Debates
Jul 2, 2026Arman Ghaffarizadeh, Danyal Mohaddes, Aliakbar Izadkhah +1LLM Agent Evaluation Multi-Agent Debate
Independent Researcher · Machine Learning Department & Heinz College Carnegie Mellon University
Coding-agents can replicate scientific machine learning papers
Jul 2, 2026Atharva Hans, Ilias BilionisCoding Agents LLM Agent Evaluation
School of Mechanical Engineering, Purdue University, West Lafayette, IN, 47907, USA
PACE: A Proxy for Agentic Capability Evaluation
Jul 2, 2026Yueqi Song, Lintang Sutawika, Jiarui Liu +8Benchmark Design LLM Agent Evaluation
Carnegie Mellon University · Salesforce AI Research
SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use
Jul 2, 2026Jiayin Zhu, Kelong Mao, Yudong Guo +4LLM Agent Evaluation LLM Agent Skill Learning
1HKUST(GZ), 2JD.COM · ∗Yutao Yue is also affiliated with Institute of Deep Perception Technology, JITRI, Wuxi, China.
CLAP: Closed-Loop Training, Evaluation, and Release Control for Domain Agent Post-training
Jul 2, 2026Fangfei Li, Chenyang Zhao, Long Wang +3LLM Agent Evaluation LLM Post-Training
MatrixOrigin China · MatrixOrigin USA
AgenticDataBench: A Comprehensive Benchmark for Data Agents
Jul 2, 2026Zhaoyan Sun, Shan Zhong, Daizhou Wen +10LLM Agent Evaluation AI Agent Benchmarks
Tsinghua University · Ant Digital Technologies, Ant Group

Jul 7, 2026, cs.CLJ/K move · Enter open · S save

LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability

Chenxu Wang, Yongkun Yang, Boyuan Du, Shiwei Lin+1

Department of Computer Science and Technology, Tsinghua University · Fuzhou University

LLM Agent Evaluation Multi-Agent Collaboration AI Agent Benchmarks Multi-Agent Reasoning

Open paper AskPDF arXiv

Deliberation plays a crucial role in collaboration; when humans work together, they naturally engage in communication to align information and reach an agreement. In this paper, we investigate deliberative large language model (LLM) agents under partially observable joint decision-making tasks. We formalize deliberative collaboration as a cooperative joint decision problem with partial and asymmetric observations, and introduce a scalable benchmark that instantiates this problem across multiple task settings and domains in which agents must exchange information through deliberation to reach a joint decision with a shared reward. We then instantiate a reference scaffold and evaluation protocol for deliberative agents and conduct a systematic evaluation of a range of representative LLMs. The results reveal that complex deliberative collaboration tasks continue to challenge state-of-the-art language models. Even with the aid of external mathematical tools, language models may fail in either the deliberation process for aligning information or the complex reasoning process for making the decision. On the other hand, diagnostic analysis reveals that the deliberation process may also provide opportunities for reflection and error correction, sometimes improving performance over centralized baselines. Altogether, our work establishes a foundation for evaluating and improving LLM agents in deliberative collaboration and provides insights into the strengths, limitations, and properties of current LLM-based multi-agent systems.

Momentum

Related topics

Latest papers 793