LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

12 Angry AI Agents: Evaluating Multi-Agent LLM Decision-Making Through Cinematic Jury Deliberation
May 3, 2026Ahmet Bahaddin ErsozLLM Alignment LLM Agent Evaluation
∗ORCID: 0000-0001-6297-7501
NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles
May 3, 2026Xiao JiaAI Agent Reliability LLM Agent Evaluation
School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong, Shenzhen
How Personas Can Influence Agents to Play Split or Steal
May 3, 2026Carlos Leon, Alexandre Rodrigues, Pedro Gamito +1Game-Playing Agents LLM Agent Evaluation
Universidade Lusófona, Portugal · Universitat de Barcelona, Spain · Université Paris Cité, France +1
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAI Agent Reliability Reward Hacking
Independent Researcher
LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation
May 2, 2026Dong Xu, Jialun Cao, Guozhao Mo +9LLM Agent Evaluation Formal Verification
Institute of Software, Chinese Academy of Sciences · The Hong Kong University of Science and Technology · Guangzhou Institute of Technology, Xidian University
Agentic AI Systems Should Be Designed as Marginal Token Allocators
May 2, 2026Siqi ZhuLLM Agent Orchestration LLM Agent Evaluation
University of Illinois Urbana-Champaign
AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
May 1, 2026Ranit Karmakar, Jayita ChatterjeeComputer-Use Agent Benchmarks LLM Agent Evaluation
Harvard University, Boston, MA 02115 · Boston, MA 02115
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
Apr 30, 2026Chenxin Li, Zhengyang Tang, Mingxin Huang +8LLM Agent Evaluation AI Agent Benchmarks
1The Chinese University of Hong Kong · 2The Chinese University of Hong Kong, Shenzhen · 3South China University of Technology +4
Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception
Apr 30, 2026Neemias B da Silva, Rodrigo Minetto, Daniel Silver +1VLM Evaluation Personality Modeling in Language Models
Universidade Tecnol´ogica Federal do Paran´a (UTFPR), Curitiba, Brazil · University of Toronto, Toronto, Canada
Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization
Apr 30, 2026Jackson Vonderhorst, Kuangshi Ai, Haichao Miao +2Scientific Visualization Data Visualization
Univ. Notre Dame · LLNL
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Apr 30, 2026Qiyao Wang, Haoran Hu, Longze Chen +4Web Application Generation Web Agent Benchmarks
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Dalian University of Technology +2
When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis
Apr 29, 2026Juergen DietrichMulti-Agent LLM Systems Stance Detection
TRUST Project — democracy-intelligence.de
Evaluating Strategic Reasoning in Forecasting Agents
Apr 28, 2026Tom Liptay, Dan Schwarz, Rafael Poyiadzi +2LLM Agent Evaluation Forecasting Benchmarks
FutureSearch
Operating-Layer Controls for Onchain Language-Model Agents Under Real Capital
Apr 28, 2026T. J. Barton, Chris Constantakis, Patti Hauseman +4AI Agent Reliability LLM Agent Evaluation
DX Research Group (DXRG)
The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications
Apr 27, 2026Zhenyu Zhao, Aparna Balagopalan, Adi Agrawal +3Financial Services LLM Sycophancy
Writer, Inc.
Agentic clinical reasoning over longitudinal myeloma records: a retrospective evaluation against expert consensus
Apr 27, 2026Johannes Moll, Jannik Lübberstedt, Christoph Nuernbergk +21Healthcare LLM Agent Evaluation
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, TUM University Hospital, School of Medicine and Health, Technical University of Munich, Munich, Germany · Department of Cardiovascular Radiology and Nuclear Medicine, German Heart Center, TUM University Hospital, School of Medicine and Health, Technical University of Munich, Munich, Germany +11
MarketBench: Evaluating AI Agents as Market Participants
Apr 26, 2026Andrey Fradkin, Rohit KrishnanMulti-Agent Coordination LLM Agent Evaluation
Boston University and the MIT Initiative on the Digital Economy. · Independent Researcher.
ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation
Apr 26, 2026Boqin Yuan, Yue Su, Renchu Song +2LLM Agent Evaluation LLM Agent Skill Learning
University of California San Diego · San Diego, CA · Carnegie Mellon University +3
DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute
Apr 26, 2026Nishant Balepur, Malachi Hamada, Varsha Kishore +9Deep Research Agents LLM Agent Evaluation
University of Maryland · New York University · Allen Institute for Artificial Intelligence (Ai2)
ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Apr 26, 2026Fanqing Meng, Lingxiao Du, Zijian Wu +46Computer-Use Agent Benchmarks LLM Agent Evaluation
Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
Apr 24, 2026Mengzhuo Chen, Junjie Wang, Fangwen Mu +4Agent Failure Analysis Multi-Agent LLM Systems
State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes
Apr 24, 2026Rubén Garzón, Pauline Baron, Vincent Grari +3LLM Agent Evaluation Human Behavior Prediction
AI Research, AXA Group Operations, Madrid, Spain. · AI Research, AXA Group Operations, Paris, France. · AI Research, AXA Group Operations, Lausanne, Switzerland. +2
Sound Agentic Science Requires Adversarial Experiments
Apr 23, 2026Dionizije Fa, Marko CuljakAI-Assisted Scientific Research LLM Agent Evaluation
Entropic
Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
Apr 23, 2026Benjamin Kohler, David Zollikofer, Johanna Einsiedler +2AI Coding Agents LLM Agent Evaluation
ETH Zurich · University of Basel
How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines
Apr 23, 2026Abel YagubyanAI Agent Reliability LLM Agent Evaluation
Independent Researcher
Efficient Agent Evaluation via Diversity-Guided User Simulation
Apr 23, 2026Itay Nakash, George Kour, Ateret Anaby-TavorAutomated Software Testing LLM Agent Evaluation
IBM Research
When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors
Apr 23, 2026Chenghao Yang, Yuning Zhang, Zhoufutu Wen +4Tool-Augmented Language Model Agents LLM Agent Evaluation
School of Cyber Science and Technology, USTC · Anhui Province Key Laboratory of Digital Security · 3M-A-P +1
Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflows
Apr 22, 2026Hardy Chen, Nancy Lau, Haoqin Tu +8Coding Agents LLM Agent Evaluation
UC Santa Cruz · UT Dallas · NUS
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Apr 22, 2026Shanshan Zhong, Yi Lu, Jingjie Ning +7Continual Learning for LLM Agents LLM Agent Evaluation
Carnegie Mellon University · Amazon AGI
Time Series Augmented Generation for Financial Applications
Apr 21, 2026Anton Kolonin, Alexey Glushchenko, Evgeny Bochkov +1LLM Agent Evaluation AI Agent Benchmarks
SingularityNET Foundation

May 3, 2026, cs.AIJ/K move · Enter open · S save

12 Angry AI Agents: Evaluating Multi-Agent LLM Decision-Making Through Cinematic Jury Deliberation

Ahmet Bahaddin Ersoz

∗ORCID: 0000-0001-6297-7501

LLM Alignment LLM Agent Evaluation Multi-Agent Debate LLM Decision-Making

Open paper AskPDF arXiv

What if the twelve jurors of Sidney Lumet's 12 Angry Men (1957) were not men, but large language models? Would the one juror who disagrees still be able to change everyone's mind? This paper instantiates that scenario as a multi-agent benchmark for LLM deliberation: twelve agents, each conditioned on a film-faithful persona, debate the film's murder case using multi-agent framework. Two models representing opposite ends of the RLHF spectrum are tested: GPT-4o (closed-source, heavy alignment) and Llama-4-Scout (open-weight, lighter alignment), across three conditions (baseline, open-minded prompt, no initial vote), with N = 3 replications per cell (18 runs total). Three findings emerge. (i) Seventeen of eighteen runs end in a hung jury (a state where the jury fails to reach a unanimous verdict); the film's central event, gradual minority-to-majority persuasion, almost never occurs, indicating that anchoring is the dominant failure mode of current LLMs in this setting. (ii) The two models exhibit sharply different internal dynamics: GPT-4o produces a mean of 1.0 vote changes per run across all conditions, while Llama-4-Scout ranges from 2.0 (baseline) to 6.0 (open-minded prompt), and is the only model to reach a NOT_GUILTY verdict (1 of 3 runs in the no-initial-vote condition). The same ``open-minded'' instruction is internalized by Llama and ignored by GPT-4o. (iii) This asymmetry suggests that the intensity of RLHF alignment training, not model capability, is the primary determinant of deliberative flexibility in multi-agent settings. Flexibility, not capability, tracks human deliberation. The work is framed as an exploratory study and discusses implications for jury-of-LLMs evaluation and multi-agent debate.

Momentum

Related topics

Latest papers 793