LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems: A Cross-Provider Empirical Extension
May 28, 2026Francisco León Zúñiga BolívarMulti-Agent LLM Systems LLM Agent Evaluation
Institución Universitaria Colegio Mayor del Cauca · Institución Universitaria Colegio Mayor del Cauca Popayán, Colombia
Realistic honeypot evaluations for scheming propensity
May 28, 2026Victoria Krakovna, David Lindner, Lewis Ho +2LLM Agent Evaluation AI Agent Safety
1Google DeepMind
PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game?
May 28, 2026Dongdong Hua, Yifei Sun, Renhong Huang +3Game-Playing Agents LLM Agent Evaluation
Zhejiang University · FinVolution Group
MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs
May 28, 2026Kevin Wang, Anna Thöni, Benjamin Kempinski +50LLM Agent Evaluation Strategic Reasoning Risks in Language Models
Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework
May 28, 2026Masafumi Enomoto, Ryoma Obara, Haochen Zhang +1Web Agent Benchmarks LLM Agent Evaluation
NEC Corporation
BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents
May 28, 2026Jiahao Huang, Fei Cheng, Junfeng Jiang +2LLM Agent Evaluation AI Agent Benchmarks
University of Tokyo · Kyoto University · National Institute of Informatics
Hallucination Mitigation with Agentic AI, Nested Learning, and AI Sustainability via Semantic Caching
May 27, 2026Diego Gosmar, Deborah A. DahlAI Agent Reliability Multi-Agent LLM Systems
Head of AI, Tesisquare · Member, Open Voice Interoperability Initiative · Linux Foundation AI & Data +4
Frontier LLM-based agents can overcome the ontology curation bottleneck for natural phenotypes
May 27, 2026James P. Balhoff, Hilmar LappLLM-Assisted Annotation LLM Agent Evaluation
Renaissance Computing Institute, University of North Carolina, Chapel Hill, North Carolina, USA · Neuromatch, USA
Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval
May 27, 2026Shiyu Chen, Tarfah Alrashed, Alon Halevy +1LLM Agent Evaluation Agentic Retrieval
Google, Mountain View, CA, USA
LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?
May 27, 2026HuiMing Fan, Xiao Wang, Zheng Chu +5Computer-Use Agent Benchmarks LLM Agent Evaluation
1Harbin Institute of Technology · 2Xiaohongshu
Evaluating the Realism of LLM-powered Social Agents: A Case Study of Reactions to Spanish Online News
May 27, 2026Alejandro Buitrago López, Alberto Ortega Pastor, Javier Pastor-Galindo +1LLM Agent Evaluation Large Language Model-Based Social Simulation
Faculty of Computer Science, University of Murcia, Murcia (Spain)
A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
May 27, 2026Tomer Keren, Nitay Calderon, Asaf Yehudai +3Computer-Use Agent Benchmarks Tool-Use Evaluation
Faculty of Data and Decision Science Technion · IBM Research
Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents
May 27, 2026Liang Cheng, Mingsheng Cai, Jiuming Jiang +1LLM Agent Evaluation AI Agent Benchmarks
†University of Edinburgh
Beyond One Path: Evaluating and Enhancing Divergent Thinking in Interactive LLM Agents
May 27, 2026Jihyeong Park, Ingeol Baek, Jeonghyun Park +1LLM Agent Evaluation AI Agent Benchmarks
Chung-Ang University, Seoul, Korea
From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets
May 27, 2026Taojie Zhu, Wentao Zhao, Rui Sun +7Benchmark Contamination LLM Agent Evaluation
1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University +3
OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents
May 27, 2026Chenyu Zhou, Xinyun Lu, Jiangyue Zhao +3LLM Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University · Shanghai University of Finance and Economics · Stanford University
Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
May 27, 2026Bin Wu, Guanyun Zou, Bingbing Wang +2LLM Agent Memory LLM Agent Evaluation
Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen) +1
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +9Computer-Use Agent Benchmarks AI Agent Reliability
1Peking University · 2Qiyuan Tech
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
Diagnosing Live Within-Policy Instruction Conflicts in LLM Agents with Witnessed Resolution Profiles
May 27, 2026Lu Yan, Xuan Chen, Xiangyu ZhangLLM Agent Evaluation LLM Agents
Purdue University
Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
May 26, 2026Aman Priyanshu, Supriti Vijay, Esha PahwaData Leakage Multi-Agent LLM Systems
Foundation AI · USA · Corvic AI
UserHarness: Harnessing User Minds for Stronger Agent Theory-of-Mind
May 26, 2026Cheng Qian, Jiayu Liu, Heng JiTheory of Mind LLM Agent Evaluation
University of Illinois Urbana-Champaign
Voluntary Collusion with Secret Tools in Competing LLM Agents
May 26, 2026Xijie Zeng, Frank RudziczLLM Agent Evaluation LLM Agent Safety
Dalhousie University · Vector Institute for Artificial Intelligence
DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents
May 26, 2026Shijie Cao, Yuan Yuan, Jing LiuLLM Agent Evaluation Combinatorial Optimization
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Shenzhen Loop Area Institute, Shenzhen, China · Qingdao Research Institute, Beihang University +3
Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
May 26, 2026Yipeng Ouyang, Xin Huang, Bingjie Liu +3Software Engineering Agents LLM Agent Evaluation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents
May 26, 2026Ye Yuan, Rui Song, Weien Li +12Social Deduction Games LLM Agent Evaluation
McGill University · Mila - Quebec AI Institute · University of Cambridge +3
It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
May 26, 2026Yong-eun ChoAI Agent Reliability LLM Agent Evaluation
KailosLab · Seoul, Republic of Korea
Towards Feedback-to-Plan Decisions for Self-Evolving LLM Agents in CUDA Kernel Generation
May 26, 2026Yee Hin Chong, Jiaming Wu, Youhui Zhang +1GPU Kernel Optimization LLM Agent Evaluation
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Beijing National Research Center for Information Science and Technology, Beijing, China
LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis
May 26, 2026Bowen QinLLM Agent Evaluation AI Agent Benchmarks
National University of Singapore
Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization
May 26, 2026Anmol Agarwal, Natalie Neamtu, Pranjal Aggarwal +6LLM Agent Evaluation Formal Verification
CMU · Amazon · Work performed while at Amazon.

May 28, 2026, cs.MAJ/K move · Enter open · S save

Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems: A Cross-Provider Empirical Extension

Francisco León Zúñiga Bolívar

Institución Universitaria Colegio Mayor del Cauca · Institución Universitaria Colegio Mayor del Cauca Popayán, Colombia

Multi-Agent LLM Systems LLM Agent Evaluation AI Agent Benchmarks Evolutionary Game Theory Emergent Behavior in Multi-Agent Systems

Open paper AskPDF arXiv

Do next-generation LLM agents inherit the cooperative biases documented in their predecessors, or does scale and provider diversity reshape equilibrium behaviour in competitive multi-agent settings? Willis et al. established a benchmark for this question using evolutionary game theory and the Iterated Prisoner's Dilemma (IPD), finding consistent cooperative biases in ChatGPT-4o and Claude 3.5 Sonnet. We extend this benchmark to four frontier models released in 2025-2026 - Claude Sonnet 4.6, Gemini 2.5 Flash, Gemini 3.1 Pro, and GPT-5.4 Mini - applying the identical protocol across three prompting styles (Default, Prose, Self-Refine) and four population compositions (balanced and biased, with and without noise). Cooperative bias persists across providers (H1): ten of twelve model-prompt combinations favour cooperative equilibria in balanced noiseless conditions. Cross-provider divergence is substantial (H3): Gemini 2.5 Flash reaches up to 77% aggressive equilibria under biased conditions, while GPT-5.4 Mini reaches 70% cooperative equilibria under Self-Refine. Support for aggressive capability parity is partial (H2): Self-Refine raises ICD in all models and Gemini 3.1 Pro Refine achieves the highest ICD in the dataset (0.925), but Default and Prose prompts show no systematic narrowing. Evidence on noise robustness is directionally positive but not robustly confirmed (H4): with n=500 Moran iterations per condition, average noise sensitivity is about 6 percentage points for Claude Sonnet 4.6 versus 13 pp for Claude 3.5 Sonnet, but this cross-study gap is not statistically significant once the predecessor's unreported sampling error is propagated. Provider identity, rather than model generation, is the strongest correlate of equilibrium outcomes; noise remains a universal challenge regardless of model size or vintage.

Momentum

Related topics

Latest papers 793