LLM Agent Evaluation

LLM: Large Language Model

Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 793

All topics
CardsList

  1. OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality

    Aug 5, 2026Yidian Chen, Yingzi Gu, Natan Vidra +2Agent Failure AnalysisAI Agent Reliability

    Anote

  2. When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs

    Aug 5, 2026Jiaming Cheng, Subhransu Das, Rajiv RamnathKV CachingLLM Agent Evaluation

    Independent Researcher · The Ohio State University

  3. Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

    Aug 5, 2026Jinyi Han, Yuanjian Xu, Ying Liao +6LLM Agent EvaluationLLM Agent Skill Retrieval

    1East China Normal University · 2Hong Kong University of Science and Technology · 3Fudan University +1

  4. Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

    Aug 5, 2026Atul Anand, Sourav ChattarajLLM Agent EvaluationTool-Using Language Agents

  5. Strategic Evaluation of Planning Strategies for LLM Agents in Cyber-Physical Systems

    Aug 4, 2026J. de Curtò, I. de ZarzàCyber-Physical SystemsLLM Planning

    Department of Computer Applications in Science and Engineering, BARCELONA Supercomputing Center, Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, Madrid, Spain · Human-Centered AI, Data and Software, LUXEMBOURG Institute of Science and Technology, Esch-sur-Alzette, Luxembourg

  6. ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?

    Aug 4, 2026Tianyi Guan, Yiding Wang, Haotong Yang +5Continual Learning for LLM AgentsLLM Agent Evaluation

    Institute for Artificial Intelligence, Peking University · 2Beijing Institute for General Artificial Intelligence

  7. GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks

    Aug 4, 2026Leijun Zhou, Zhihao Liu, Xiang Qu +9LLM Agent EvaluationAI Agent Benchmarks

    PrismShadow · New York University

  8. Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems

    Aug 4, 2026Sebastián Andrés Cajas Ordóñez, Agastya Munnangi, Aldo Marzullo +13AI Agent ReliabilityMulti-Agent LLM Systems

    MIT Critical Data, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States · Georgia State University, Atlanta, Georgia, United States · Politecnico di Milano, Milan, Italy +12

  9. When Outputs Disperse, Does Epistemic Revision Follow? A Black-Box Diagnostic for Machine Collectives

    Aug 4, 2026Molood ArmanLLM Agent EvaluationMulti-Agent Debate

    Independent Researcher Lyon, France

  10. DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction

    Aug 4, 2026Xuyang Liu, Yibin Han, Zhenwei Zhang +8Retrieval-Augmented GenerationLLM Agent Evaluation

    KLISS, BNRist, School of Software, Tsinghua University, Beijing, China · CRRC Corporation Limited, Beijing, China

  11. DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

    Aug 4, 2026Boyan Li, Zhuowen Liang, Yupeng Xie +11LLM Agent EvaluationMultimodal Agents

    HKUST(GZ) · Tsinghua University

  12. EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners

    Aug 4, 2026Unggi Lee, Sookbun Lee, Yeil Jeong +3Intelligent Tutoring SystemsAI in Education

    Korea University Sejong Campus · Opentutorials · Indiana University +2

  13. Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation

    Aug 4, 2026Saqib Shouqi, Abdullah Nazly, Januki Wanniarachchi +1Adversarial Attacks on LLMsLLM Agent Evaluation

    School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka

  14. UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks

    Aug 4, 2026Jiayu Cao, Xingyuan Zeng, feiyu Li +6LLM Agent EvaluationAI Agent Benchmarks

  15. An Explainable LLM Agent Layer for Open-World Anomaly Detection in Oil Wells

    Aug 4, 2026Lucas Gouveia Omena Lopes, Thales Miranda de Almeida Vieira, Eduardo Toledo de Lima Junior +1Interpretable Anomaly DetectionLLM Agent Evaluation

    Laboratory of Scientific Computing and Visualization, Federal University of Alagoas, Maceio, Alagoas, Brazil · Computing Institute, Federal University of Alagoas, Maceio, Alagoas, Brazil

  16. Everyone Conforms, No One Believes: Pluralistic Ignorance in LLM Agent Populations

    Aug 3, 2026Yashwanth YSMulti-Agent LLM SystemsNormative Conformity in Language Models

    Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA

  17. ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

    Aug 3, 2026Wei-Jung Huang, Bonan ShenPairwise ComparisonLLM Agent Evaluation

    Independent Researcher · United States

  18. ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step

    Aug 3, 2026Vernon Toh, Navonil Majumder, Zhengyuan Liu +2LLM Agent EvaluationAI Agent Benchmarks

    DeCLaRe Lab, Nanyang Technological University, Singapore · Agency for Science, Technology, and Research (A*STAR), Singapore

  19. From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents

    Aug 3, 2026Jiajia Song, Bobo Li, Haiwen Yi +6LLM PersonalizationLLM Agent Evaluation

    National University of Singapore · University of Toronto · University of Minnesota Twin Cities +2

  20. GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks

    Aug 3, 2026Jiarui Tan, Zhongjian Zhang, YaBo Guo +5LLM Agent EvaluationAI Agent Benchmarks

    Beijing University of Posts and Telecommunications · The Chinese University of Hong Kong · Peking University

  21. GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks

    Aug 3, 2026Abhinav Pothuri, Zhe Jiang, Zelin Xu +1LLM Agent EvaluationAI Agent Benchmarks

    Department of Computer & Information Science & Engineering, University of Florida · Department of Geography, University of Florida

  22. Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments

    Aug 2, 2026Dotan Davidovich, Yair Amar, Hai Rozencwajg +1Software Engineering AgentsLLM Agent Evaluation

    Accomplish AI

  23. Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

    Aug 2, 2026Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh +9LLM Agent EvaluationAI Agent Safety

    Ho Chi Minh City University of Science Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Ho Chi Minh City University of Technology (HCMUT) Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Vrije Universiteit Brussel Université Libre de Bruxelles Brussels, Belgium +1

  24. CallScreenBench: Benchmarking On-Device Models as Phone Secretaries

    Aug 2, 2026Simiao RenOn-Device Language Model InferenceLLM Agent Evaluation

  25. Who Belongs in the Eval Set? A Capability-Taxonomy-Driven Pipeline for Curating Regression Eval Sets in Agent-Extensibility Platforms

    Aug 2, 2026Tezan Sahu, Aritra Das, Pankaj Mittal +1Benchmark ConstructionLLM Agent Evaluation

    Microsoft Hyderabad, India · Indian Institute of Technology Roorkee Roorkee, India

  26. Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations

    Aug 1, 2026Xinshun Feng, Ziqi Miao, Lijun Li +1AI Agents for Scientific DiscoveryHallucination in Language Models

    Shanghai Artificial Intelligence Laboratory Shanghai, China

  27. Measuring Cross-Task Behavioral Consistency in Language Model Agents

    Jul 31, 2026Amritesh Banerjee, Pranil RaichuraAI Agent ReliabilitySoftware Engineering Agents

    University of Massachusetts Amherst · Mantis AI Research (MIT CSAIL)

  28. ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

    Jul 31, 2026Boyang Zhang, Adrian Lyjak, Eli Stewart +2LLM GroundingLLM Agent Evaluation

  29. AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?

    Jul 31, 2026Dong Yan, Jian Liang, Dapeng Hu +4LLM Agent EvaluationAI Agent Benchmarks

    School of Artificial Intelligence, University of Chinese Academy of Sciences · Microsoft · Institute of Automation, Chinese Academy of Sciences +1

  30. AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

    Jul 31, 2026Tianyu Huai, Tingshuo Fan, Xinchi Chen +5LLM Agent EvaluationAI Agent Benchmarks

    Shanghai Innovation Institute · East China Normal University · OpenMOSS +1