LLM Agent Evaluation

LLM: Large Language Model

Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 793

All topics
CardsList

  1. ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management

    Jul 22, 2026Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko +1LLM Agent EvaluationAI Agent Benchmarks

    University of Toronto · Vector Institute

  2. LLMs Get Lost in Evolving User Intent

    Jul 22, 2026Jihoon Tack, Philippe Laban, Jennifer NevilleLLM Agent EvaluationAI Agent Benchmarks

    Microsoft Research

  3. OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

    Jul 22, 2026Qiyuan Liu, Tingfeng Hui, Kun Zhan +2LLM Agent EvaluationAI Agent Safety

    Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Beijing University of Posts and Telecommunications +1

  4. Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning

    Jul 22, 2026Mohit Jiwatode, Ronja Fuchs, Robin Schmöcker +2Game-Playing AgentsLLM Agent Evaluation

    Institute for Information Processing · SDU Metaverse Lab

  5. Agents in the Wild: Where Research Meets Deployment

    Jul 21, 2026Grace Hui Yang, Pranav N. Venkit, Hooman Sedghamiz +3AI Agent ReliabilityMulti-Agent LLM Systems

    Georgetown University · Washington, D.C., USA · Salesforce +7

  6. Guardrails as Scapegoats: Auditing Unfaithful Safety Refusals in Tool-Augmented LLM Agents

    Jul 21, 2026Aarushi SinghAI Agent ReliabilityLLM Agent Evaluation

    Independent Researcher India

  7. MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

    Jul 21, 2026Guofeng Zhang, Yizeng Quan, Huaiyi Fang +6LLM Agent EvaluationMulti-Turn Dialogue Evaluation

    1Baidu, Inc. · 2Peking University · 3Zhejiang University

  8. Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agents

    Jul 21, 2026SangJin Park, Myungsub Choi, Jineok Kim +1LLM Agent EvaluationAI Agent Security

    Tynapse, Seoul, Republic of Korea.

  9. Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions

    Jul 20, 2026Pin Qian, Su Wang, Yihang Chen +5LLM Agent EvaluationAI Agent Benchmarks

    Carnegie Mellon University Pittsburgh, USA · Georgia Institute of Technology Atlanta, USA · Cornell University Ithaca, USA +1

  10. Operational Hallucination and Safety Drift in AI Agents

    Jul 20, 2026Shasha Yu, Fiona Carroll, Barry L. BentleyLLM Agent EvaluationAI Agent Safety

    Cardiff School of Technologies, Cardiff Metropolitan University Cardiff, UK · Harvard Medical School, Harvard University Boston, USA

  11. LLMs and Agentic AI Systems for Smart Grids: A Tutorial on Architectures and Applications

    Jul 20, 2026Daniela Rojas, Abdulwahab Albassam, Aidan G. Leung +11LLM Agent OrchestrationLLM Agent Evaluation

    Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, 92093, USA · Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, T6G 1H9, Canada

  12. Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation Data

    Jul 20, 2026Nursultan Askarbekuly, Mohamad Al Mdfaa, Ahmed Helaly +2AI Agent ReliabilityReward Hacking

    Innopolis University, Innopolis, Republic of Tatarstan, Russia · Skolkovo Institute of Science and Technology, Moscow, Russia

  13. Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

    Jul 20, 2026Devina Jain, David Hartmann, Chuan LiAdversarial Attacks on LLMsLLM Agent Evaluation

    Lambda

  14. Stress Testing Concept Erasure with Large Language Model Agents

    Jul 20, 2026Yuyang Xue, Feng Chen, Zhihua Liu +4LLM Agent EvaluationLLM Safety Evaluation

    School of Engineering, University of Edinburgh · The University of Manchester · The University of Melbourne

  15. FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches

    Jul 20, 2026Jiacheng Ding, Cong Guo, Jason XuLLM Agent EvaluationForecasting Benchmarks

    University of Memphis Memphis, Tennessee, USA · QuantaInsight Seattle, Washington, USA

  16. ProEvent: An Event-centric Benchmark for Proactive Agents

    Jul 20, 2026Guanzhen Li, Liangming Pan, Leye WangLLM Agent EvaluationProactive Assistance

    School of Computing, Peking University

  17. Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows

    Jul 20, 2026Jinyuan Deng, Zhengrui Chen, Xufeng Wei +4Electronic Design AutomationLLM Agent Evaluation

    1Zhejiang University, China · 2ChipFlux, China

  18. Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptions

    Jul 19, 2026Chen Xia, Zexi Kuang, Yuqing HuLLM GroundingHuman Behavior Simulation

    Department of Civil Engineering Technology, Environmental Management and Safety, Rochester Institute of Technology, Rochester, NY 14623, United States of America · Khoury College of Computer Sciences, Northeastern University, Boston, MA, United States of America · Digital and Interdependent Built Environment Laboratory, Department of Architectural Engineering, The Pennsylvania State University, University Park, PA, United States of America

  19. Agentic ERP: Multi-Agent Large Language Model Architecture for Autonomous Enterprise Resource Planning

    Jul 19, 2026Zhihao Liu, Tianyu Wang, Xi Vincent Wang +1Multi-Agent OrchestrationLLM Agent Evaluation

    Department of Production Engineering, KTH Royal Institute of Technology, Stockholm, Sweden

  20. OTAP: Structure-Aware Optimal Transport for Evaluating Planning and Execution in Agent Trajectories

    Jul 19, 2026Babak Barazandeh, Subhabrata Majumdar, George MichailidisLLM Agent EvaluationAI Agent Evaluation

    Fortinet · IIMB · UCLA

  21. Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

    Jul 19, 2026Arunabh DastidarAI Agent ReliabilityLLM Answer Verification

    for the Leni Team · Leni Inc.

  22. Lomekwi: Resource-Bounded Tool Discovery in LLM Agents

    Jul 18, 2026Roshan Klein-Seetharaman, Daniel Wang, Andrew XuLanguage Model Scaling LawsLLM Agent Evaluation

    Sea12 Technologies · Yale University

  23. Binding Drift in Multi-Step Tool-Augmented Agents

    Jul 17, 2026Rahul Suresh Babu, Shashank IndukuriLLM Agent EvaluationTool-Using Agents

  24. Fantastic Adaptive Taxonomies and How to Use Them

    Jul 17, 2026Mert Cemri, Andrei Cojocaru, Melissa Pan +9Agent Failure AnalysisLLM Agent Evaluation

    University of California, Berkeley · Apple · Bespoke Labs

  25. Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents

    Jul 17, 2026Lujia Zhang, Xingzhou Chen, Hongwei FengLLM Agent EvaluationDocument Information Extraction

  26. SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

    Jul 17, 2026Yanze Wang, Pengfei Yao, Tianyi Sun +7LLM Agent EvaluationLLM Agent Skill Retrieval

    1EverMind · 2Shanda Group · 3Peking University

  27. StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

    Jul 16, 2026Sizhong Qin, Yi Gu, Yao Jiang +13LLM Agent EvaluationAI Agent Benchmarks

  28. Copy-on-Write Scoring: Application-Specific Agent Evaluations

    Jul 15, 2026Joanna Roy, Sven HoelzelTool-Use EvaluationLLM Agent Evaluation

    trail-ml, Munich, Germany

  29. AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

    Jul 15, 2026Kai Chen, Zichen Ding, Jiaye Ge +20LLM Agent EvaluationAI Agent Benchmarks

    AgentCompass Team, Shanghai AI Laboratory

  30. STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle

    Jul 15, 2026Sagar Deb, Ashwanth KrishnanInventory ControlLLM Agent Evaluation

    QpiAI