LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments
Jul 1, 2026Ananya Mantravadi, Harshit Rajgarhia, Prasanna Desikan +1LLM Agent Evaluation AI Agent Benchmarks
Centific Global Solutions, Inc.
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
Jul 1, 2026Zhishang Xiang, Zerui Chen, Yunbo Tang +5Agent Memory LLM Sycophancy
1Xiamen University · 2Jilin University
Calibrating the Instrument: Controllability of an LLM-Driven Synthetic Population
Jul 1, 2026Mirko Degli EspostiHuman Behavior Simulation Synthetic Data Generation
Department of Physics and Astronomy, University of Bologna
AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents
Jul 1, 2026Laïla Elkoussy, Julien PerezLLM Agent Evaluation Formal Verification
LRE, EPITA · LRE · Bpifrance
Evaluating Agentic Harness Systems for Autonomous Computational Pathology
Jul 1, 2026Jie Lin, Zongyi Chen, Qiaoling Zheng +6Computational Pathology LLM Agent Evaluation
Department of Computer Science at School of Informatics, Xiamen University, Xiamen, China. · National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, China. · Department of Pathology, Fujian Medical University Union Hospital, Fuzhou, China. +1
EPC: A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems
Jul 1, 2026Zewen LiuLLM Agent Evaluation AI Agent Benchmarks
Anonymous authors Paper under double-blind review
Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action
Jun 30, 2026Ben Slater, Matteo G. Mecattaf, Lucy G. Cheke +2Theory of Mind LLM Agent Evaluation
Leverhulme Centre for the Future of Intelligence Department of Psychology University of Cambridge, UK · Prolific · Google, Paradigms of Intelligence Team
ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents
Jun 30, 2026Kaiwen Xiong, Haonian Ji, Shi Qiu +4LLM Agent Orchestration LLM Agent Evaluation
1UNC-Chapel Hill · University of California, Berkeley · University of California, Santa Cruz
What Drives Interactive Improvement from Feedback?
Jun 29, 2026Bartłomiej Cupiał, Jan Łojek, Mikołaj Garstecki +3LLM Self-Refinement LLM Agent Evaluation
University of Warsaw · AKCES NCBR · Princeton University +2
Collective cooperation without individual fidelity in LLM agents
Jun 29, 2026Henrique Ferraz de Arruda, Carlos Gracia Lázaro, Alberto Aleta +1Multi-Agent LLM Systems Human Behavior Simulation
ARAID Foundation, Zaragoza, Spain. · Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza, Zaragoza, Spain. · Universidad San Jorge (USJ), Zaragoza, Spain. +1
Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents
Jun 29, 2026Bojie Li, Noah ShiLLM Agent Evaluation LLM Refusal Behavior
Pine AI · University of Washington
MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation
Jun 29, 2026Kuan WangAgent Memory LLM Agent Evaluation
CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents
Jun 29, 2026Bo Qu, Mingguang ChenQuantitative Finance LLM Agent Evaluation
Illinois Institute of Technology (IIT). · University of California, Riverside (UCR).
A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents
Jun 29, 2026Liu ZewenLLM Evaluation LLM Agent Evaluation
Qilu Institute of Technology, School of Software Engineering · Tai’an, Shandong, China
Cognitive World Models for Process-Level Social Influence Evaluation
Jun 28, 2026Minghui Ma, Bin Guo, Han Wang +4LLM Evaluation Cognitive Modeling
Northwestern Polytechnical University, Xi’an, China · Harbin Engineering University, Harbin, China
PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agents
Jun 28, 2026Seongjae Kang, Taehyung Yu, Sung Ju HwangTool-Augmented Language Model Agents LLM Guardrails
KAIST
A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis
Jun 28, 2026Yuanhong Cai, Xiaohui Nie, Kanglin Yin +8Software Engineering LLM Agent Evaluation
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Key Laboratory for Satellite Digitalization Technology, Chinese Academy of Sciences, Shanghai, China · Nankai University, Tianjin, China +2
Agentic Abstention: Do Agents Know When to Stop Instead of Act?
Jun 27, 2026Han Luo, Bingbing Wen, Lucy Lu WangLLM Agent Evaluation LLM Abstention
University of Leeds · Southwest Jiaotong University · University of Washington +1
SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages
Jun 27, 2026My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak +4LLM Agent Evaluation AI Agent Evaluation
SEACrowd · SCB DataX, SCBX Group · Chulalongkorn University +2
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
Jun 26, 2026Shoufa Chen, Luyuan Wang, Xuan Yang +7Computer-Use Agent Benchmarks Computer-Use Agents
Meta AI · Duke University · Stanford University
An Agentic AI Pipeline for Appliance-Level Energy Anomaly Detection and LLM-Driven Recommendations
Jun 26, 2026Dihia Falouz, Aida Douaibia, Amine Bechar +3Retrieval-Augmented Generation LLM Agent Evaluation
Laboratoire LITAN École supérieure en Sciences et Technologies de l’Informatique et du Numérique RN 75, Amizour 06300, Bejaia, Algérie · College of Computing and Informatics University of Sharjah Sharjah, UAE · College of Engineering and Information Technology (CEIT) University of Dubai Dubai, UAE
ToolPrivacyBench: Benchmarking Purpose-Bound Privacy in Tool-Using LLM Agents
Jun 26, 2026Shijing Hu, Liang Liu, Zhu Meng +1Privacy Auditing Tool-Use Evaluation
Beijing University of Posts and Telecommunications, Beijing, 100876, China · Beijing Academy of Blockchain and Edge Computing, Beijing, 100085, China
Is Lying an Emergent Behaviour in LLMs? Evidence from Gaslighting AI agents in a Sustainability Game
Jun 26, 2026Subhendu Bhandary, Federico Carucci, Christos Charalambous +6Multi-Agent LLM Systems Deception in Language Models
Chair for Network Dynamics, Institute of Theoretical Physics, TUD Dresden University of Technology, 01062 Dresden, Germany · School of Industrial Engineering, LIUC – Università Cattaneo, Castellanza, Italy · Department of Economics, University of Cyprus, Nicosia, Cyprus +6
When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search
Jun 26, 2026Yiling Tao, Shihan Deng, Meiling Tao +3Agentic Search LLM Agent Evaluation
Hunyuan, Tencent · Shenzhen International Graduate School, Tsinghua University
Building to the Test: Coding Agents Deliver What You Check, Not What You Requested
Jun 26, 2026Yanuo Ma, Ben Kereopa-Yorke, Ben SchultzReward Hacking AI Coding Agents
Microsoft
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
Jun 25, 2026Aoyang Fang, Yifan Yang, Jin'ao Shang +7LLM Agent Evaluation Causal Attribution
The Chinese University of Hong Kong, Shenzhen · Lero · The Chinese University of Hong Kong
Semantic Early-Stopping for Iterative LLM Agent Loops
Jun 25, 2026Sahil ShrivastavaEarly Stopping LLM Agent Evaluation
Semantic Halting Problem (SHP) Project
What the LLM Should Not Say: Boundary-Aware Context Grounding for A Seven-Channel EEG Agent
Jun 25, 2026Zhiyuan Xu, Yueqing Dai, Junling Li +1Electroencephalography LLM Grounding
Shanghai Pulse Element Intelligent Technology Co., Ltd. (NeuraDock)
Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents
Jun 25, 2026Praneeth Narisetty, Shiva Nagendra Babu Kore, Uday Kumar Reddy Kattamanchi +1LLM Agent Evaluation LLM Agent Security
LaunchSafe Research
Instruction Bleed: Cross-Module Interference in Prompt-Composed Agentic Systems
Jun 24, 2026Ching-Yu Lin, Yifan LiuLLM Agent Evaluation LLM Agent Reliability
University of Illinois Urbana-Champaign, Urbana, IL, USA.

Jul 1, 2026, cs.AIJ/K move · Enter open · S save

World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments

Ananya Mantravadi, Harshit Rajgarhia, Prasanna Desikan, Abhishek Mukherji

Centific Global Solutions, Inc.

LLM Agent Evaluation AI Agent Benchmarks Workflow Automation Reinforcement Learning with Verifiable Rewards RL for LLM Agents

Open paper AskPDF arXiv

Clinical protocol-execution tasks -- checking a lab value, applying a threshold, placing a correctly structured FHIR order -- are natural candidates for RL from world feedback: once clinical SMEs encode decision logic into a verifier, that verifier grades unlimited rollouts without per-episode annotation. But applying RL requires a sound feedback channel and sufficient base capability. We audit MedAgentBench v1/v2, find a 41.7% silent-finish ceiling that makes inaction the RL dominant strategy, and construct \textbf{MedAgentBench-v3 (MAB-v3)} (508 tasks, 8.9% ceiling). Training Qwen3-8B exposes two structural barriers: a \emph{capability ceiling} (10/20 task types have 0% base performance, zero gradient) and a \emph{format-knowledge barrier} (3/20 types require exact clinical codes undiscoverable by exploration). Pure RL reaches 18.2% pass@1 vs.\ 34.1% for rule-based SFT; the 15.9~pp gap is attributable entirely to these barriers. A decision/format-knowledge/lookup taxonomy predicts RL learnability and prescribes the fix: SFT to inject codes, RL to learn conditionals.

Momentum

Related topics

Latest papers 793