LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

Will the Agent Recuse Itself? Measuring LLM-Agent Compliance with In-Band Access-Deny Signals
Jun 4, 2026Thamilvendhan MunirathinamLLM Agent Evaluation LLM Refusal Behavior
CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments
Jun 4, 2026Jiaju Chen, Bo Sun, Yuxuan Lu +3Multi-Agent LLM Systems Multi-Agent Coordination
Northeastern University · Microsoft Research Asia
ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs
Jun 4, 2026Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber +1Tool-Augmented Language Model Agents LLM Auditing
SAP Labs
Evaluating Agentic Configuration Repair for Computer Networks
Jun 4, 2026Rufat Asadli, Benjamin Hoffman, Ioannis Protogeros +1LLM Agent Evaluation AI Agent Benchmarks
Department of Information Technology and Electrical Engineering, ETH Zurich.
Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows
Jun 4, 2026Yuhang Fu, Ruishan Fang, Jiaqi Shao +4Multi-Agent LLM Systems LLM Agent Evaluation
1Beijing University of Posts and Telecommunications · *Work was done during Yuhang’s internship in Westlake University. · 3Zhejiang University +4
Enhancing Software Engineering Through Closed-Loop Memory Optimization
Jun 4, 2026Xuehang Guo, Zora Zhiruo Wang, Qingyun Wang +2Software Engineering Agents LLM Agent Evaluation
1William & Mary · 2Carnegie Mellon University · 3OpenHands +1
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
Jun 4, 2026Jiayu Liu, Cheng Qian, Zhenhailong Wang +10LLM Planning LLM Agent Evaluation
University of Illinois Urbana-Champaign
SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations
Jun 4, 2026Taewon Yun, Hyeonseong Park, Jeonghwan Choi +3LLM Agent Evaluation AI Agent Benchmarks
1KAIST · 2Chungnam National University
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
Jun 4, 2026Kaixuan Liu, Guojun Xiong, Weinan Zhang +1World Model Learning Off-Policy Evaluation
Department of Computer Science Emory University Atlanta, GA, USA · School of Computer Science Shanghai Jiao Tong University Shanghai, China
AURA: Intent-Directed Probing for Implicit-Need Surfacing in Situated LLM Agents
Jun 4, 2026Yang Li, Jiaxiang Liu, Jiang Cai +1LLM Agent Evaluation Intent Classification
Guangdong Institute of Intelligence Science and Technology
ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?
Jun 4, 2026Woojung Song, Nalim Kim, Sangjun Song +3Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models
Graduate School of Data Science, Seoul National University
ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developer
Jun 4, 2026Jintao Huang, Xiaomin Li, Gaurav Mittal +1AI Coding Agents LLM Agent Evaluation
The Ohio State University · Microsoft
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Jun 3, 2026Cheng Liang, Pengcheng Qiu, Ya Zhang +3LLM Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
How Far Did They Go? The Persuasive Tactics of Covert LLM Agents in a Discontinued Field Experiment
Jun 3, 2026Kokil Jaidka, Saifuddin AhmedLLM Auditing LLM Agent Evaluation
Department of Communications and New Media, & Centre for Trusted Internet & Community, National University of Singapore, Singapore · Wee Kim Wee school of Communication and Information, Nanyang Technological University, Singapore
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Jun 3, 2026Haoyu Sun, Wenxuan Wang, Mingyang Song +5Tool-Augmented Language Model Agents LLM Agent Evaluation
Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology +5
Parthenon Law: A Self-Evolving Legal-Agent Framework
Jun 3, 2026Hejia Geng, Leo LiuLLM Agent Evaluation LLM Agent Self-Improvement
Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation
Jun 3, 2026Yongjie Wang, Xinyue Zhang, Kunhong Yao +4Deep Research Agents LLM Agent Evaluation
Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · College of Computing & Data Science · Tongyi Lab, Alibaba Group
Exploring Cross-Scenario Generality of Agentic Memory Systems: Diagnostics and a Strong Baseline
Jun 3, 2026Zhikai Chen, Jialiang Gu, Junyu Yin +6Agent Memory LLM Agent Memory
Michigan State University · George Mason University · Purdue University
SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
Jun 2, 2026Joel Sol, Homayoun NajjaranMulti-Agent Coordination LLM Agent Evaluation
Faculty of Engineering and Computer Science University of Victoria Victoria, BC, Canada
MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents
Jun 2, 2026Jia Yu, Zilong Wang, Xinyang Jiang +2Computer-Use Agent Benchmarks Computer-Use Agents
Microsoft Research Asia, Shanghai, China · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China · Shanghai Key Laboratory of MICCAI, Shanghai, China
The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberation
Jun 2, 2026Herun Wan, Jiaying Wu, Minnan Luo +4Multi-Agent LLM Systems LLM Agent Evaluation
Xi’an Jiaotong University · National University of Singapore · Yunnan University +1
Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks
Jun 1, 2026Dipesh KC, Anjila BudathokiCoding Agents Software Engineering Benchmarks
Independent Researcher · Georgia State University
The Epi-LLM Framework: probing LLM behavioral priors through epidemiological agent-based models
Jun 1, 2026Petra Ferencz, Ava Keeling, Tobias O'Keefe +4Human Behavior Simulation LLM Agent Evaluation
Big Data Institute, Li Ka Shing Center for Health Information and Discovery, University of Oxford, Oxford, United Kingdom · Leverhulme Centre for Demographic Science, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom · Pandemic Sciences Institute, Nuffield Department of Medicine, University of Oxford, Oxford, United Kingdom +3
$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues
Jun 1, 2026Peixuan Han, Hongyi Du, Jiayu Liu +3LLM Personalization LLM Agent Evaluation
University of Illinois Urbana-Champaign
HERO'S JOURNEY: Testing Complex Rule Induction with Text Games
Jun 1, 2026Anshun Asher Zheng, Kanishka Misra, David I. Beaver +1LLM Agent Evaluation AI Agent Benchmarks
Department of Linguistics The University of Texas at Austin
MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation
Jun 1, 2026Wenhao Wang, Peizhi Niu, Gongyi Zou +9LLM Agent Evaluation AI Agent Benchmarks
Multi-Agent Governance & Intelligence Crew (MAGIC), Shanghai Jiao Tong University, Shanghai, China · Department of Computer Science and Technology, Zhejiang University, Hangzhou, China · University of Illinois at Urbana-Champaign, Illinois, USA +2
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Jun 1, 2026Nahyun Lee, Dongkeun Yoon, Guijin Son +12Computer-Use Agent Benchmarks LLM Evaluation
Chung-Ang University1 · KAIST2 · Seoul National University3 +3
SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
Jun 1, 2026Yuyan Bu, Haowei Li, Qirui Zheng +7LLM Agent Evaluation AI Agent Safety
Beijing Academy of Artificial Intelligence · University of Science and Technology of China · Peking University +2
Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains
Jun 1, 2026Garvin Guo, Donglei Yu, Yu Chen +6Tool-Use Evaluation LLM Agent Evaluation
University of Chinese Academy of Sciences · Tongyi Lab, Alibaba Group · Amap, Alibaba Group +1
POIROT: Interrogating Agents for Failure Detection in Multi-Agent Systems
Jun 1, 2026Iñaki Dellibarda Varela, R. Sendra-Arranz, Pablo Romero-Sorozabal +5Agent Failure Analysis AI Agent Reliability
Center for Automation and Robotics, Spanish National Research Council (CSIC-UPM), Madrid, Spain. · ETSI Telecomunicación, Universidad Politécnica de Madrid (UPM), Madrid, Spain.

Jun 4, 2026, cs.CRJ/K move · Enter open · S save

Will the Agent Recuse Itself? Measuring LLM-Agent Compliance with In-Band Access-Deny Signals

Thamilvendhan Munirathinam

LLM Agent Evaluation LLM Refusal Behavior AI Agent Governance Tool Access Control for LLM Agents

Open paper AskPDF arXiv

As autonomous LLM agents increasingly hold real credentials and operate infrastructure without a human in the loop, operators have no standard way to tell an agent that a resource is off-limits. Access controls either let the agent in (it has valid credentials) or hard-fail it (indistinguishable from any other client). We propose a third mode: a lightweight, published in-band deny signal -- the Recuse Signal -- that a server emits over a protocol's existing channels (an SSH banner, a PostgreSQL NOTICE) asking a connecting automated agent to voluntarily withdraw. This is a cooperative governance control, the robots.txt analogue for live access; it is explicitly not a security boundary. Its value is entirely empirical and, to our knowledge, unmeasured: do compliant LLM agents actually honor such a signal? We define the signal as an open mini-standard, implement three zero- or low-footprint adapters (an SSH banner/PAM hook, a PostgreSQL wire-protocol proxy, and a Kubernetes admission webhook), deploy them on a live production host, and run a controlled experiment in which fresh agents are given a benign operations task and observed for recusal. In a pilot (SSH; OpenAI GPT-4o and GPT-4o-mini; and Claude Code as a deployed agent), the signal cleanly induces recusal -- 100% recusal when present versus 100% task completion in a no-signal control -- and, revealingly, behaves as a cooperative rather than absolute signal: an explicit operator-authorization framing flips the most capable model to proceed, while other agents continue to defer to the on-host policy. We release the standard, adapters, and experiment harness for reproduction.

Momentum

Related topics

Latest papers 793