LLM Agent Reliability

LLM: Large Language Model

LLM agent reliability studies whether agents can complete tasks correctly, follow instructions and safety constraints, use evidence faithfully, and behave consistently across repeated or extended interactions. Common approaches combine structured harnesses and deterministic tool interfaces with state tracking, evidence checks, runtime monitoring, calibrated abstention, and recovery from failures; training methods also teach agents to verify actions or learn from execution feedback. Evaluations increasingly use interactive benchmarks, noisy or changing tools, repeated trials, and trajectory-level measures alongside final task success. A central tension is that added reasoning, delegation, memory, or orchestration can improve some tasks but also introduce new failure paths, so reliability depends on the model, harness, tools, and evaluation setting together.

Latest papers

843 papers181 papers in the last four weeks, up 93% on the four weeks before. 1.0% of all new papers.

Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success
Oct 29, 2026Amir H. Rezaeian, Weiyi Sun, Yassine Benajiba +1Tool-Use Evaluation Tool-Using Agents
Oracle AI
Locating and Closing Information Gaps in Multi-Agent Systems
Oct 29, 2026Yaxin Gong, Chongming Gao, Gangyi Zhang +7LLM Agent Workflow Optimization Multi-Agent Systems
University of Science and Technology of China · Qwen Business Unit of Alibaba · National University of Singapore
AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access Barriers
Oct 29, 2026Zhiyi Liu, Haohua Du, Haoqi Sun +1Web Agents LLM Agent Reliability
University of Science and Technology of China Hefei, China · Beihang University Beijing, China · Peking University Beijing, China
Think Before You Act: Decision Gate for LLM Agents
Oct 28, 2026Iftach Shoham, Yuval David, Offer Akrabi +5LLM Agent Verification Runtime Enforcement for AI Agents
IBM SIL
Evaluating and improving AI agents for sequencing protocol understanding
Oct 27, 2026Chi-Lam Poon, Michael Goneos, Zezheng Song +1Scientific Agent Benchmarks Scientific Reasoning in Language Models
Memorial Sloan Kettering Cancer Center · Weill Cornell Medicine · NASA Ames Research Center +2
Grounding LLM Agents in Business Processes: Process-Aware Knowledge Graphs for Reliable Multi-Step Service Invocation
Oct 27, 2026Saiful Khan, Afrah FareaEnterprise AI Agents Tool Discovery for LLM Agents
Scientific Computing, Science and Technology Facilities Council, RAL, OX11 0QX, UK. · Department of Engineering Science, University of Oxford, Parks Road, OX1 3PJ, UK.
CASEL: Consequence-Aware Selective Execution for Reliable Long-Horizon Tool-Using Agents
Oct 27, 2026Yanyan Zhang, Fei Shen, Ting Yan +1Runtime Enforcement for AI Agents Tool-Using Agents
Carnegie Mellon University Pittsburgh, PA 15213, USA · Feinity LLC Sacramento, CA 95816, USA · Computer Science Department Dartmouth College Hanover, NH 03755, USA +1
When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents
Oct 27, 2026Chiming Duan, Tong Jia, Yingxi Chen +3LLM Agents LLM Agent Reliability
Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China
Work Agents: Systems, Capabilities, and Optimization
Oct 26, 2026Zishan Xu, Enjun Du, Mingqi Gao +13LLM Agent Reliability LLM Agent Workflow Optimization
Shanghai Jiao Tong University · The University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) +5
Quantifying Claim Smoothing by LLM Research Agents
Oct 26, 2026Tiasa Singha Roy, Manya Wadhwa, Greg DurrettLLM Reliability LLM Agent Reliability
New York University
Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
Oct 26, 2026Tingzhu Bi, Xun Zhang, Jinglin Li +2Root Cause Analysis LLM Agent Harnesses
Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China
CIVER: Causal Intervention Verification for Reliable Code Review
Oct 26, 2026Xinyuan Song, Hasibul Haque, Jay Somasundaram +3Automated Code Review LLM Agent Reliability
Causal Dynamics Lab USA
Benchmarking Coding Agents under Evolving Requirements
Oct 26, 2026Jia Feng, Alyssa Wu, Cuiyun Gao +3Coding Agent Benchmarks Code Generation
Hunyuan Team, Tencent · The Chinese University of Hong Kong
Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation
Oct 26, 2026Atri Vivek Sharma, Brian Formento, Alessio LomuscioLLM Agent Evaluation User Simulation
Imperial College London · Safe Intelligence
RepoTrace: Improving Requirement Tracing in Multi-Round Agent Interactions
Oct 26, 2026Huanyu Liu, Yezheng Cheng, Jinliang You +5Software Engineering Agents LLM Agent Reliability
Key Laboratory of High Confidence Software Technologies, Ministry of Education; School of Computer Science, Peking University · Xidian University · School of Cyber Science and Engineering, Wuhan University
Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents
Oct 26, 2026Ying Zhang, Ruoqi Guo, Yi Liu +4LLM Agent Reliability LLM Agent Efficiency
Wake Forest University Winston-Salem, USA · Griffith University Gold Coast, Australia · Nanyang Technological University Singapore +2
R2A-Sales: A Policy-Atom-Matched Benchmark for Rule-to-Action Gaps in Goal-Directed Sales Agents
Oct 25, 2026Yihua Qin, Jiajie Jerry ZhaoLLM Agent Evaluation LLM Agent Reliability
Tsinghua University · Futura Lab, Futura.AI
Strategic Calibration in LLM Multi-Agent Systems: Audited Pivot Scoring for Truthful Elicitation
Oct 24, 2026Jinhao Zhou, Zhaohan Zhang, Xijian Xu +1Multi-Agent LLM Systems LLM Agent Reliability
Graduate School of Information, Production and Systems, Waseda University, Fukuoka, Japan · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, U.K. · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Estimation of Propagated Risk in LLM-based Multi-Agent Systems
Oct 23, 2026Shreyan Ganguly, Souradeep Chattopadhyay, Qisai Liu +1Multi-Agent LLM Systems LLM Agent Reliability
Iowa State University
Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance
Oct 23, 2026Antonio Ken Iannillo, Joshua Segun Owotogbe, Roberto Natella +3LLM Agent Reliability LLM Agent Evaluation
SnT, University of Luxembourg · Tilburg University · Gran Sasso Science Institute +1
Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents
Oct 23, 2026Detao Zeng, Guilin Ou, Nuo Chen +5LLM Agent Evaluation AI Agent Benchmarks
University of Electronic Science and Technology of China
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
Oct 23, 2026Zhiyong Cao, Hengjie Zhu, Yongxin Wang +20LLM Agent Harnesses LLM Agent Reliability
Institute of Automation, CAS · Zhongguancun Academy · Institute of Information Engineering, CAS +2
SWE-Shift: Evaluating Coding Agents in Evolving Workspaces
Oct 23, 2026Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin +7Coding Agents Software Engineering Benchmarks
MMLab, The Chinese University of Hong Kong · Somewhere on the Earth · Fudan NLP Lab, Fudan University +2
Who Verifies the Tools? How Warped Formatters Mislead Agents and Their Verifiers
Oct 23, 2026William JeffriesLLM Agent Reliability LLM Agent Verification
Understanding Communication Failures in Multi-Agent LLM Systems
Oct 23, 2026Priya Pitre, Jun Hou, Yi Fang +2Multi-Agent LLM Systems Multi-Agent Communication
Virginia Tech, Blacksburg, VA 24061, USA
Phronix: The Capability-Centric Learning Paradigm for Agents
Oct 22, 2026Hariharan Devarajan, Neeraj Rajesh, Aniruddha Marathe +3LLM Agent Skill Learning LLM Agent Harnesses
Lawrence Livermore National Laboratory · Illinois Institute of Technology · University of Tennessee, Knoxville
EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence
Oct 22, 2026Qijin Zhang, Hanzhen Wang, Yuehan Zhao +4AI Agent Safety Benchmarks LLM Agent Safety
City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
Adaptive Skill Granularity for Coding Agents: Runtime Routing and Macro-to-Micro Recovery
Oct 22, 2026Dingxuan Zhang, Yanhui Jiang, Jun Wang +1Coding Agents LLM Agent Workflow Optimization
Shanghai Jiao Tong University, China · Kuaishou Technology Co., Ltd., China · University of Warwick, UK
Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash
Oct 22, 2026Atharv Chandratre, Jai DatarAutomated Software Engineering Workflow Automation
DoorDash, Inc., San Francisco, CA, USA
Consistent but Biased: Chair Personas Trade Noise for Bias in Multi-Agent Clinical Decisions
Oct 22, 2026Yitong Bai, Chengyi Zhang, Ziyang WangLLM Agents Clinical Decision-Making
Columbia University · Swansea University · Aston University

Oct 29, 2026J/K move · Enter open · S save

Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success

Amir H. Rezaeian, Weiyi Sun, Yassine Benajiba, Dan Roth

Oracle AI

Tool-Use Evaluation Tool-Using Agents Process-Based Agent Evaluation Tool Use LLM Agent Reliability

Open paper AskPDF arXiv

Tool-use benchmarks often score agents by final task success, but terminal outcomes can obscure whether the intermediate tool-call trajectory was executable. We propose prefix executability as a trajectory-level evaluation protocol for tool-using agents. The protocol measures whether every prefix of a predicted tool-call sequence remains executable under tool schemas, state preconditions, and data dependencies, and summarizes reliability using Prefix Executability Curves (PEC), First-Error Depth (FED), Trajectory Executability Rate (TER), horizon-normalized AUC-PEC, and Recovery Rate (RR). We instantiate the protocol on stratified samples from BFCL-v3 multi-turn and ToolSandbox, and on the complete Airline/Retail base splits of

τ^3

-bench. We observe that outcome-only and prefix-level rankings diverge: agents whose execution is conditioned on an unverified plan often achieve higher point-estimate final success than non-plan-conditioned baselines while substantially lowering TER and AUC-PEC. These divergences are precisely the failure modes that prefix executability is designed to expose: they reveal whether apparent success rests on an executable trajectory, where invalidity first enters, and whether success follows recovery from an earlier failure. As a validation intervention, explicit pre-execution plan checking can improve executable-prefix survival while maintaining competitive terminal success. These results show that final success alone is insufficient for evaluating tool-use agents and that prefix-level executability provides an actionable reliability signal for comparing agents, diagnosing failures, and assessing validation mechanisms.