LLM Agent Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

90 papers in the last four weeks, up 109% on the four weeks before. 0.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 558

All topics

Diagnosing with Insights: Structured Analysis of Agent Failures via Behavioral Abstractions
Sep 2, 2026Jiayi Bi, Yanjie Gao, Yuanmin Xie +4Fault Localization LLM Agent Reliability
1Tsinghua University · 2Microsoft Research · 3Microsoft +1
The Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory Agents
Sep 1, 2026Jundong Hu, Shekar RamachandranKnowledge Conflicts in Language Models LLM Agent Memory
PayPal AI
Agents That Model Agents: Five Principles Toward a Theory of Mind for 6G Networks
Sep 1, 2026Hatim Chergui, Carolina Fernández-Martínez, Mehdi Bennis +16G Networks Theory of Mind
i2CAT Foundation, Spain · University of Oulu, Finland · Research Institute for Digital Future, Khalifa University, 127788 Abu Dhabi, UAE
HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution
Sep 1, 2026Wen Jiang, Mingmin Chu, Yimeng Tian +6Agent Harness Optimization LLM Agent Reliability
ICT AI Competence Center, Huawei Technologies, Shanghai, China
Polished but Unresolved: Identifying Late-Stage Pressure States in Long-Horizon Tool-Use Agents
Sep 1, 2026Haoyang Chen, Yi Liu, Jianzhi Shao +3LLM Agents Long-Horizon LLM Agents
State Key Laboratory for Novel Software Technology, Nanjing University, China · Alibaba Group, China
Self-Reports Are Not Verification: Environment-Grounded Auditing of LLM Operators in Evolutionary Search
Sep 1, 2026Enrong Pan, Ryan Zhou, Ting HuLLM Auditing Language Model Self-Assessment
Queen’s University
Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
Sep 1, 2026Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi +3Computer-Use Agents Human-AI Interaction
Stony Brook University · Old Dominion University
Invalidation Contracts for Cross-Episode Agent Memory
Aug 31, 2026Michael Wu, Arquimedes CanedoLLM Agent Memory Agent Memory Management
Siemens Digital Industries Software Princeton, NJ, USA
Don't Let the Model Write the YAML: Deterministic, Minimal-Diff GitOps Remediation from LLM-Proposed Field Changes
Aug 31, 2026Pruthvi DavineniLLM Agent Reliability
Independent Researcher
Lazy Grounding: Attacking Search Agents with Factual Evidence
Aug 31, 2026Yulin Zhang, Yukun Huang, Sanxing Chen +4Evidence-Grounded Reasoning RAG Poisoning Attacks
Duke University, Durham, NC 27708, USA
CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
Aug 31, 2026Amir Saeidi, Zehua Zhang, Rishitosh Singh +6LLM Agents LLM Agent Training
Arizona State University · Cisco Research
Source-Dependent Deference in Medical Imaging Agents Under Falsified Findings: A Pilot Audit
Aug 30, 2026Ridam Roy, Md Shahriar Rashid, Md. Rajib MiaTool-Use Evaluation Medical VQA
Daffodil International University Savar, Dhaka, Bangladesh · Boise State University Boise, Idaho, USA
Detect Before You Attribute: Cascade Failure Attribution for Multi-Agent Systems
Aug 30, 2026Jiayi Zhang, Zexin Wang, Degang Sun +4Agent Failure Analysis Multi-Agent LLM Systems
Computer Network Information Center, Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
Logos: An Agent Harness on a Cross-Process Bus
Aug 28, 2026Hanzhang Jia, Liheng Zeng, Hao Cheng +2Multi-Agent Systems LLM Agent Harnesses
University of Sussex Brighton, United Kingdom · Zhejiang Gongshang University Hangzhou, China · Shanghai Shuyuan Information Technology Co., Ltd. Shanghai, China
EvoUndo: Recoverability-Constrained Self-Evolution for LLM Agent Harnesses
Aug 28, 2026Tanmay Sah, Dolly Sah, Harshul Jain +1LLM Agent Self-Improvement LLM Agent Harnesses
Independent Researcher
Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
Aug 26, 2026Jia-Hao Ji, Sijie Li, Jiabei Cheng +3LLM-as-a-Judge Multi-Agent Reasoning
Department of Neurology and National Center for Neurological Diseases, Huashan Hospital, State Key Laboratory of Medical Neurobiology and Ministry of Education Frontiers Center for Brain Science, Shanghai Medical College, Fudan University, Shanghai 200040, China. · Institute of Science and Technology for Brain-Inspired Intelligence, MOE Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, MOE Frontiers Center for Brain Science, Fudan University, Shanghai 200433, China. · Shanghai Academy of Artificial Intelligence for Science, Shanghai 200003, China. +1
JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
Aug 26, 2026Guibin Zhang, Leo Lu, Fangzhou Xie +15Tool-Augmented Language Model Agents Agent Harness Optimization
JIT-Agent Team
TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents
Aug 24, 2026Wenhao Wu, Menghao Zhang, Xin Wang +3Tool-Augmented Language Model Agents LLM Agent Orchestration
Xiaomi Inc. · Nanjing University · Tsinghua University
Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses
Aug 24, 2026Jiachen Xu, Torben Bach Pedersen, Zhongming Yao +2Agent Failure Analysis AI Agent Reliability
Department of Computer Science Aalborg University Aalborg, Denmark · College of Computer Science and Technology Zhejiang University Hangzhou, China · School of Information Science and Engineering Northeastern University Shenyang, China
K-Bench: measuring model performance on real scientific agent requests
Aug 21, 2026Aubrey M. Brueckner, Darshil Patel, Yuhuan He +1AI for Science LLM Agent Evaluation
1K-Dense, Inc.
Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents
Aug 21, 2026Yanze Jiang, Mingxuan Li, Yuhao Wang +2LLM Agents LLM Agent Reliability
National University of Singapore
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Aug 20, 2026Zhipeng Xu, Jiahao Lu, Yining Zheng +2Software Engineering Automated Program Repair
Shanghai Innovation Institute · Fudan University
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
Aug 20, 2026Zhuochun Li, Youngmin Ko, Ali Keramati +11Business Process Automation AI Agent Benchmarks
University of Pittsburgh · Northwestern University · University of California, Irvine +1
AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit
Aug 17, 2026Yuchen Yuan, Zhenghuang Wu, Yuangan Li +2LLM Safety Benchmarks AI Agent Safety Benchmarks
School of Aeronautic Science and Engineering, Beihang University
QuoteBench: How Matched Scores Can Hide Command-Path Failures
Aug 13, 2026Shangao Li, Yao Zhang, Volker Tresp +1LLM Agent Evaluation Agent Evaluation
1Stony Brook University · 2LMU Munich · 3Munich Center for Machine Learning
Reconcile Once, Write Anytime: A Trust-Tiered Librarian and a Multi-Agent Writer for Drift-Free, Point-in-Time Research
Aug 13, 2026Xing Zhang, Yanwei Cui, Guanghui Wang +1Long-Form Document Generation Data Provenance
AWS Generative AI Innovation Center
SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries
Aug 12, 2026Oguz Serdar, Cuneyt MertayakLLM Agent Evaluation Tool-Using Agents
AgentDock
Convergent Detour Hijacking: Task-Preserving Resource Amplification in Skill-Based LLM Agents
Aug 12, 2026Junliang Liu, Ruoyu Li, Wenxin Tang +4Adversarial Attacks on LLMs LLM Agent Security
School of Computer Science and Software Engineering, Shenzhen University · The Chinese University of Hong Kong · The Chinese University of Hong Kong, Shenzhen
Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection
Aug 12, 2026Chaoran Chen, Vy Nguyen, Ziji Zhang +7Curriculum RL AI Agent Benchmarks
Amazon
Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents
Aug 12, 2026Gen Dong, Yanjie Gao, Liqun Li +3LLM Agent Evaluation LLM Agent Reliability
Huazhong University of Science and Technology · Microsoft Research · Microsoft +1

Sep 2, 2026, cs.AIJ/K move · Enter open · S save

Diagnosing with Insights: Structured Analysis of Agent Failures via Behavioral Abstractions

Jiayi Bi, Yanjie Gao, Yuanmin Xie, Liqun Li+3

1Tsinghua University · 2Microsoft Research · 3Microsoft+1

Fault Localization LLM Agent Reliability Neuro-Symbolic Reasoning

Open paper AskPDF arXiv

With the proliferation of LLM agents, the ability to understand and diagnose failures in agents is essential to achieving superior effectiveness and trustworthiness. As agent failures often manifest via long and complex trajectories, manually finding the needles in the haystack is untenable. However, traditional diagnosis techniques for software bugs can hardly address LLM agent failures, while completely relying on LLMs as the judge yields unreliable diagnosis results. To overcome these challenges, this paper presents AGENTSCOPE, a new neuro-symbolic approach for agent failure mode diagnosis. The key principle of AGENTSCOPE is to abstract agent behavior, based on its trajectories, into structured representations. Furthermore, AGENTSCOPE introduces the concept of neural invariants to specify agent behavior properties. AGENTSCOPE leverages LLM-guided reasoning atop the structured representation against neural invariants to pinpoint both the failure step and its type in the trajectory. We show the effectiveness of AGENTSCOPE on publicly available agent failure datasets (Who&When) and a more comprehensive dataset created by us (AgentErrata), where AGENTSCOPE significantly outperforms the current state of the art in fault localization and attribution accuracy. Our work shows that integrating structured abstractions with LLM-guided reasoning enables effective, reliable, and interpretable diagnosis for agent failures.

Momentum

Related topics

Latest papers 558