LLM Agent Reliability

LLM: Large Language Model

LLM agent reliability studies whether agents can complete tasks correctly, follow instructions and safety constraints, use evidence faithfully, and behave consistently across repeated or extended interactions. Common approaches combine structured harnesses and deterministic tool interfaces with state tracking, evidence checks, runtime monitoring, calibrated abstention, and recovery from failures; training methods also teach agents to verify actions or learn from execution feedback. Evaluations increasingly use interactive benchmarks, noisy or changing tools, repeated trials, and trajectory-level measures alongside final task success. A central tension is that added reasoning, delegation, memory, or orchestration can improve some tasks but also introduce new failure paths, so reliability depends on the model, harness, tools, and evaluation setting together.

Latest papers

843 papers181 papers in the last four weeks, up 93% on the four weeks before. 1.0% of all new papers.

Diagnosing Live Within-Policy Instruction Conflicts in LLM Agents with Witnessed Resolution Profiles
May 27, 2026Lu Yan, Xuan Chen, Xiangyu ZhangLLM Agent Evaluation LLM Agents
Purdue University
Detection Without Correction: A Two-Parameter Decomposition of Multi-Stage LLM Pipelines
May 26, 2026Prashanti Nilayam, Kiran Ramanna, Prashil TumbadeLLM Self-Correction LLM Evaluation
Servicenow CA, USA
Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments
May 26, 2026Yuxin Chen, Xiaodong Cai, Junfeng Fang +9LLM Agents LLM Agent Training
National University of Singapore · Meituan · Tsinghua University +2
It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
May 26, 2026Yong-eun ChoAI Agent Reliability LLM Agent Evaluation
KailosLab · Seoul, Republic of Korea
Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems
May 25, 2026Jianing Zhu, Yeonju Ro, John Robertson +5AI Agent Reliability AI Agent Benchmarks
The University of Texas at Austin
VeriTrace: Evolving Mental Models for Deep Research Agents
May 25, 2026Haolang Zhao, Yunbo Long, Lukas Beckenbauer +1Cognitive Modeling Deep Research Agents
Department of Engineering, University of Cambridge · TUM School of Management, Technical University of Munich · The Alan Turing Institute
When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation
May 25, 2026Liyun Zhang, Jiayi GuoCoT Reasoning LLM Agent Evaluation
School of Information and Software Engineering, UESTC · Jacobs School of Engineering, UC San Diego
Meta-Engineering Harnesses for AI-Native Software Production: A Contract-Driven Adversarial Verification Architecture with Early Deployment Report
May 25, 2026Satadru Sengupta, Tamunokorite Briggs, Ivan MyshakivskyiSoftware Engineering Agents Automated Software Engineering
HireNimbus
A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration
May 25, 2026Hiroki FukuiMulti-Agent LLM Systems LLM Evaluation
Research Institute of Criminal Psychiatry · Department of Neuropsychiatry, Graduate School of Medicine, Kyoto University
CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures
May 25, 2026Akash Bonagiri, Devang Borkar, Gerard Janno Anderias +2Agent Failure Analysis Causal Interventions in Language Models
Department of Computer Science University of California, Davis Davis, CA 95616
Meta-Agent: From Task Descriptions to Verified Multi-Agent Systems
May 24, 2026Andy Xu, Yu-Wing TaiMulti-Agent Orchestration Multi-Agent Systems
Dartmouth College
Memory-Induced Tool-Drift in LLM Agents
May 24, 2026Mahavir Dabas, Jihyun Jeong, Ming Jin +1AI Agent Benchmarks LLM Tool Use
Virginia Tech
PRIMA: Operational Patterns for Resilient Multi-Agent Research with Verifiable Identity and Convergent Feedback
May 23, 2026Sasank AnnapureddyMulti-Agent Coordination Multi-Agent Orchestration
CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programming
May 23, 2026Peisong Wang, Bowen Liu, Zehua Li +4Code Generation LLM-Based Program Synthesis
The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Push Your Agent: Measuring and Enforcing Quantitative Goal Persistence in Long-Horizon LLM Agents
May 22, 2026Yuandao Cai, Yuzhang Zhu, Liyou Gao +2Long-Horizon Planning Long-Horizon Agent Evaluation
Independent Researcher · Xidian University
When Planning Fails Despite Correct Execution: On Epistemic Calibration for LLM-Based Multi-Agent Systems
May 22, 2026Zehao Wang, Shilong Jin, Zhao Cao +1Multi-Agent LLM Systems Multi-Agent Planning
College of Intelligence and Computing, Tianjin University, Tianjin, China · School of New Media and Communication, Tianjin University, Tianjin, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
DART: Semantic Recoverability for Structured Tool Agents
May 22, 2026Ke Yang, Panpan Li, Zonghan Wu +3Agent Reliability Runtime Assurance
1MOS Intelligent Connectivity Technology Co. Ltd. · 2Sichuan Vocational College of Post and Telecom · 3East China Normal University +2
More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries
May 21, 2026Hongwen Song, Song WeiLLM Agent Evaluation LLM Agent Skill Learning
Databricks Inc.
Evaluating Large Language Models as Live Strategic Agents: Provider Performance, Hybrid Decomposition, and Operational Gaps in Timed Risk Play
May 21, 2026H. C. EkneLLM Agent Evaluation AI Agent Benchmarks
Diagnosis Is Not Prescription: Linguistic Co-Adaptation Explains Patching Hazards in LLM Pipelines
May 21, 2026Yoon Jeonghun, Kim DongchanLLM Agents Causal Interventions in Language Models
KAIST (Korea Advanced Institute of Science and Technology), Republic of Korea · NAVER Corp., Seongnam, Republic of Korea · NAVER Corp., Bellevue, WA, USA
Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling
May 20, 2026Caleb Winston, Ron Yifeng Wang, Azalia Mirhoseini +1Computer-Use Agents LLM Agent Orchestration
Stanford University, Stanford, CA, USA.
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
May 20, 2026Akshay Manglik, Vijay S. Kalmath, Jason Qin +6Multi-Agent LLM Systems LLM Agent Evaluation
Scale AI
What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems
May 19, 2026Yong Jin Chun, Iftekhar AhmedMulti-Agent Collaboration Multi-Agent Systems
Department of Informatics University of California, Irvine Irvine, CA, 92618
AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
May 19, 2026Parsa Mazaheri, Kasra MazaheriLLM Agent Reliability Computer-Use Agent Benchmarks
University of California, Santa Cruz · Massachusetts Institute of Technology
Modeling Emotional Dynamics in Agent-to-Agent Interactions on Moltbook
May 19, 2026Syed Mhamudul Hasan, Abdur R. ShahidMulti-Agent LLM Systems LLM Agent Evaluation
Southern Illinois University Carbondale, IL, USA
A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
May 19, 2026Vasundra SrinivasanLLM Agent Orchestration LLM Agents
AI Architect, Independent Researcher · Stanford School of Engineering
Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving
May 19, 2026Oussama Zenkri, Oliver BrockLarge Language Model-Based Robot Planning LLM Agent Evaluation
Robotics and Biology Laboratory, Technische Universität Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Robotics Institute Germany (RIG)
Code as Agent Harness
May 18, 2026Xuying Ning, Katherine Tieu, Dongqi Fu +39LLM Agent Orchestration Long-Horizon LLM Agents
AI for Auto-Research: Roadmap & User Guide
May 18, 2026Lingdong Kong, Xian Sun, Wei Chow +17AI-Assisted Scientific Research Scientific Workflow Automation
Awesome AI Auto-Research Team
Causely: A Causal Intelligence Layer for Enterprise AI A Benchmark Study on SRE and Reliability Workflows
May 18, 2026Dhairya Dalal, Endre Sara, Ben Yemini +2Software Engineering AI Agent Benchmarks
Causely

May 27, 2026J/K move · Enter open · S save

Diagnosing Live Within-Policy Instruction Conflicts in LLM Agents with Witnessed Resolution Profiles

Lu Yan, Xuan Chen, Xiangyu Zhang

Purdue University

LLM Agent Evaluation LLM Agents LLM Agent Reliability

Open paper AskPDF arXiv

LLM agents are governed by long-lived natural-language prompt policies, but individually reasonable standing rules can interact in uninspected ways. We study live intra-policy rule-conflict diagnosis: finding rule pairs inside a single prompt policy that can co-govern a realistic state, and measuring how models resolve that pressure in responses or tool actions. We introduce WIRE, a Witnessed Intra-policy Rule Evaluation pipeline. WIRE extracts source-grounded rules, encodes them as PyRule clauses, uses satisfiability checks to retain same-surface hard-collision candidates, realizes those candidates as concrete co-governance witnesses, and judges model outputs against the original source-rule text. Across six public prompt policies, WIRE extracts 276 source rules and 560 atomic clauses, classifies 30,944 within-policy clause-pair comparisons, retains 170 encoded hard-collision candidate source-rule pairs, and realizes them as 1,402 concrete witnesses. In policy-only evaluation, these witnesses yield 13,335 post- generation trials where both source rules govern and both compliance labels are judgeable. Only 35.4% fall in joint compliance; 64.6% violate at least one governed source rule. These profiles are conditional diagnostics for WIRE-selected candidates, not deployment-frequency or causal excess failure estimates, but they reveal distinct policy, model, and tool-action resolution patterns.