LLM Agent Reliability

LLM: Large Language Model

LLM agent reliability studies whether agents can complete tasks correctly, follow instructions and safety constraints, use evidence faithfully, and behave consistently across repeated or extended interactions. Common approaches combine structured harnesses and deterministic tool interfaces with state tracking, evidence checks, runtime monitoring, calibrated abstention, and recovery from failures; training methods also teach agents to verify actions or learn from execution feedback. Evaluations increasingly use interactive benchmarks, noisy or changing tools, repeated trials, and trajectory-level measures alongside final task success. A central tension is that added reasoning, delegation, memory, or orchestration can improve some tasks but also introduce new failure paths, so reliability depends on the model, harness, tools, and evaluation setting together.

Latest papers

847 papers181 papers in the last four weeks, up 93% on the four weeks before. 1.0% of all new papers.

Diagnosing CFG Interpretation in LLMs
Apr 22, 2026Hanqi Li, Lu Chen, Kai YuLLM Evaluation In-Context Learning
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 3Shanghai Innovation Institution, Shanghai, China · 4Jiangsu Key Lab of Language Computing, Suzhou, China +2
SWE-chat: Coding Agent Interactions From Real Users in the Wild
Apr 22, 2026Joachim Baumann, Vishakh Padmakumar, Xiang Li +3Coding Agents Human-AI Collaboration
Stanford University
Toward Reliable Design of LLM-Enabled Agentic Workflows: Optimizing Latency-Reliability-Cost Tradeoffs
Apr 21, 2026Ya-Ting Yang, Quanyan ZhuCost-Aware Inference Agentic Workflows
New York University Brooklyn, NY, USA
CogniConsole: Externalizing Inference-Time Control as a Formal Abstraction for Reliable LLM Interactions
Apr 21, 2026Vanessa Figueiredo, Wilter FranceschiAI Agent Reliability Language Model-Based Control
Orbital Sea Regina, SK, Canada
Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment
Apr 21, 2026Bobo Li, Rui Wu, Zibo Ji +5AI Agent Benchmarks Multi-Agent Reasoning
National University of Singapore · Sichuan University · University of Minnesota Twin Cities +2
Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery
Apr 21, 2026Abhinav AgarwalSoftware Vulnerability Detection LLMs for Cybersecurity
AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3
JTPRO: A Joint Tool-Prompt Reflective Optimization Framework for Language Agents
Apr 20, 2026Sandip Ghoshal, Anshul Mittal, Jyotika Singh +9AI Agent Reliability LLM Tool Use
Oracle AI
Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots
Apr 20, 2026Shiquan Zhang, Tianyi Zhang, Le Fang +3Mobile GUI Automation LLM Agent Evaluation
School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · School of Computer Science, University of Auckland, Auckland, New Zealand
Towards Self-Improving Error Diagnosis in Multi-Agent Systems
Apr 19, 2026Jiazheng Li, Emine Yilmaz, Bei Chen +1Multi-Agent LLM Systems Multi-Agent Systems
King’s College London · Amazon Alexa AI · University College London
Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity
Apr 19, 2026Leon Engländer, Sophia Althammer, Ahmet Üstün +2LLM Agent Evaluation Tool-Using Agents
Cohere · Poolside
Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs
Apr 19, 2026Carissa Cullen, Harry Garland, Alexander Roman +3Agent Reliability AI Agent Safety
University of Oxford · University College London · New College of Florida +3
Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems
Apr 19, 2026Tianyi Huang, Samuel Xu, Jason Tansong Dang +2Uncertainty Calibration LLM Agent Reliability
Ryquo · App-In Club.
Seeing Isn't Believing: Mitigating Belief Inertia via Active Intervention in Embodied Agents
Apr 19, 2026Hanlin Wang, Chak Tou Leong, Jian Wang +1Embodied AI LLM Agent Reliability
Department of Computing, The Hong Kong Polytechnic University · College of Computer Science, Sichuan University
HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads
Apr 18, 2026Justice Owusu Agyemang, Jerry John Kponyo, Obed Kwasi Somuah +3LLM Inference Efficiency AI Coding Agents
Sperix Labs · VIA Cybersecurity Lab, KNUST · Quantum and Assistive Technologies Lab, KNUST
Harness as an Asset: Enforcing Determinism via the Convergent AI Agent Framework (CAAF)
Apr 18, 2026Tianbao ZhangSafety-Critical Control LLM Guardrails
Independent Researcher
Know When to Trust the Skill: Delayed Appraisal and Epistemic Vigilance for Single-Agent LLMs
Apr 17, 2026Eren UnluLLM Agents LLM Uncertainty Estimation
Globeholder Paris, France
Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents
Apr 17, 2026Eren UnluLLM Agent Evaluation LLM Refusal Behavior
Globeholder Paris, France
When Agents Go Quiet: Output Generation Capacity and Format-Cost Separation for LLM Document Synthesis
Apr 17, 2026Justice Owusu Agyemang, Michael Agyare, Miriam Kobbinah +2LLM Inference Efficiency Long-Form Document Generation
Sperix Labs · GCTU, Ghana · NCA, Ghana +1
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
Apr 17, 2026Bhaskar GurramAI Agent Reliability Tool-Augmented Language Model Agents
Zasti Inc., Ashburn, VA, USA
Weak-Link Optimization for Multi-Agent Reasoning and Collaboration
Apr 17, 2026Haoyu Bian, Chaoning Zhang, Jiaquan Zhang +4Multi-Agent Collaboration Inference-Time Compute Allocation
University of Electronic Science and Technology of China, Chengdu 611731, China · Beihang University, Beijing 100191, China · Xi’an University of Architecture and Technology, Xi’an 710064, China
LLMs Corrupt Your Documents When You Delegate
Apr 17, 2026Philippe Laban, Tobias Schnabel, Jennifer NevilleLLM Reliability Long-Horizon LLM Agents
Microsoft Research
The cognitive companion: a lightweight parallel monitoring architecture for detecting and recovering from reasoning degradation in LLM agents
Apr 15, 2026Rafflesia Khan, Nafiul Islam KhanAI Agent Monitoring LLM Agent Reliability
Software Developer, IBM Dublin, Ireland · Diploma of Computer Science and Engineering Citi Polytechnic Institute Khulna, Bangladesh
Towards Robust Real-World Spreadsheet Understanding with Multi-Agent Multi-Format Reasoning
Apr 14, 2026Houxing Ren, Mingjie Zhan, Zimu Lu +4Multi-Agent LLMs LLM Agent Reliability
CUHK MMLab · SenseTime Research · Shenzhen Loop Area Institute +1
The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
Apr 13, 2026Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun +7Long-Horizon Agent Tasks Agent Failure Analysis
University of Wisconsin–Madison · University of California, Berkeley · Georgia Institute of Technology
From Agent Loops to Structured Graphs:A Scheduler-Theoretic Framework for LLM Agent Execution
Apr 13, 2026Hu WeiLLM Agent Orchestration LLM Agents
Guardrails Beat Guidance: A Large-Scale Study of Rules, Skills, and Persistent Configuration for Coding Agents
Apr 13, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Software Engineering Agents LLM Agent Reliability
AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China.
Resilient Write: A Six-Layer Durable Write Surface for LLM Coding Agents
Apr 12, 2026Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah +2LLM Agent Reliability AI Coding Agents
Sperixlabs, Ghana · Kwame Nkrumah University of Science and Technology, Kumasi, Ghana · VIA Cybersecurity Lab, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana
From Helpful to Trustworthy: LLM Agents for Pair Programming
Apr 11, 2026Ragib Shahariar AyonAI-Assisted Software Development Software Engineering Agents
Texas State University, San Marcos, TX, USA
Verify Before You Commit: Towards Faithful Reasoning in LLM Agents via Self-Auditing
Apr 9, 2026Wenhao Yuan, Chenchen Lin, Jian Chen +3Faithfulness of Language Model Reasoning LLM Agent Reliability
The University of Hong Kong · Sun Yat-sen University

Apr 22, 2026J/K move · Enter open · S save

Diagnosing CFG Interpretation in LLMs

Hanqi Li, Lu Chen, Kai Yu

1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 3Shanghai Innovation Institution, Shanghai, China · 4Jiangsu Key Lab of Language Computing, Suzhou, China+2

LLM Evaluation In-Context Learning LLM Agent Reliability Compositional Generalization

Open paper AskPDF arXiv

As LLMs are increasingly integrated into agentic systems, they must adhere to dynamically defined, machine-interpretable interfaces. We evaluate LLMs as in-context interpreters: given a novel context-free grammar, can LLMs generate syntactically valid, behaviorally functional, and semantically faithful outputs? We introduce RoboGrid, a framework that disentangles syntax, behavior, and semantics through controlled stress-tests of recursion depth, expression complexity, and surface styles. Our experiments reveal a consistent hierarchical degradation: LLMs often maintain surface syntax but fail to preserve structural semantics. Despite the partial mitigation provided by CoT reasoning, performance collapses under structural density, specifically deep recursion and high branching, with semantic alignment vanishing at extreme depths. Furthermore, "Alien" lexicons reveal that LLMs rely on semantic bootstrapping from keywords rather than pure symbolic induction. These findings pinpoint critical gaps in hierarchical state-tracking required for reliable, grammar-agnostic agents.