LLM Agent Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

90 papers in the last four weeks, up 109% on the four weeks before. 0.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 558

All topics

Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving
May 19, 2026Oussama Zenkri, Oliver BrockLarge Language Model-Based Robot Planning LLM Agent Evaluation
Robotics and Biology Laboratory, Technische Universität Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Robotics Institute Germany (RIG)
Code as Agent Harness
May 18, 2026Xuying Ning, Katherine Tieu, Dongqi Fu +39LLM Agent Orchestration Long-Horizon LLM Agents
AI for Auto-Research: Roadmap & User Guide
May 18, 2026Lingdong Kong, Xian Sun, Wei Chow +17AI-Assisted Scientific Research Scientific Workflow Automation
Awesome AI Auto-Research Team
Causely: A Causal Intelligence Layer for Enterprise AI A Benchmark Study on SRE and Reliability Workflows
May 18, 2026Dhairya Dalal, Endre Sara, Ben Yemini +2Software Engineering AI Agent Benchmarks
Causely
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
May 18, 2026Jiawei He, Jie Jia, Chenbo Liu +4AI Coding Agents LLM Agent Evaluation
Amap, Alibaba Group
STRIDE: A Self-Reflective Agent Framework for Reliable Automatic Equation Discovery
May 18, 2026Jiarui Su, Songjun Tu, Bei Sun +1LLM Self-Refinement AI Agents for Scientific Discovery
Central South University · Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory
VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems
May 17, 2026Hezhe Qiao, Hanghang Tong, Ee-Peng Lim +2Agent Failure Analysis Multi-Agent LLM Systems
Singapore Management University · University of Illinois at Urbana-Champaign · University of Illinois at Chicago
Taming "Zombie'' Agents: A Markov State-Aware Framework for Resilient Multi-Agent Evolution
May 17, 2026Taolin Zhang, Pukun Zhao, Qizhou Chen +5Multi-Agent LLM Systems Multi-Agent Coordination
School of Computer Science and Information Engineering, Hefei University of Technology · Guangdong University of Finance and Economics · East China Normal University +1
ContractBench: Can LLM Agents Preserve Observation Contracts?
May 17, 2026Jicheng Wang, Yifeng He, Zili Wang +3AI Agent Reliability LLM Agent Evaluation
University of California, Davis · University of Southern California · University of Hong Kong
Reliability and Effectiveness of Autonomous AI Agents in Supply Chain Management
May 16, 2026Carol Xuan Long, David Simchi-Levi, Feng Zhu +3AI Agent Reliability Multi-Agent Systems
Harvard University · MIT/Purdue · MIT +2
To Call or Not to Call: Diagnosing Intrinsic Over-Calling Bias in LLM Agents
May 16, 2026Wei Shi, Ziheng Peng, Sihang Li +4Tool-Augmented Language Model Agents Causal Interventions in Language Models
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Renmin University of China +2
R2V Agent: Teaching SLMs When to Ask for Help
May 15, 2026Raghu Vamshi Hemadri, Humaira Firdowse Mohammed, Rishabh Maheshwary +5AI Agent Reliability Tool-Augmented Language Model Agents
New York University Tandon School of Engineering · University of California, San Diego · ServiceNow Research +3
PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures
May 15, 2026Yunan Lu, Luigi Liu, Omar Yahia +2AI Agent Reliability Adversarial Prompt Generation
Columbia University · University of California San Diego · Walmart
The Scaling Laws of Skills in LLM Agent Systems
May 15, 2026Charles Chen, Qiming Yu, Yuhang Gu +12Language Model Scaling Laws LLM Agent Reliability
Evolvent AI Team
paper.json: A Coordination Convention for LLM-Agent-Actionable Papers
May 15, 2026Arquimedes CanedoDocument Information Extraction LLM Agents
PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI
May 15, 2026Keshava Chaitanya, Jahnavi GundakaramLLM Prompting LLM Agent Evaluation
STAR: A Stage-attributed Triage and Repair framework for RCA Agents in Microservices
May 15, 2026Junle Wang, Xingchuang Liao, Wenjun WuAgent Failure Analysis Root Cause Analysis
School of Artificial Intelligence, Beihang University
Runtime-Structured Task Decomposition for Agentic Coding Systems
May 14, 2026Shubhi Asthana, Bing Zhang, Chad DeLuca +2Software Engineering Agents Agentic Workflows
IBM Research · San Jose, California · Zoom +2
Trustworthy Multi-Agent Systems: Mitigating Semantic Drift with the Argent Signaling Protocol
May 14, 2026Anantha SharmaAI Agent Reliability Multi-Agent LLM Systems
Synechron Inc · Charlotte, NC, USA
Good to Go: The LOOP Skill Engine That Hits 99% Success and Slashes Token Usage by 99% via One-Shot Recording and Deterministic Replay
May 14, 2026Xiaohua Wang, Kai Yu, XuXiao Liang +2LLM Inference Efficiency AI Agent Reliability
1Artificial Intelligence Research Center, Bengbu Medical University, Bengbu 233000, China. · 2CHARMMIRAEL Biotech Co., Ltd, Nanjing 210000, China.
ChromaFlow: A Negative Ablation Study of Orchestration Overhead in Tool-Augmented Agent Evaluation
May 13, 2026Tarun MittalAI Agent Reliability LLM Agent Orchestration
Octave-X
Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use
May 13, 2026Yize Cheng, Chenrui Fan, Mahdi JafariRaviz +2LLM Tool Use LLM Agent Reliability
University of Maryland, College Park
Proof-Carrying Certificates for LLM Pipelines: A Trust-Boundary Architecture
May 13, 2026George KoomullilLLM Grounding LLM Auditing
Ascendr, Inc.
AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents
May 13, 2026Hailin Zhong, Shengxin ZhuSoftware Engineering Agents Agent Evaluation
Hong Kong Baptist University, Hong Kong, China · Beijin Normal University, Zhuhai, China
An Agentic LLM-Based Framework for Population-Scale Mental Health Screening
May 13, 2026Giuliano Lorenzoni, Paulo Alencar, Donald CowanLLM Agent Orchestration Clinical NLP
University of Waterloo · Waterloo, Ontario, Canada
Useful Memories Become Faulty When Continuously Updated by LLMs
May 13, 2026Dylan Zhang, Yanshan Lin, Zhengkun Wu +4AI Agent Reliability Persistent Memory for Language Models
University of Illinois Urbana-Champaign · IIIS, Tsinghua University
When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems
May 13, 2026Young Hyun Cho, Will Wei SunAnytime-Valid Inference Sequential Hypothesis Testing
Department of Statistics, Purdue University. · Mitch Daniels School of Business, Purdue University.
No Action Without a NOD: A Heterogeneous Multi-Agent Architecture for Reliable Service Agents
May 12, 2026Zixu Yang, Hang Zheng, Nan Jiang +5AI Agent Reliability Multi-Agent LLM Systems
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China · 3Jiangsu Key Lab of Language Computing, Suzhou, China
CAX-Agent: A Lightweight Agent Harness for Reliable APDL Automation
May 12, 2026Chenying Lin, Yichen Hai, Yi He +3LLM Agent Orchestration LLM Agent Evaluation
Shanghai Ultradimension Technology Co., Ltd., Shanghai, China · College of Logistics Engineering, Shanghai Maritime University, Shanghai, 201306, China · School of Civil Aviation, Northwestern Polytechnical University, Xi’an 710072, China +3
gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy
May 11, 2026Tousif Islam, Digvijay Wadekar, Zihan ZhouGravitational-Wave Astronomy AI Coding Agents
Kavli Institute for Theoretical Physics, University of California Santa Barbara, Kohn Hall, Lagoon Rd, Santa Barbara, CA 93106 · Center for Gravitational Physics, University of Texas at Austin, Austin, TX 78712, USA · Department of Physics, Princeton University, Princeton, NJ 08540, USA

May 19, 2026, cs.AIJ/K move · Enter open · S save

Probing an Embodied LLM: When Higher Observation Fidelity Hurts Problem Solving

Oussama Zenkri, Oliver Brock

Robotics and Biology Laboratory, Technische Universität Berlin, Germany · Science of Intelligence, Research Cluster of Excellence, Berlin, Germany · Robotics Institute Germany (RIG)

Large Language Model-Based Robot Planning LLM Agent Evaluation Embodied AI Robotic Perception LLM Agent Reliability Embodied Reasoning

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly proposed as cognitive components for robotic systems, yet their opaque decision processes make it difficult to explain success or failure in closed-loop embodied tasks. Following an empirical AI methodology, we study an embodied LLM agent behaviorally by varying the available information and measuring the resulting changes in behavior. Using the Lockbox, a sequential mechanical puzzle with hidden interdependencies, we evaluate LLMs across RGB, RGB-D, and ground-truth symbolic observations in a physical robotic setup and use simulation to probe the resulting behavior. Counterintuitively, agents perform best under raw RGB input and worst under perfect ground-truth observations. In simulation, we probe this effect by randomly flipping perceived action outcomes and find that moderate noise improves performance, peaking at a 40% flip probability with a 2.85-fold success rate increase over the noise-free baseline. Further analysis links this gain to a reduction in repetitive action loops. These findings suggest that success rates alone are insufficient for evaluating LLMs, as measured performance may reflect the interaction between perceptual errors and reasoning failures rather than robust problem solving.

Momentum

Related topics

Latest papers 558