LLM Agent Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

90 papers in the last four weeks, up 109% on the four weeks before. 0.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 558

All topics

UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development
Jul 2, 2026Temitayo Olamilekan Ogunsusi, Lijun Qian, Xishuang DongSoftware Engineering Agents Multi-Agent Collaboration
Department of Electrical and Computer Engineering Prairie View A&M University, Prairie View, TX 77446, USA
Atomic Task Graph: A Unified Framework for Agentic Planning and Execution
Jul 2, 2026Yue Zhang, Sihan Chen, Ziwen Huang +3Graph-Based Planning LLM Agent Planning
South China University of Technology · Tsinghua Shenzhen International Graduate School, Tsinghua University
CLAP: Closed-Loop Training, Evaluation, and Release Control for Domain Agent Post-training
Jul 2, 2026Fangfei Li, Chenyang Zhao, Long Wang +3LLM Agent Evaluation LLM Post-Training
MatrixOrigin China · MatrixOrigin USA
When Should Service Agents Reconsider? Difficulty-Routed Control in Customer-Service Operations
Jul 1, 2026Qian Chen, Chengyuan Liu, Xin YuCustomer Support Automation Tool-Using Agents
†Department of Supply Chain and Information Systems · ‡Department of Statistics The Pennsylvania State University
Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use
Jul 1, 2026Song-Lin Lv, Weiming Wu, Rui Zhu +2Fine-Tuning Distribution Shift
School of Intelligence Science and Technology, Nanjing University, Nanjing, China · National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows
Jul 1, 2026Jalal Mahmud, Eser KandoganAgent Failure Analysis Agentic Workflows
Megagon Labs California, USA
PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents
Jul 1, 2026Ke Zhang, Sahchit Chundur, Mohammad Javad Qomi +1Computer-Use Agent Benchmarks AI for Science
University of California, Riverside · University of California, Irvine
FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agents
Jun 30, 2026Muhammad Usman Safder, Ayesha Gull, Rania Elbadry +7Personality Modeling in Language Models Long-Horizon Agent Evaluation
Steve · MBZUAI · The University of Tokyo +2
Entity Binding Failures in Tool-Augmented Agents
Jun 29, 2026Rahul Suresh Babu, Shashank IndukuriTool-Using Agents LLM Agent Safety
Independent Researcher United States of America
A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents
Jun 29, 2026Liu ZewenLLM Evaluation LLM Agent Evaluation
Qilu Institute of Technology, School of Software Engineering · Tai’an, Shandong, China
SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution
Jun 29, 2026Aojie Yuan, Yi Nian, Haiyue Zhang +2AI Agent Reliability Process Reward Models
University of Southern California, Los Angeles, USA · University of Michigan, Ann Arbor, USA
Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds
Jun 28, 2026Mengdie Flora Wang, Haochen Xie, Guanghui Wang +2AI Agent Reliability Multi-Agent LLM Systems
1AWS Generative AI Innovation Center, USA · General Motors (GM), USA
Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts
Jun 28, 2026Alex KwonAI Agent Reliability LLM Agent Memory
Independent Researcher
Minority Sentinel: When to Overturn Majority Voting in Multi-Agent LLM Debates
Jun 28, 2026Chuan He, Zebin Chen, Zhengyi Yang +5Multi-Agent Debate Multi-Agent Consensus
University of New South Wales · Euler AI · Macquarie University
Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystem
Jun 27, 2026Yutian Tang, Yuming Zhou, Huaming ChenAgentic Workflows Human-in-the-Loop AI
School of Computing Science, University of Glasgow · School of Computer Science, Nanjing University · The University of Sydney
Memory as an Attack Surface in LLM Agents: A Study on Multiple-Choice Question Answering
Jun 27, 2026Shahnewaz Karim Sakib, Anindya Bijoy DasAgent Memory Poisoning LLM Agent Reliability
University of Tennessee at Chattanooga, TN 37403, USA · The University of Akron, OH 44325, USA
Preventing Error Propagation in Multi-Agent AI through Runtime Monitoring
Jun 27, 2026Shahnewaz Karim Sakib, Anindya Bijoy DasAI Agent Reliability Multi-Agent Systems
University of Tennessee at Chattanooga, TN 37403, USA · The University of Akron, OH 44325, USA
Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents
Jun 25, 2026Vassili Philippov, Pavel Katunin, Dmitry Andreev +2Multi-Agent Coding Autonomous Research Agents
1Glite · School of Biosciences, The University of Sheffield, Sheffield, UK
Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection
Jun 25, 2026Bo ChenConstrained Generative Modeling Web Agents
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
What the LLM Should Not Say: Boundary-Aware Context Grounding for A Seven-Channel EEG Agent
Jun 25, 2026Zhiyuan Xu, Yueqing Dai, Junling Li +1Electroencephalography LLM Grounding
Shanghai Pulse Element Intelligent Technology Co., Ltd. (NeuraDock)
Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge
Jun 25, 2026Neeraj YadavRetrieval-Augmented Generation Temporal IR
MemStrata.dev — Called It Inc. (Enterprise)
Instruction Bleed: Cross-Module Interference in Prompt-Composed Agentic Systems
Jun 24, 2026Ching-Yu Lin, Yifan LiuLLM Agent Evaluation LLM Agent Reliability
University of Illinois Urbana-Champaign, Urbana, IL, USA.
Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
Jun 24, 2026Yang Tian, Zhengpeng Shi, Yu Zhou +1AI Agent Reliability AI Agent Benchmarks
School of AI, Shanghai Jiao Tong University1 · School of Cyber Science and Engineering, Southeast University2
Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints
Jun 24, 2026Fangzheng Li, Aimin Zhang, Chen LvConstrained Decoding LLM Tool Use
1Focus AI Center, Focus Technology Co., Ltd. · 2Nanjing University of Science and Technology
TRUSTMEM: Learning Trustworthy Memory Consolidation for LLM Agents with Long-Term Memory
Jun 23, 2026Tianyu Yang, Sudipta Paul, Vijay Srinivasan +2LLM Agent Memory Agent Memory Management
AI Center-Mountain View, Samsung Electronics · University of Notre Dame
Agon: An Autonomous Large-Scale Omnidisciplinary Research System Built on Prompt Economy
Jun 23, 2026Youran Sun, Xingyu Ren, Chugang Yi +4AI Agents for Scientific Discovery Human-in-the-Loop AI
University of Maryland, College Park, College Park, MD, USA · The Chinese University of Hong Kong, Hong Kong, China · Stanford University, Stanford, CA, USA
When Robots Rate Their Own Interactions: Engagement Validity and the Strangeness Failure
Jun 22, 2026Victor Lockwood, Hasan Mahmud, Mohammad Javad Khojasteh +2LLM Agent Evaluation Human-Robot Interaction
MS in AI Program Rochester Institute of Technology Rochester, New York, USA · School of Communication Rochester Institute of Technology Rochester, New York, USA · Electrical and Microelectronic Engineering Rochester Institute of Technology Rochester, New York, USA
Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts
Jun 22, 2026Steven Young EuligAI Agent Reliability Reasoning Evaluation
Department of Physics and Laboratory for Particle Physics and Cosmology (LPPC), Harvard University, Cambridge, MA 02138, USA.
When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agents
Jun 22, 2026Aman MehtaAI Agent Reliability LLM Agent Evaluation
Snowflake AI Research
GRADE: Graph Representation of LLM Agent Dependency and Execution
Jun 22, 2026Yue ZhaoLLM Agent Evaluation LLM Agent Reliability
University of Southern California

Jul 2, 2026, cs.AIJ/K move · Enter open · S save

UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

Temitayo Olamilekan Ogunsusi, Lijun Qian, Xishuang Dong

Department of Electrical and Computer Engineering Prairie View A&M University, Prairie View, TX 77446, USA

Software Engineering Agents Multi-Agent Collaboration Multi-Agent Coding LLM Agent Reliability

Open paper AskPDF arXiv

Software development is a complex task that demands cooperation among agents with diverse roles. Large language models (LLMs) have enabled autonomous multi-agent software development frameworks that leverage role-based collaboration to automate requirements analysis, coding, testing, and refinement. However, existing approaches typically assume that intermediate agent outputs are equally reliable, leaving them vulnerable to hallucination propagation, where incorrect decisions generated in early development phases are transferred to downstream agents and negatively impact final software quality. To address this challenge, we propose UA-ChatDev, an uncertainty-aware multi-agent software development framework that integrates uncertainty quantification into agent interactions. It introduces a lightweight uncertainty estimation mechanism based on token-level log probabilities to assess the confidence of agent responses and employs phase-aware threshold calibration to selectively trigger retrieval-based verification when uncertainty exceeds acceptable levels. Extensive experiments on the SRDD benchmark demonstrate that UA-ChatDev consistently outperforms existing single-agent and multi-agent software development frameworks across completeness, executability, consistency, and overall quality metrics. Further ablation studies and communication analyses verify that uncertainty-aware interactions enhance code execution reliability.

Momentum

Related topics

Latest papers 558