LLM Agent Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

90 papers in the last four weeks, up 109% on the four weeks before. 0.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 558

All topics

RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents
Jun 21, 2026Meher Bhaskar Madiraju, Meher Sai Preetam MadirajuAI Agent Reliability Coding Agents
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
Jun 21, 2026Jiayu Liu, Qihan Lin, Cheng Qian +8Long-Horizon Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign
Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identity
Jun 20, 2026Andrew TannerAI Agent Monitoring LLM Agent Reliability
Anisotrope AI
CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks
Jun 19, 2026Ashwin Vinod, Ying Ding, Elias Stengel-EskinLLM Answer Verification LLM Agent Verification
The University of Texas at Austin
Don't Blindly Trust It: How Unreliable Feedback Breaks Tool-Using LLM Agents
Jun 19, 2026Chubin Zhang, Zhenglin Wan, Xingrui Yu +5AI Agent Reliability Tool-Use Evaluation
Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore +3
LedgerAgent: Structured State for Policy-Adherent Tool-Calling Agents
Jun 18, 2026Md Nayem Uddin, Amir Saeidi, Eduardo Blanco +1Tool-Augmented Language Model Agents Customer Support Automation
♠Arizona State University · ♢University of Arizona
Dual-Agent Framework for Cross-Model Verified Translation of Natural-Language Protocols into Robotic Laboratory Platform
Jun 18, 2026Hyeonna Choi, Jung Yup Kim, Hyuneui Lim +1LLM Agent Orchestration Laboratory Automation
Department of Bionic Machinery, Research Institute of AI Robot, Korea Institute of Machinery & Materials, 156, Gajeongbuk-ro, Yuseong-gu, Daejeon 34103, South Korea · 156, Gajeongbuk-ro, Yuseong-gu, Daejeon 34103, South Korea · Department of Nano-devices & displays, Nano-convergence Manufacturing Research Division, Korea Institute of Machinery & Materials, 156, Gajeongbuk-ro, Yuseong-gu, Daejeon 34103, South Korea
Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery
Jun 18, 2026Anushree Sinha, Srivaths Ranganathan, Abhishek Dharmaratnakar +1Legal IR LLM Agent Orchestration
Google LLC Mountain View, CA, USA
Uncertainty Decomposition for Clarification Seeking in LLM Agents
Jun 17, 2026Gregory MatsnevLLM Agents LLM Uncertainty Estimation
AI Talent Hub, ITMO University Saint Petersburg 197101, Russia
Written by AI, Managed by AI: Semantic Space Control and Index Sickness Elimination Across 391 Consecutive Sessions
Jun 17, 2026Hui Zhang, Shuren SongLong-Horizon LLM Agents LLM Context Management
Shenzhen Yunxi Technology Co., Ltd. · Information Technology Center, Tsinghua University
EARS: Explanatory Abstention for Reliable Sub-Agent Modeling in Large-scale Multi-Agent Systems
Jun 17, 2026Shuang Xie, Yunan Lu, Han Li +1AI Agent Reliability Multi-Agent LLM Systems
Shopify · Columbia University
ToolChain-CRC: Conformal Risk Control for Agentic AI Under Retrieval and Tool-Use Drift
Jun 16, 2026Jeffery Opoku, David BanaheneAI Agent Reliability Distribution Shift
The University of Texas Rio Grande Valley, Edinburg, TX, USA · Florida International University, Miami, FL, USA
Intelligence Entropy Principle and the ADE Stability Engineering Framework
Jun 16, 2026Dexing LiuAgent Failure Analysis Multi-Agent LLM Systems
Shanghai Qijing Digital Technology · Shanghai Qijing Digital Technology Co., Ltd.
ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents
Jun 16, 2026Ander Alvarez, Santhiya Rajan, Samuel Mugel +1LLM Answer Verification Data Provenance
1Multiverse Computing, Parque Cientifico y Tecnol´ogico de Gipuzkoa, Paseo de Miram´on, 170, 20014 Donostia / San Sebasti´an, Spain · 2Multiverse Computing, Centre for Social Innovation, 192 Spadina Avenue Suite 509, Toronto, ON M5T 2C2, Canada · 3Donostia International Physics Center, Paseo Manuel de Lardizabal 4, E-20018 San Sebasti´an, Spain +1
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
Jun 16, 2026Congjie Zheng, Chuanyi Xue, Bin Liang +2LLM Agent Evaluation AI Agent Benchmarks
Department of Automation, Tsinghua University, Beijing, 100084, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Jun 15, 2026Xianxuan Long, Zhikai Chen, Shenglai Zeng +3Temporal Reasoning in Language Models LLM Agent Memory
Michigan State University · Case Western Reserve University
How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation
Jun 15, 2026Yimeng Chen, Zhe Ren, Firas Laakom +3LLM Agent Evaluation AI Agent Security
Center of Excellence for Generative AI, KAUST · 2Jilin University · 3Zhejiang University +2
GIST-CMTF: Goal-State Inference for Causal Minimal Tool Filtering in LLM Agents
Jun 15, 2026Rahul Suresh Babu, Rohit ShuklaClarification Question Generation Tool-Using Agents
Independent Researcher United States of America
Misinformation Propagation in Benign Multi-Agent Systems
Jun 15, 2026Jonas Becker, Jan Philip Wahle, Terry Ruas +1Multi-Agent Debate Multi-Agent Communication
University of Göttingen, Germany · LKA NRW, Germany
VeriGraph: Towards Verifiable Data-Analytic Agents
Jun 15, 2026Jiajie Jin, Zhao Yang, Wenle Liao +5Data Provenance Neuro-Symbolic Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China
Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents
Jun 15, 2026Shiyang ChenAI Agent Reliability Tool-Augmented Language Model Agents
Beijing Institute of Technology
DeepRoot: A KG-Coordinated Multi-Agent System for Therapeutic Reasoning over Historical Medical Texts
Jun 14, 2026Zijian Carl Ma, Sean J. Wang, Sijbren Kramer +1Historical Document Analysis Multi-Agent LLM Systems
Department of Bioengineering · Stanford University · Amazon AWS AI
LLM-as-Code: Agentic Programming for Agent Harness
Jun 14, 2026Junjia Qi, Zichuan Fu, Jingtong Gao +4AI Agent Reliability LLM Agent Orchestration
City University of Hong Kong · Tencent Jarvis Lab
Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?
Jun 14, 2026Liran Tal, Johannes Kloos, Arsenii Rudich +2Software Vulnerability Detection Static Code Analysis
Snyk
From Question Answering to Task Completion: A Survey on Agent System and Harness Design
Jun 14, 2026Jianyuan Guo, Zhiwei Hao, Chengcheng Wang +14LLM Agent Evaluation LLM Agents
Department of Computer Science, City University of Hong Kong, HKSAR, China
Resilient Consensus in Agentic AI
Jun 12, 2026Sribalaji C. Anand, George J. PappasMulti-Agent LLM Systems Multi-Agent Consensus
1KTH, Sweden. · University of Pennsylvania, United States.
When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
Jun 12, 2026Wei WuAgent Failure Analysis AI Agent Monitoring
Independent researcher
PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions
Jun 12, 2026Chenxin Li, Zhengyao Fang, Zhengyang Tang +18Mobile GUI Automation Tool-Using Agents
Tencent Hunyuan · The Chinese University of Hong Kong · The Chinese University of Hong Kong, Shenzhen +1
When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More
Jun 12, 2026Zhongyuan Wang, Pratyusha VemuriGraph Neural Networks LLM Reasoning with Graphs
Faithful, Not Corrective: Model Capability Governs Message-Format Effects in Multi-Hop Agent Relays
Jun 12, 2026Sicheng ZengLLM Agents Multi-Agent Communication
Independent Researcher

Jun 21, 2026, cs.SEJ/K move · Enter open · S save

RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents

Meher Bhaskar Madiraju, Meher Sai Preetam Madiraju

AI Agent Reliability Coding Agents AI Agent Evaluation AI Agent Benchmarks Code Generation Evaluation LLM Agent Reliability

Open paper AskPDF arXiv

Agentic coding harnesses - such as Agent-Skills, Superpowers, and Agent-Rigor - are increasingly deployed to augment underlying LLMs for real-world software engineering tasks. Existing benchmarks evaluate these agents almost exclusively on outcome correctness: whether generated code passes tests or resolves issues. We argue that this outcome-only lens is insufficient: an agent that arrives at a correct solution through reckless trial-and-error, without planning, verification, or graceful recovery, is fundamentally less reliable than one that follows sound engineering discipline. We introduce RigorBench, the first benchmark designed to measure process discipline in AI coding agents. RigorBench evaluates these harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. A composite RigorScore aggregates these dimensions into a single metric via a weighted sum. We curate a suite of 30 tasks spanning five categories - Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, and Don't Break the Build-and evaluate leading harnesses in a controlled with/without experimental design against baseline coding assistants. Our results show that structured process discipline not only improves process quality scores by an average of 41% but also raises downstream outcome correctness by 17%, providing the first quantitative evidence that how agents code matters as much as what they produce. We release the full benchmark, scoring rubrics, and trajectory analysis tools as open-source artifacts.

Momentum

Related topics

Latest papers 558