LLM Agent Reliability

LLM: Large Language Model

LLM agent reliability studies whether agents can complete tasks correctly, follow instructions and safety constraints, use evidence faithfully, and behave consistently across repeated or extended interactions. Common approaches combine structured harnesses and deterministic tool interfaces with state tracking, evidence checks, runtime monitoring, calibrated abstention, and recovery from failures; training methods also teach agents to verify actions or learn from execution feedback. Evaluations increasingly use interactive benchmarks, noisy or changing tools, repeated trials, and trajectory-level measures alongside final task success. A central tension is that added reasoning, delegation, memory, or orchestration can improve some tasks but also introduce new failure paths, so reliability depends on the model, harness, tools, and evaluation setting together.

Latest papers

843 papers181 papers in the last four weeks, up 93% on the four weeks before. 1.0% of all new papers.

TimeSeek: Temporal Reliability of Agentic Forecasters
Apr 5, 2026Hamza Mostafa, Om Shastri, Dennis LeeLLM Agent Reliability Forecasting Benchmarks
The Wharton School, University of Pennsylvania
Your Agent Has a Genome: Sequence-Level Behavioral Analysis and Runtime Governance of LLM-Powered Autonomous Agents
Apr 5, 2026Sidi DengAI Agent Governance Runtime Enforcement for AI Agents
Independent Researcher
Profile-Then-Reason: Bounded Semantic Complexity for Tool-Augmented Language Agents
Apr 5, 2026Paulo Akira F. EnabeTool-Using Language Agents LLM Agents
Escola Politénica, University of São Paulo
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
Apr 3, 2026Delip Rao, Eric Wong, Chris Callison-BurchCitation Verification LLM Hallucination Detection
University of Pennsylvania
EMS: Multi-Agent Voting via Efficient Majority-then-Stopping
Apr 3, 2026Yiqing Liu, Hantao Yao, Wu Liu +1Multi-Agent Reasoning Early Stopping
University of Science and Technology of China
Improving Role Consistency in Multi-Agent Collaboration via Quantitative Role Clarity
Apr 3, 2026Guoling Zhou, Wenpei Han, Fengqin Yang +3Behavioral Consistency of LLM Agents LLM Agents
School of Information Science and Technology, Northeast Normal University, Changchun, Jilin · School of Computer Science and Engineering, Guangxi Normal University, Guilin, Guangxi
Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents
Apr 2, 2026Xuan QiEfficient Language Model Reasoning LLM Agent Reliability
IIIS, Tsinghua University
Exploring Robust Multi-Agent Workflows for Environmental Data Management
Apr 2, 2026Boyuan Guan, Jason Liu, Yanzhao Wu +1Multi-Agent LLM Systems LLM Agent Reliability
Florida International University, Miami, Florida, United States
No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agents
Apr 1, 2026Tiankai Yang, Jiate Li, Yi Nian +5LLM Agent Security LLM Agent Safety
University of Southern California · Michigan State University · Northwestern University +1
From Data to Theory: Autonomous Large Language Model Agents for Materials Science
Apr 1, 2026Samuel Onimpa Alfred, Veera SundararaghavanLarge Language Model-Guided Scientific Discovery Scientific Discovery
Department of Aerospace Engineering, University of Michigan, Ann Arbor, MI 48109
$\texttt{YC-Bench}$ : Benchmarking AI Agents for Long-Term Planning and Consistent Execution
Apr 1, 2026Muyu He, Adit Jain, Anand Kumar +4Long-Horizon Agent Evaluation AI Agent Benchmarks
HabitatAgent: An End-to-End Multi-Agent System for Housing Consultation
Apr 1, 2026Hongyang Yang, Yanxin Zhang, Yang She +5AI-Assisted Decision Making Multi-Agent LLM Systems
Fangdongdong · Columbia University · Tsinghua University
Ontology-Constrained Neural Reasoning in Enterprise Agentic Systems: A Neurosymbolic Architecture for Domain-Grounded AI Agents
Apr 1, 2026Thanh Luong Tuan, Abhijit SanyalOntology-Based Reasoning Enterprise AI Agents
Golden Gate University, San Francisco · Foundation AgenticOS (FAOS) · Associate Director, Data, Digital & IT, Novartis Healthcare Pvt. Ltd., Hyderabad, India
Asymmetric Actor-Critic for Multi-turn LLM Agents
Mar 31, 2026Shuli Jiang, Zhaoyang Zhang, Yi Zhang +3Asynchronous Advantage Actor-Critic LLM Agent Reliability
AWS Agentic AI
Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents
Mar 31, 2026Hy Dang, Quang Dao, Meng JiangTool-Using Agents LLM Agent Reliability
University of Notre Dame · Rose-Hulman Institute of Technology
Near-Miss: Latent Policy Failure Detection in Agentic Workflows
Mar 31, 2026Ella Rabinovich, David Boaz, Naama Zwerdling +1LLM Agent Evaluation AI Agent Auditing
IBM Research
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Mar 31, 2026Aaditya Khanal, Yangyang Tao, Junxiu ZhouLLM Reliability LLM Agent Reliability
School of Computing and Analytics, Northern Kentucky University
Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs
Mar 29, 2026K M Ferdous, Dipayan Banik, Kowshik Chowdhury +1AI Coding Agents LLM Agent Reliability
Kennesaw State University Marietta, Georgia, USA · Quanta Technology Raleigh, North Carolina, USA
On the Reliability Limits of LLM-Based Multi-Agent Planning
Mar 27, 2026Ruicheng Ao, Siyang Gao, David Simchi-LeviLLM Agent Reliability Multi-Agent LLM Systems
Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA 02139. · Department of Systems Engineering, City University of Hong Kong, Kowloon, Hong Kong. · Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA 02139. +1
WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing
Mar 26, 2026Fanheng Kong, Jingyuan Zhang, Yang Yue +10Automated Software Testing Computer-Use Agent Benchmarks
Northeastern University · Kuaishou Technology
ElephantBroker: A Knowledge-Grounded Cognitive Runtime for Trustworthy AI Agents
Mar 26, 2026Cristian Lupascu, Alexandru LupascuLLM Agent Memory LLM Agent Reliability
Elephant Broker, 050141 Bucharest, Romania
Confident and Wrong: Silent Semantic Failures in Coding Agents
Mar 26, 2026Aman MehtaAI Agent Reliability AI Coding Agents
Snowflake AI Research
TrustTrade: Human-Inspired Selective Consensus Reduces Decision Uncertainty in LLM Trading Agents
Mar 23, 2026Minghan Li, Rachel Gonsalves, Weiyue Li +2Algorithmic Trading LLM Agent Reliability
Harvard AI and Robotics Lab, Harvard University . · Harvard Business School, Harvard University . · Daegu Gyeongbuk Institute of Science and Technology . +1
AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning
Mar 22, 2026Liang DingLLM Agent Evaluation Rubric-Based Evaluation
The University of Sydney
Solver-Aided Verification of Policy Compliance in Tool-Augmented LLM Agents
Mar 20, 2026Cailin Winston, Claris Winston, René JustTool Access Control for LLM Agents LLM Agent Verification
University of Washington Seattle, USA
Why Cognitive Robotics Matters: Lessons from OntoAgent and LLM Deployment in HARMONIC for Safety-Critical Robot Teaming
Mar 20, 2026Sanjay Oruganti, Sergei Nirenburg, Marjorie McShane +5Cognitive Architectures for AI Agents Large Language Model-Based Robot Planning
Cognitive Science Department, Rensselaer Polytechnic Institute, Troy, NY 12180, USA. · University of Georgia, Athens, GA, 30605, USA. · Aerospace Engineering and Engineering Mechanics, University of Texas at Austin, TX, US.
Can Blindfolded LLMs Still Trade? An Anonymization-First Framework for Portfolio Optimization
Mar 18, 2026Joohyoung Jeon, Hongchul LeeAlgorithmic Trading LLM Agent Reliability
Korea University · Mirae Asset Securities
Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)
Mar 18, 2026Zichen Tang, Zirui Zhang, Qian Wang +3Large Language Model-Based Recommendation LLM Agent Reliability
The Hong Kong University of Science and Technology · National University of Singapore · The Hong Kong University of Science and Technology (Guangzhou)
RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments
Mar 17, 2026Linghua Zhang, Jun Wang, Jingtong Wu +1Long-Horizon Agent Evaluation AI Agent Benchmarks
Ant Group · City University of Hong Kong
MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verification
Mar 16, 2026MiroMind Team, S. Bai, L. Bing +41Autonomous Research Agents Long-Horizon LLM Agents

Apr 5, 2026J/K move · Enter open · S save

TimeSeek: Temporal Reliability of Agentic Forecasters

Hamza Mostafa, Om Shastri, Dennis Lee

The Wharton School, University of Pennsylvania

LLM Agent Reliability Forecasting Benchmarks Probabilistic Forecasting Prediction Markets

Open paper AskPDF arXiv

We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.