Towards a Science of AI Agent Reliability

Stephan Rabanser; Sayash Kapoor; Peter Kirgis; Kangheng Liu; Saiteja Utpala; Arvind Narayanan

cs.AIFeb 18, 2026

Towards a Science of AI Agent Reliability

Authors: Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan

Organizations: Princeton University.

Abstract

AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safety-critical engineering, we provide a holistic performance profile by proposing twelve concrete metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 15 models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for reasoning about how agents perform, degrade, and fail.

Figures & tables

Figure 1: Reliability gains lag behind accuracy improvements. Overall reliability shows minimal improvement over time, despite 24 months of model releases. Moreover, improving accuracy alone does not guarantee reliability gains on complex real-world tasks. All frontier model providers cluster similarly, indicating reliability is an industry-wide plateau rather than a vendor-specific limitation.

Dimension	Cross-Domain Notion	Domain-Specific Exemplars
Consistency	Repeatable outcomes under nominal conditions; low variance across repeated trials	FAA requires deterministic execution of flight-critical software [ 51 ] ; NRC sets mandatory response times for digital computers in nuclear reactors [ 56 ] .
Robustness	Graceful degradation under input, environment, tool perturbations; stable performance across the full operational envelope	NASA investigation of software-related unintended acceleration in Toyota cars leads to recall [ 42 ] ; FAA mandates aviation sensor testing at extreme temperatures, turbulence, and vibration [ 12 ] .
Predictability	Prediction confidence aligned with accuracy; detect limits and defer/escalate under uncertainty	NRC models thousands of potential failure modes in nuclear reactors [ 55 ] ; Aviation uses tiered risk classification with explicit probabilities [ 13 ] .
Safety	Bounded harm even when failures occur; worst-case severity remains acceptable	SIL 4 standard requires dangerous failure probability less than $10^{-5}$ [ 21 ] ; FAA uses a one catastrophic error per billion flight hours target [ 13 ] .

Table 1: Reliability dimensions derived from cross-domain safety-critical engineering practices.

	Metric	Measurement Protocol
Consistency ( $\mathcal{R}_{\text{Con}}$ )	Outcome $C_{\text{out}}=\frac{1}{T}\sum_{t=1}^{T}(2\hat{p}_{t}-1)^{2}$	Run each task $t$ a total of $K$ times, yielding outcomes $y_{t,k}\in\{0,1\}$ . Compute per-task success rate $\hat{p}_{t}=\frac{1}{K}\sum_{k}y_{t,k}$ . Per-task consistency $(2\hat{p}_{t}-1)^{2}$ normalizes biased sample variance by maximum Bernoulli variance ( $0.25$ ). Average across $T$ tasks.
	Trajectory (Distributional) $C_{\text{traj}}^{d}=1-\frac{2\sum_{t}\sum_{i<j}\text{JSD}_{t}^{(i,j)}}{TK(K-1)}$	For task $t$ , collect action sequences from $K$ runs. Convert each sequence to distribution $\mathbf{p}_{t}^{(k)}$ over action types. Compute $\text{JSD}_{t}^{(i,j)}=\text{JSD}(\mathbf{p}_{t}^{(i)},\mathbf{p}_{t}^{(j)})$ as pairwise Jensen-Shannon divergence. The coefficient $\frac{2}{TK(K-1)}$ averages over $\binom{K}{2}$ pairs per task and $T$ tasks.
	Trajectory (Sequence) $C_{\text{traj}}^{s}=1-\frac{2\sum_{t}\sum_{i<j}\hat{d}_{t}^{(i,j)}}{TK(K-1)}$	For task $t$ , collect action sequences $\mathbf{a}^{(1)},\ldots,\mathbf{a}^{(K)}$ from all $K$ runs. Compute normalized pairwise Levenshtein distance $\hat{d}_{t}^{(i,j)}=d_{\text{lev}}(\mathbf{a}_{t}^{(i)},\mathbf{a}_{t}^{(j)})/\max(\|\mathbf{a}_{t}^{(i)}\|,\|\mathbf{a}_{t}^{(j)}\|)\in[0,1]$ . Average across all pairs and tasks as above.
	Resource $C_{\text{res}}=\exp\left(-\frac{1}{\|R\|}\sum_{r\in R}\text{CV}_{r}\right)$	For each task, record resource usage across $K$ runs. Let $R$ be the set of resource types (e.g., cost, time, API calls). For each $r\in R$ , compute coefficient of variation $\text{CV}_{r}=\sigma_{r}/\mu_{r}$ . Average across resource types and apply exponential transform.
Robustness ( $\mathcal{R}_{\text{Rob}}$ )	Fault $R_{\text{fault}}=\min\left(\frac{\text{Acc}_{\text{fault}}}{\text{Acc}_{0}},1\right)$	Run all tasks under baseline conditions to get $\text{Acc}_{0}=\frac{1}{N}\sum_{i}y_{i}^{(0)}$ . Re-run under fault injection (e.g., tool timeouts, error responses) to get $\text{Acc}_{\text{fault}}$ . Compute clamped ratio.
	Environment $R_{\text{env}}=\min\left(\frac{\text{Acc}_{\text{pert}}}{\text{Acc}_{0}},1\right)$	Run all tasks under baseline conditions to obtain $\text{Acc}_{0}$ . Re-run with environment perturbations (e.g., reordered fields, altered tool interfaces) to obtain $\text{Acc}_{\text{pert}}$ .

Table 2: Reliability metrics overview. All scores

\in[0,1]

with higher values indicating better reliability. Notation:

T

= number of tasks;

K

= runs per task;

\epsilon

= small constant (

10^{-8}

);

\mathds{1}[\cdot]

= indicator function.

Figure 2: Outcome consistency across models. Results show only modest consistency across the board; even current frontier models do not reliably improve across both benchmarks.

Figure 3: Prompt robustness across models. Many models remain susceptible to surface-level prompt reformulations. Latest frontier models generally show modest improvements.

Figure 4: Calibration and discrimination across models. Calibration, the alignment between predicted confidence and accuracy, generally improves in latest frontier models. Discrimination performance, the ability to distinguish correct and incorrect predictions, is inconsistent across benchmarks (even worsens on GAIA).

Figure 5: Safety analysis on

\tau

-bench. Top : Average violations per evaluation run stratified by severity level. Bottom : Breakdown of violations by constraint category. The most recent frontier models exhibit significantly lower overall violation rates. Financial accuracy (i.e., incorrect charges/refunds) remains the most common failure mode across all models.

Figure 6: Comparison of

\tau

-bench vs.

\tau

-bench (clean) . Accuracy improves significantly across the board. Many agents also show improved reliability across dimensions on the verified subset of

\tau

-bench. Predictability sees the most noticeable improvement.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Provider	Model	Release Date	Category
OpenAI	GPT-4 Turbo	2024-04-09	Frontier
	GPT-4o mini	2024-07-18	Efficient
	o1	2024-12-05	Reasoning
	GPT-5.2	2025-12-11	Frontier
	GPT-5.2 (medium)	2025-12-11	Reasoning
	GPT-5.5	2026-04-23	Frontier

Appendix

Table 3: Models evaluated in our reliability study, organized by provider and release date.

Level	Temp.	Transformation Types
mild	0.7	Synonym substitution, formality changes, voice changes (active/passive), minor restructuring
medium	0.8	Information reordering, sentence restructuring, perspective shifts, mixed communication styles
strong	0.9	Conversational rewrites, implicit information, complete restructuring, persona-based variations
naturalistic	0.9	Realistic user behavior: typos, abbreviations, inconsistent capitalization, fragments, casual punctuation

Appendix

Table 4: Prompt perturbation strength levels. Higher temperatures encourage greater diversity in variations.

Fault Type	Prob.	Description
timeout	30%	Simulated request timeout ( TimeoutError )
error_response	25%	HTTP 500 Internal Server Error
rate_limit	20%	HTTP 429 Too Many Requests
network_error	15%	Connection refused / network failure
partial_failure	5%	Incomplete or truncated response data
invalid_response	3%	Malformed or unparseable response

Appendix

Table 5: Fault type distribution. Probabilities reflect relative frequency when a fault is triggered.

Category	GAIA	$\tau$ -bench
Text formatting	Case changes (lowercase, mixed), whitespace normalization, noise word injection	—
Data formats	Number formats (commas, words), date formats (ISO $\to$ verbose)	Time (24h $\to$ 12h/compact), dates (ISO $\to$ US/compact), status values (lowercase $\to$ abbreviated)
Structural	Instruction style changes (formal, terse), bullet reordering, irrelevant context injection	Key naming (snake_case $\to$ camelCase), response wrapping, parameter renaming, nesting/flattening, cabin class codes
Tool outputs	Search result reformatting, webpage noise (headers, footers), metadata wrapping	Tool response key transformation, tool definition parameter renaming

Appendix

Table 6: Environment perturbation categories by benchmark.

Replit agent	Safety ( $S_{\text{harm}}$ ): Error severity analysis reveals high-harm failures (e.g., irreversible actions like database deletion).
Replit agent	Robustness ( $R_{\text{prompt}}$ ): Prompt robustness testing would reveal whether the “do not delete the database” constraint holds under rephrased instructions or varied task contexts.
OpenAI Operator	Safety ( $S_{\text{comp}}$ ): Compliance testing detects actions without required user confirmation for financial transactions.
OpenAI Operator	Consistency ( $C_{\text{traj}}$ ): Trajectory divergence analysis would flag unexpected behavioral patterns, such as completing a purchase without pausing for user confirmation.
NYC chatbot	Predictability ( $P_{\text{cal}}$ ): Calibration testing exposes chatbot overconfidence when returning incorrect legal guidance.
NYC chatbot	Consistency ( $C_{\text{out}}$ ): Low outcome consistency would reveal that the chatbot gives different answers to the same question across users asking the same question.

Appendix

Table 7: Mapping real-world agent failures to reliability dimensions and metrics that could have provided early warning signals. Systematic evaluation on these dimensions could have identified each vulnerability prior to release.

Figure 7: Reliability gains lag behind accuracy gains. Overall reliability shows slow improvement over time. While accuracy rises steadily across both benchmarks (left), reliability trails behind (center), and the relationship between the two varies across benchmarks (right), indicating that accuracy gains do not automatically yield reliability.

Panel of Figure 8 — Figure 8: Trends across agents and benchmarks. Many of our reliability metrics show only marginal improvements over time and with increased model utility. Predictability and safety on $\tau$ -bench is a notable exception.

Figure 9: Reliability by model type (top: GAIA, bottom:

\tau

-bench). Larger and reasoning models improve reliability on average over smaller models. Notably though, reasoning models do not significantly improve in predictability on GAIA or in consistency on

\tau

-bench.

Figure 10: Reliability by provider (top: GAIA, bottom:

\tau

-bench).

Panel of Figure 11 — Figure 11: Consistency results across agents and benchmarks. Consistency metrics reveal agents reliably choose similar actions but vary in execution order: a ”what but not when” gap highlighting limitations in current planning capabilities. Outcome consistency remains the most challenging metric across both benchmarks. Resource consistency shows noticeable variance in compute usage (mostly GAIA).

Panel of Figure 12 — Figure 12: Predictability results across agents and benchmarks. Claude models generally show the strongest calibration performance (confidence matching actual success). Discrimination (ranking task difficulty) shows mixed results with recent models sometimes improving and sometimes regressing.

Figure 13: Reliability plots of different agent models on GAIA. Calibration plots show confidence (x-axis) versus actual accuracy (y-axis), where the diagonal represents perfect calibration. Agents are noticeably better calibrated compared to

\tau

-bench. Anthropic models (Claude Opus 4.5 in particular) stand out as well calibrated. Notably, Opus 4.7 is the only model that is consistently under-confident.

Figure 14: Calibration plots of different agent models on

\tau

-bench. All agents suffer from severe overconfidence. Only newer Claude models show modest calibration improvements.

Figure 15: Selective prediction curves of different agent models on GAIA. Accuracy-coverage curves show whether models can improve accuracy by abstaining on low-confidence predictions; the ideal curve rises steeply while the random baseline indicates confidence provides no signal. Most models demonstrate modest selective prediction ability, with Gemini 2.5 Flash, GPT-5.2, and Claude Opus 4.5 showing particularly strong performance.

Figure 16: Selective prediction curves of different agent models on

\tau

-bench. Selective prediction largely fails in this constrained benchmarking setup: most models produce curves indistinguishable from the random baseline, indicating confidence scores carry no information about correctness. Only Claude Opus 4.5 and 4.7 attain modest selective prediction ability.

Panel of Figure 17 — Figure 17: Robustness results across agents and benchmarks. Prompt robustness shows the largest amount of variation with environment and fault robustness plateauing for our chosen perturbations.

Figure 18: Safety results across agents on

\tau

-bench. Extension of Figure 5 .

Figure 19: Comparison of reasoning vs non-reasoning models. We observe that reasoning models are generally more reliable than non-reasoning models, albeit reliability improves slower than accuracy.

Figure 20: Reliability metrics stratified by task difficulty on GAIA. Accuracy degrades as expected on harder tasks, while Claude and Gemini models invest significantly more actions on difficult problems. Outcome consistency shows mixed patterns driven by accuracy-dependent failure modes. Robustness metrics (aside from prompt robustness) remain relatively stable across difficulty levels, suggesting robustness is largely orthogonal to task complexity.

Explore similar work

May 11, 2026

Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability

This paper establishes a rigorous measurement science for AI agent reliability, providing a foundational framework for quantifying consistency under semantically preserving perturbations. By leveraging

U

-statistics for output-level reliability and kernel-based metrics for trajectory-level stability, we offer a principled approach to evaluating agents across diverse operating conditions. Our proposal highlights the important distinction between the core capability and execution robustness of an agent, showing that minor task-level variations can induce complete strategy breakdowns despite the agent possessing the requisite knowledge for the task. We validate our framework through extensive experiments on three agentic benchmarks, demonstrating that trajectory-level consistency metrics provide far greater diagnostic sensitivity than traditional pass@1 rates. By providing the mathematical tools to isolate where and why agents deviate, we enable the identification and rectification of architectural concerns that hinder the deployment of agents in high-stakes, real-world environments.

Harsh Raj, Niranjan Orkat, Suvrorup Mukherjee +3

Northeastern University · University of P´ecs · University of Michigan +2

AI Agent Reliability Agent Reliability AI Agent Evaluation

Ask PDF

Jul 15, 2026

AI Agents Do Not Fail Alone:The Context Fails First

Context engineering has become central to building reliable AI agents, yet it remains largely unmeasured. Agents do not fail in isolation: their behavior is shaped by the instructions, tools, memory, retrieved knowledge, guardrails, and untrusted inputs accumulated in their context. When this context is weak, agents drift, hallucinate, misuse tools, ignore constraints, become vulnerable to injection, and waste tokens. This paper validates context-engineering quality as an independent leading indicator of agent reliability. We implement the measurement in ProofAgent-Harness, an open-source infrastructure for AI agent evaluation that uses multi-juror, consensus-based scoring. The harness assesses context across seven criteria: role clarity, guardrail coverage, instruction consistency, tool schema quality, grounding sufficiency, injection hardening, and token efficiency. Crucially, the context score is isolated from behavioral metrics and release decisions, enabling a non-circular validation. Through a controlled context-quality study across regulated agent domains, holding frontier LLM agents fixed and varying only their operating context, we show that context-quality criteria consistently predict their corresponding behavioral outcomes. Grounding sufficiency predicts hallucination resistance, guardrail coverage predicts manipulation resistance, instruction consistency predicts instruction following, and tool-schema quality predicts tool use. These findings establish context measurement as a validated preflight signal for agent reliability and position context engineering as an auditable layer of agent evaluation and governance.

Fouad Bousetouane

The University of Chicago, USA

AI Agent Reliability Agent Reliability AI Agent Evaluation LLM Context Management

Ask PDF

Sep 30, 2026

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

Michael Hardy, Ruhana Azam, Anka Reuel +2

Stanford University · University of Illinois Urbana-Champaign · Senior Author

AI Agent Evaluation Agent Evaluation AI Agent Benchmarks

Ask PDF