cs.AIFeb 18, 2026
SaveTowards a Science of AI Agent Reliability
Organizations: Princeton University.
Abstract
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safety-critical engineering, we provide a holistic performance profile by proposing twelve concrete metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 15 models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for reasoning about how agents perform, degrade, and fail.
Figures & tables
Figure 1: Reliability gains lag behind accuracy improvements. Overall reliability shows minimal improvement over time, despite 24 months of model releases. Moreover, improving accuracy alone does not guarantee reliability gains on complex real-world tasks. All frontier model providers cluster similarly, indicating reliability is an industry-wide plateau rather than a vendor-specific limitation.
| Dimension | Cross-Domain Notion | Domain-Specific Exemplars |
|---|---|---|
| Consistency | Repeatable outcomes under nominal conditions; low variance across repeated trials | FAA requires deterministic execution of flight-critical software [ 51 ] ; NRC sets mandatory response times for digital computers in nuclear reactors [ 56 ] . |
| Robustness | Graceful degradation under input, environment, tool perturbations; stable performance across the full operational envelope | NASA investigation of software-related unintended acceleration in Toyota cars leads to recall [ 42 ] ; FAA mandates aviation sensor testing at extreme temperatures, turbulence, and vibration [ 12 ] . |
| Predictability | Prediction confidence aligned with accuracy; detect limits and defer/escalate under uncertainty | NRC models thousands of potential failure modes in nuclear reactors [ 55 ] ; Aviation uses tiered risk classification with explicit probabilities [ 13 ] . |
| Safety | Bounded harm even when failures occur; worst-case severity remains acceptable | SIL 4 standard requires dangerous failure probability less than [ 21 ] ; FAA uses a one catastrophic error per billion flight hours target [ 13 ] . |
Table 1: Reliability dimensions derived from cross-domain safety-critical engineering practices.
| Metric | Measurement Protocol | |
| Consistency ( ) | Outcome | Run each task a total of times, yielding outcomes . Compute per-task success rate . Per-task consistency normalizes biased sample variance by maximum Bernoulli variance ( ). Average across tasks. |
| Trajectory (Distributional) | For task , collect action sequences from runs. Convert each sequence to distribution over action types. Compute as pairwise Jensen-Shannon divergence. The coefficient averages over pairs per task and tasks. | |
| Trajectory (Sequence) | For task , collect action sequences from all runs. Compute normalized pairwise Levenshtein distance . Average across all pairs and tasks as above. | |
| Resource | For each task, record resource usage across runs. Let be the set of resource types (e.g., cost, time, API calls). For each , compute coefficient of variation . Average across resource types and apply exponential transform. | |
| Robustness ( ) | Fault | Run all tasks under baseline conditions to get . Re-run under fault injection (e.g., tool timeouts, error responses) to get . Compute clamped ratio. |
| Environment | Run all tasks under baseline conditions to obtain . Re-run with environment perturbations (e.g., reordered fields, altered tool interfaces) to obtain . |
Table 2: Reliability metrics overview. All scores with higher values indicating better reliability. Notation: = number of tasks; = runs per task; = small constant ( ); = indicator function.
Figure 2: Outcome consistency across models. Results show only modest consistency across the board; even current frontier models do not reliably improve across both benchmarks.
Figure 3: Prompt robustness across models. Many models remain susceptible to surface-level prompt reformulations. Latest frontier models generally show modest improvements.
Figure 4: Calibration and discrimination across models. Calibration, the alignment between predicted confidence and accuracy, generally improves in latest frontier models. Discrimination performance, the ability to distinguish correct and incorrect predictions, is inconsistent across benchmarks (even worsens on GAIA).
Figure 5: Safety analysis on -bench. Top : Average violations per evaluation run stratified by severity level. Bottom : Breakdown of violations by constraint category. The most recent frontier models exhibit significantly lower overall violation rates. Financial accuracy (i.e., incorrect charges/refunds) remains the most common failure mode across all models.
Figure 6: Comparison of -bench vs. -bench (clean) . Accuracy improves significantly across the board. Many agents also show improved reliability across dimensions on the verified subset of -bench. Predictability sees the most noticeable improvement.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Provider | Model | Release Date | Category |
| OpenAI | GPT-4 Turbo | 2024-04-09 | Frontier |
| GPT-4o mini | 2024-07-18 | Efficient | |
| o1 | 2024-12-05 | Reasoning | |
| GPT-5.2 | 2025-12-11 | Frontier | |
| GPT-5.2 (medium) | 2025-12-11 | Reasoning | |
| GPT-5.5 | 2026-04-23 | Frontier |
Table 3: Models evaluated in our reliability study, organized by provider and release date.
| Level | Temp. | Transformation Types |
|---|---|---|
| mild | 0.7 | Synonym substitution, formality changes, voice changes (active/passive), minor restructuring |
| medium | 0.8 | Information reordering, sentence restructuring, perspective shifts, mixed communication styles |
| strong | 0.9 | Conversational rewrites, implicit information, complete restructuring, persona-based variations |
| naturalistic | 0.9 | Realistic user behavior: typos, abbreviations, inconsistent capitalization, fragments, casual punctuation |
Table 4: Prompt perturbation strength levels. Higher temperatures encourage greater diversity in variations.
| Fault Type | Prob. | Description |
|---|---|---|
| timeout | 30% | Simulated request timeout ( TimeoutError ) |
| error_response | 25% | HTTP 500 Internal Server Error |
| rate_limit | 20% | HTTP 429 Too Many Requests |
| network_error | 15% | Connection refused / network failure |
| partial_failure | 5% | Incomplete or truncated response data |
| invalid_response | 3% | Malformed or unparseable response |
Table 5: Fault type distribution. Probabilities reflect relative frequency when a fault is triggered.
| Category | GAIA | -bench |
|---|---|---|
| Text formatting | Case changes (lowercase, mixed), whitespace normalization, noise word injection | — |
| Data formats | Number formats (commas, words), date formats (ISO verbose) | Time (24h 12h/compact), dates (ISO US/compact), status values (lowercase abbreviated) |
| Structural | Instruction style changes (formal, terse), bullet reordering, irrelevant context injection | Key naming (snake_case camelCase), response wrapping, parameter renaming, nesting/flattening, cabin class codes |
| Tool outputs | Search result reformatting, webpage noise (headers, footers), metadata wrapping | Tool response key transformation, tool definition parameter renaming |
Table 6: Environment perturbation categories by benchmark.
| Replit agent | Safety ( ): Error severity analysis reveals high-harm failures (e.g., irreversible actions like database deletion). |
|---|---|
| Robustness ( ): Prompt robustness testing would reveal whether the “do not delete the database” constraint holds under rephrased instructions or varied task contexts. | |
| OpenAI Operator | Safety ( ): Compliance testing detects actions without required user confirmation for financial transactions. |
| Consistency ( ): Trajectory divergence analysis would flag unexpected behavioral patterns, such as completing a purchase without pausing for user confirmation. | |
| NYC chatbot | Predictability ( ): Calibration testing exposes chatbot overconfidence when returning incorrect legal guidance. |
| Consistency ( ): Low outcome consistency would reveal that the chatbot gives different answers to the same question across users asking the same question. |
Table 7: Mapping real-world agent failures to reliability dimensions and metrics that could have provided early warning signals. Systematic evaluation on these dimensions could have identified each vulnerability prior to release.
Figure 7: Reliability gains lag behind accuracy gains. Overall reliability shows slow improvement over time. While accuracy rises steadily across both benchmarks (left), reliability trails behind (center), and the relationship between the two varies across benchmarks (right), indicating that accuracy gains do not automatically yield reliability.
Figure 8: Trends across agents and benchmarks. Many of our reliability metrics show only marginal improvements over time and with increased model utility. Predictability and safety on -bench is a notable exception.
Figure 9: Reliability by model type (top: GAIA, bottom: -bench). Larger and reasoning models improve reliability on average over smaller models. Notably though, reasoning models do not significantly improve in predictability on GAIA or in consistency on -bench.
Figure 10: Reliability by provider (top: GAIA, bottom: -bench).
Figure 11: Consistency results across agents and benchmarks. Consistency metrics reveal agents reliably choose similar actions but vary in execution order: a ”what but not when” gap highlighting limitations in current planning capabilities. Outcome consistency remains the most challenging metric across both benchmarks. Resource consistency shows noticeable variance in compute usage (mostly GAIA).
Figure 12: Predictability results across agents and benchmarks. Claude models generally show the strongest calibration performance (confidence matching actual success). Discrimination (ranking task difficulty) shows mixed results with recent models sometimes improving and sometimes regressing.
Figure 13: Reliability plots of different agent models on GAIA. Calibration plots show confidence (x-axis) versus actual accuracy (y-axis), where the diagonal represents perfect calibration. Agents are noticeably better calibrated compared to -bench. Anthropic models (Claude Opus 4.5 in particular) stand out as well calibrated. Notably, Opus 4.7 is the only model that is consistently under-confident.
Figure 14: Calibration plots of different agent models on -bench. All agents suffer from severe overconfidence. Only newer Claude models show modest calibration improvements.
Figure 15: Selective prediction curves of different agent models on GAIA. Accuracy-coverage curves show whether models can improve accuracy by abstaining on low-confidence predictions; the ideal curve rises steeply while the random baseline indicates confidence provides no signal. Most models demonstrate modest selective prediction ability, with Gemini 2.5 Flash, GPT-5.2, and Claude Opus 4.5 showing particularly strong performance.
Figure 16: Selective prediction curves of different agent models on -bench. Selective prediction largely fails in this constrained benchmarking setup: most models produce curves indistinguishable from the random baseline, indicating confidence scores carry no information about correctness. Only Claude Opus 4.5 and 4.7 attain modest selective prediction ability.
Figure 17: Robustness results across agents and benchmarks. Prompt robustness shows the largest amount of variation with environment and fault robustness plateauing for our chosen perturbations.
Figure 18: Safety results across agents on -bench. Extension of Figure 5 .
Figure 19: Comparison of reasoning vs non-reasoning models. We observe that reasoning models are generally more reliable than non-reasoning models, albeit reliability improves slower than accuracy.
Figure 20: Reliability metrics stratified by task difficulty on GAIA. Accuracy degrades as expected on harder tasks, while Claude and Gemini models invest significantly more actions on difficult problems. Outcome consistency shows mixed patterns driven by accuracy-dependent failure modes. Robustness metrics (aside from prompt robustness) remain relatively stable across difficulty levels, suggesting robustness is largely orthogonal to task complexity.