A Hierarchical Statistical Framework for Evaluating Agent Reliability
Organizations: Microsoft
Abstract
Agents operating in complex scenarios often face uncertainty due to incomplete information about the environment they operate in, ambiguous user queries, complex business rules and the inherently stochastic dynamics of the governing policy. Several new benchmarks have been proposed to evaluate LLM-based agents on complex tasks in multi-tool environments, including settings with policy constraints and varying levels of intent ambiguity. These conditions make it important to evaluate agentic systems systematically and quantify their performance under different conditions to assess their reliability in production. This need is evident in the recently introduced Thinkingbox benchmark, where even the strongest models struggle to perform consistently across diverse business scenarios. This paper presents a statistical framework for evaluating uncertainty and reliability in agentic systems. We define a hierarchy of scenarios, instances, and samples to assess model performance and variability systematically. We apply this framework to common metrics found in the literature: mean pass rate, , and , with uncertainty estimates for each metric. Aggregating metrics at different levels of the hierarchy reveals different structures and properties of the policy model; we connect this hierarchical aggregation with the known process of coarse-graining as intended in the physics and statistical literature. We independently derive exact finite-sample expressions for the sampling variances of the unbiased and estimators; related results also appear in recent submissions. Our central contribution is a constructive derivation of hierarchical uncertainty propagation, making explicit its correspondence with coarse-graining and separating execution noise from task heterogeneity.
Figures & tables
| Metric | Main intuition |
|---|---|
| Pass@1 | Typical single-attempt performance |
| Pass@k | Can retries rescue the task? |
| Pass^k | Is success consistently reproducible? |
| \begin{array}[]{c@{\quad}c}\hfil\hskip 9.24994pt&\begin{array}[]{cccc}\rho_{1}&\rho_{2}&\cdots&\rho_{M}\end{array}\\[-2.0pt] \hat{X}=\hfil\hskip 9.24994pt&\begin{pmatrix}X_{11}&X_{12}&\cdots&X_{1M}\\ X_{21}&X_{22}&\cdots&X_{2M}\\ \vdots&\vdots&\ddots&\vdots\\ X_{N1}&X_{N2}&\cdots&X_{NM}\end{pmatrix}\end{array} |
| Metric | booking | auto insurance | consulting services | retail | banking support |
|---|---|---|---|---|---|
| number of instances | 104 | 100 | 101 | 98 | 104 |
| baseline mps ste | |||||
| candidate mps ste | |||||
| mean gap ste | |||||
| tests improved | 62 | 58 | 57 | 45 | 69 |
| tests degraded | 24 | 28 | 30 | 16 | 28 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
AI Agent Reliability
AI agent reliability studies whether tool-using systems act correctly, consistently, safely, and recoverably across changing tasks and environments.
See all 364 papers