cs.AINov 14, 2026

A Hierarchical Statistical Framework for Evaluating Agent Reliability

Authors: Mirco Milletarì, Tuhin Kundu

Organizations: Microsoft

Abstract

Agents operating in complex scenarios often face uncertainty due to incomplete information about the environment they operate in, ambiguous user queries, complex business rules and the inherently stochastic dynamics of the governing policy. Several new benchmarks have been proposed to evaluate LLM-based agents on complex tasks in multi-tool environments, including settings with policy constraints and varying levels of intent ambiguity. These conditions make it important to evaluate agentic systems systematically and quantify their performance under different conditions to assess their reliability in production. This need is evident in the recently introduced Thinkingbox benchmark, where even the strongest models struggle to perform consistently across diverse business scenarios. This paper presents a statistical framework for evaluating uncertainty and reliability in agentic systems. We define a hierarchy of scenarios, instances, and samples to assess model performance and variability systematically. We apply this framework to common metrics found in the literature: mean pass rate, pass@kpass@k, and passhatkpasshat^k, with uncertainty estimates for each metric. Aggregating metrics at different levels of the hierarchy reveals different structures and properties of the policy model; we connect this hierarchical aggregation with the known process of coarse-graining as intended in the physics and statistical literature. We independently derive exact finite-sample expressions for the sampling variances of the unbiased pass@kpass@k and passhatkpasshatk estimators; related results also appear in recent submissions. Our central contribution is a constructive derivation of hierarchical uncertainty propagation, making explicit its correspondence with coarse-graining and separating execution noise from task heterogeneity.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

    Sep 30, 2026Michael Hardy, Ruhana Azam, Anka Reuel +2AI Agent EvaluationAgent Evaluation

  2. More Agents, More Reliable? System-Level Reliability of Multi-Agent LLMs under Operational-Profile Uncertainty

    Nov 4, 2026Burkay Tunçtürk, Koorosh Aslansefat, Yiannis Papadopoulos +1Multi-Agent LLM SystemsLLM Agent Reliability

  3. UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities

    May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent EvaluationAI Agent Benchmarks