The ability to rigorously estimate the failure rates of large language models (LLMs) is a prerequisite for their safe deployment. Currently, however, practitioners often face a tradeoff between expensive human gold standards and potentially severely-biased automatic annotation schemes such as "LLM-as-a-Judge" labeling. In this paper, we propose a new, practical, and efficient approach to LLM failure rate estimation based on constrained maximum-likelihood estimation (MLE). Our method integrates three distinct signal sources: (i) a small, high-quality human-labeled calibration set, (ii) a large corpus of LLM-judge annotations, and, most importantly, (iii) additional side information via domain-specific constraints derived from known bounds on judge performance statistics. We validate our approach through a comprehensive empirical study, benchmarking it against state-of-the-art baselines like Prediction-Powered Inference (PPI). Across diverse experimental regimes -- spanning varying judge accuracies, calibration set sizes, and LLM failure rates -- our constrained MLE consistently delivers more accurate and lower-variance estimates than existing methods. By moving beyond the "black-box" use of automated judges to a flexible framework, we provide a principled, interpretable, and scalable pathway towards LLM failure-rate certification.
Figures & tables
Figure 1: Our approach for estimating the true failure rate ( θ ) of a target LLM using noisy judge evaluations. The framework generates ground-truth labels SM via human experts and automated labels SJ via an LLM judge. These labels form a small, high-quality set ( DM ) and a much larger but noisy set ( DJ ), where nJ≫nM . The estimator is then applied to these combined sources to estimate the failure rate θ alongside the judge’s performance parameters. In practice, the proposed approach can also ingest partial prior knowledge about the judge’s reliability parameters derived from in-domain calibration datasets collected under the same target LLM and judge.
Figure 2: MSE of different estimators on synthetic data. Panels sweep the constraint width δ , the number of labeled samples nM , the judge TPR, and the judge FPR, respectively, while holding all other parameters fixed. The parameter δ controls the width of the constraint region, with smaller values imposing tighter constraints on the judge parameters (see details in Appendix F ). Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Figure 3: Difference in MSE between the CMLE and PPI++ under misspecified judge parameters. The x- and y-axes represent deviations of the assumed TPR and FPR from their true values, with nM=50 and nJ=10,000 . Colors indicate the relative MSE difference, where lighter colors correspond to smaller differences and darker red or blue indicate larger deviations. The boxed region highlights anchor (TPR,FPR) values for which the true parameters remain contained within the CMLE constraint.
Figure 4: Mean, variance, and MSE of different estimators on the Jigsaw dataset, with nM=50 and fixed nJ=10,000 . Qwen2.5-0.5B-Instruct is used as a classifier, and LLaMA-3.1-8B-Instruct serves as the judge (TPR = 0.939, FPR = 0.053). Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Figure 5: Mean, variance, and MSE of different estimators on the Jigsaw dataset, with nM=50 and fixed nJ=10,000 . Qwen2.5-0.5B-Instruct is used as classifiers, LLaMA-3.1-8B-Instruct serves as the judge (TPR = 0.939, FPR = 0.053), and CMLE constraints are centered at TPR/FPR estimates (TPR = 0.948, FPR = 0.063) obtained from the Hate Speech Offensive dataset using the same LLMs. Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Mean and variance of different estimators on synthetic data. Panels correspond to sweeping δ , nM , TPR , and FPR , respectively, while all other parameters are held fixed.
Figure 7: Difference in MSE between the CMLE and PPI++ under misspecified judge parameters. The x- and y-axes represent deviations of the assumed TPR and FPR from their true values, with nM=50 and nJ=10,000 . Colors indicate the relative MSE difference, where lighter colors correspond to smaller differences and darker red or blue indicate larger deviations. The boxed region highlights anchor (TPR,FPR) values for which the true parameters remain contained within the CMLE constraint.
Figure 8: Bias and variance of CMLE under misspecified TPR and FPR parameters. The x- and y-axes show deviations of the postulated TPR and FPR from their true values, nM=50 and nJ=10000 . Colors represent the bias and MSE value, with lighter colors indicating lower values and darker red or blue indicating larger values. The box indicates the set of postulated (TPR,FPR) that still contains the true values within the δ -constraint.
Figure 9: Mean and variance of different estimators on synthetic data. Panels correspond to sweeping δ , nM , TPR , and FPR , respectively, while all other parameters are held fixed. In particular, we additionally include both the Judge Estimator (Appendix F.2.2 ) and the PPI++ (Projected) Estimator (Appendix F.2.6 ) in the comparison, illustrating the effect of enforcing feasibility constraints via TPR and FPR priors and of direct estimation using only judge labels.
Figure 10: Mean, variance, and MSE of different estimators on the Jigsaw dataset, with nM=50 and fixed nJ=10,000 . LLaMA-3.2-1B-Instruct is used as a classifier, and LLaMA-3.1-8B-Instruct serves as the judge (TPR = 0.948, FPR = 0.063). Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Figure 11: Mean, variance, and MSE of different estimators on the Hate Speech Offensive dataset, with nM=50 and fixed nJ=10,000 . Qwen2.5-0.5B-Instruct and LLaMA-3.2-1B-Instruct serve as the target classifiers, while LLaMA-3.1-8B-Instruct acts as the judge. The corresponding judge performance parameters are (TPR,FPR)=(0.949,0.085) and (0.939,0.126) , respectively. Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Figure 12: Mean, variance, and MSE of different estimators on the Jigsaw dataset, with nM=50 and fixed nJ=10,000 . LLaMA-3.2-1B-Instruct acts as the classifier, and LLaMA-3.1-8B-Instruct acts as the judge. (TPR = 0.948, FPR = 0.063), and CMLE constraints are centered at TPR/FPR estimates (TPR = 0.939, FPR = 0.126) obtained from the Hate Speech Offensive dataset using the same LLMs. Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Figure 13: Mean, variance, and MSE of different estimators on the Hate Speech Offensive dataset, with nM=50 and fixed nJ=10,000 . Qwen2.5-0.5B-Instruct and LLaMA-3.2-1B-Instruct serve as the target classifiers, while LLaMA-3.1-8B-Instruct acts as the judge. The corresponding judge performance parameters are (TPR,FPR)=(0.949,0.085) and (0.939,0.126) , respectively. CMLE constraints are centered at TPR/FPR estimates (TPR = 0.939, FPR = 0.053) and (TPR = 0.948, FPR = 0.063), respectively obtained from the Jigsaw dataset using the same LLMs. Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Figure 14: Mean, variance, and MSE of different estimators on the SafeRLHF dataset, with nM=50 and fixed nJ=10,000 . LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct serve as the judge (TPR = 0.819, FPR = 0.032) and (TPR = 0.984, FPR = 0.411). Note that the curves for UMLE and PPI++ nearly coincide due to their similar performance.
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous evaluation of their reliability. In real-world deployment, however, achieving extremely high reliability (e.g., "five-nines" (99.999%) vs. "three-nines" (99.9%)) is fundamentally critical, as this gap results in an order-of-magnitude increase in failures, which is catastrophic in reliability-critical applications. Still, estimating such a rare failure probability with tight confidence bounds requires prohibitively large LLM inference sizes, making standard Monte Carlo evaluation infeasible under limited compute budgets. In this paper, we observe that LLM failures exhibit strong systematic patterns: across broad parameterized input spaces, a small subset of inputs disproportionately accounts for the majority of failures. Leveraging this observation, we propose to learn a sampling distribution concentrated on failure-prone inputs via the cross-entropy method (CEM). We evaluate our framework on three LLMs, Qwen2.5-Math-7B-Instruct, gpt-oss-20b-low, and Gemini 2.5 Flash Lite, across parameterized GSM8K templates and achieve up to 156.22x reduction in required inferences compared to naive uniform sampling. Our estimates reveal that models with indistinguishable accuracy on standard benchmarks can differ substantially in estimated failure rates, underscoring that reliability is a distinct and measurable axis of model quality. Our simple yet practical framework enables the evaluation of extreme reliability in LLMs, a distinct and underexplored dimension of evaluation beyond existing benchmarks, for their growing use in reliability-sensitive applications.
LLM-as-a-Judge evaluation has become a standard tool for assessing base model performance. However, characterizing performance via the naive estimator, i.e., raw judge outputs, is systematically biased. Recent work has proposed estimators to correct this bias, but their reliability depends critically on judge quality and, for model comparisons, on calibration stability. Sharing calibration across compared models is practically attractive but can introduce severe bias, including cases where the comparison estimate points in the wrong direction with high apparent confidence. We study these failure modes through analytical results, simulations over judge quality (J) and cross-model calibration instability (ΔJ), and a real-data MMLU-Pro case study with sign reversal. We propose J and ΔJ as diagnostics for when corrected estimates, especially shared-calibration comparisons, are likely unreliable, and provide reporting guidance for LaaJ evaluation.
Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists. However, objective tasks introduce a distinct reliability challenge for reference-free LLM judging. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation. Although the former enables efficient evaluation, the judge may hallucinate or lack sufficient evidence for its verdict. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and requires an appropriate mechanism to determine when and how that evidence should be used reliably. More importantly, neither approach alone provides formal control over the risk of accepted verdicts or guarantees their reliability at a specified level. We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level~α with high probability, using finite-sample Clopper--Pearson intervals. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval-augmented mode, where the judge gathers web evidence and re-evaluates the instance under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scales, the framework maintains the target error rate while achieving substantially higher coverage than single-mode baselines.
Sher Badshah, Ali Emami, Hassan Sajjad
Dalhousie University · New York University Abu Dhabi · Emory University