cs.LGNov 13, 2026

Black-Box Uncertainty Quantification for Large Language Models via Ensemble-of-Ensembles

Authors: Wang Ma, Debarun Bhattacharjya, Junkyu Lee, Nhan H Pham, Harsha Kokel, Qiang Ji

Organizations: Rensselaer Polytechnic Institute · Lirio · IBM Software Innovation Lab · Independent Researcher

Abstract

Reliable use of large language models (LLMs) requires uncertainty estimates that separate ambiguity intrinsic to a prompt from genuine knowledge gaps. Bayesian and deep-ensemble methods deliver this aleatoric/epistemic decomposition in principle, but are computationally prohibitive at LLM scale and require white-box access. Existing black-box methods, in contrast, summarize to a single scalar that cannot make the distinction. We introduce a two-level ensemble that bridges this gap. Stochastic decoding of a fixed prompt forms the inner ensemble; meaning-preserving semantic perturbations of the prompt form the outer ensemble. Variability is measured in a continuous embedding space, and the law of total covariance gives an exact within-/between-perturbation decomposition to estimate aleatoric and epistemic uncertainty (AU/EU); it requires only black-box sampling access. On top of this estimator, we contribute three theoretical results that ground the framework: (i) a finite-sample bias identity for nested Monte Carlo with a closed-form bias-corrected epistemic estimator; (ii) a Bayesian bridge bounding the gap between perturbation-population AU/EU and Bayesian targets; and (iii) a separation result identifying a class of failure modes that any consistency-only black-box method provably misses while perturbation-based EU detects. Across five short QA benchmarks and four instruction-tuned LLMs spanning two families and two scales, our estimator is competitive with strong white-box baselines, and at a matched generation budget it matches leading black-box methods on error ranking while decomposing AU and EU; controlled paired interventions show that AU and EU respond selectively to distinct, actionable sources of failure: prompt ambiguity and model knowledge gaps; and a long-form pilot on the NQ long-answer split shows the framework extends to claim-level factuality without retraining.

Figures & tables

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models

    Jun 18, 2026Jiayi Wang, Xu-Yao ZhangLLM EvaluationUncertainty Quantification

  2. Improved Confidence Estimates for Black-Box Large Language Models

    Aug 19, 2026Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-GanemConfidence Estimation in Language ModelsConfidence Estimation

  3. Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification

    Apr 18, 2026Kimia Hamidieh, Veronika Thost, Walter Gerych +2Selective PredictionConfidence Estimation in Language Models