Hierarchy-Aware Supervised Uncertainty Estimation for Black-box LLM Taxonomic Reasoning

Shuting Xie; Nathaniel Lesperance; Graham W. Taylor

cs.LGAug 24, 2026

Hierarchy-Aware Supervised Uncertainty Estimation for Black-box LLM Taxonomic Reasoning

Authors: Shuting Xie, Nathaniel Lesperance, Graham W. Taylor

Organizations: Vector Institute for AI, Toronto, ON · University of Guelph, Guelph, ON

Abstract

Large language models (LLMs) are increasingly used for scientific decision support, yet reliable confidence estimation remains difficult in black-box settings. We study uncertainty estimation for hierarchical taxonomic reasoning generated by a black-box LLM in a long-tailed biodiversity monitoring pipeline. Using proxy features extracted by an open-source tool LLM, we train lightweight supervised estimators with hierarchy-aware supervision to predict rank-wise correctness. Across three tool LLMs, the supervised estimators consistently outperform a token-likelihood baseline for micro discrimination and selective prediction under a single global rejection threshold, improving micro AUROC from 0.57 to 0.75--0.80. The best results are achieved by a rank-specific multi-head design (H3), suggesting that accounting for hierarchical output structure is important when a unified abstention rule is required. Our code is publicly available at https://github.com/uoguelph-mlrg/hierarchy-aware-llm-uq

Figures & tables

Figure 1: End-to-end pipeline for LLM uncertainty estimation. A target LLM generates taxonomic predictions from VLM captions and RAG-retrieved context. A tool LLM extracts proxy features from each prompt–response pair, which a lightweight uncertainty estimator maps to a confidence score. Blue boxes denote inputs and intermediate outputs. Red boxes denote prediction targets.

Tool LLM	Benchmarks	Ours
	Baseline	H1	H2	H3
Gemma 7B	0.573	0.710 $\pm$ 0.050	0.694 $\pm$ 0.060	0.796 $\pm$ 0.023
Qwen3 30B A3B Instruct		0.743 $\pm$ 0.050	0.719 $\pm$ 0.080	0.749 $\pm$ 0.017
GPT-OSS 20B		0.733 $\pm$ 0.069	0.783 $\pm$ 0.044	0.790 $\pm$ 0.028

Table 1: All-rank micro AUROC (mean

\pm

std over 5 seeds). Highest mean is bolded.

Panel of Figure 2 — Figure 2: Global-threshold risk–coverage curve. Lower is better. Curves show the mean over 5 random seeds, with shaded regions indicating $\pm 1$ standard deviation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Rank	Attempts	Coverage (%)	Accuracy	F1	HP	HR	HF
Class	950	100	0.966	0.973	0.981	0.981	0.981
Order	922	97	0.872	0.885	0.955	0.955	0.955
Family	745	78	0.693	0.710	0.914	0.914	0.914
Genus	223	23	0.453	0.532	0.865	0.865	0.865
Species	55	6	0.455	0.500	0.847	0.847	0.847

Appendix

Table 2: Rank coverage and prediction accuracy in the upstream system before filtering.

Tool LLM	Middle layer	Total Number of Layers	Hidden size
Gemma 7B ( google/gemma-7b )	14	28	3072
Qwen3 30B A3B Instruct ( Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 )	24	48	2048
GPT-OSS 20B ( openai/gpt-oss-20b )	12	24	2880

Appendix

Table 3: Details of tool LLMs. The first column reports the model name, followed by the exact checkpoint identifier in parentheses.

Figure 3: Feature tree demonstrates the feature types extracted by tool LLMs.

Tool LLM	Benchmarks	Ours
	Baseline	H1	H2	H3
Gemma 7B	0.621	0.451 $\pm$ 0.009	0.454 $\pm$ 0.013	0.498 $\pm$ 0.036
Qwen3 30B A3B Instruct		0.520 $\pm$ 0.015	0.514 $\pm$ 0.016	0.491 $\pm$ 0.021
GPT-OSS 20B		0.509 $\pm$ 0.029	0.528 $\pm$ 0.044	0.515 $\pm$ 0.031

Appendix

Table 4: All-rank aggregated macro AUROC across tool LLMs. Results are reported as mean ± std over 5 random seeds. Bold indicates the best macro AUROC across all methods, and underlining marks the best performance among our three uncertainty estimators.

Panel of Figure 4 — Figure 4: Per-rank AUROC (pre-calibration). Markers show the mean over 5 random seeds, and horizontal bars indicate $\pm 1$ standard deviation.

Tool LLM	Method	NLL		ECE		Brier
Tool LLM	Method	pre	cal	pre	cal	pre	cal
None	Baseline	1.030 (0.000)	0.706 $\downarrow$ (0.000)	0.114 (0.000)	0.299 $\uparrow$ (0.000)	0.408 (0.000)	0.256 $\downarrow$ (0.000)
Gemma 7B	H1	0.488 (0.055)	0.465 $\downarrow$ (0.039)	0.057 (0.018)	0.064 $\uparrow$ (0.021)	0.150 (0.013)	0.148 $\downarrow$ (0.011)
	H2	0.470 (0.040)	0.435 $\downarrow$ (0.026)	0.081 (0.023)	0.035 $\downarrow$ (0.009)	0.151 (0.015)	0.139 $\downarrow$ (0.011)
	H3	0.403 (0.017)	0.411 $\uparrow$ (0.018)	0.057 (0.020)	0.059 $\uparrow$ (0.006)	0.128 (0.008)	0.127 $\downarrow$ (0.006)
GPT-OSS 20B	H1	0.437 (0.048)	0.427 $\downarrow$ (0.038)	0.045 (0.024)	0.037 $\downarrow$ (0.019)	0.144 (0.020)	0.139 $\downarrow$ (0.015)

Appendix

Table 5: All-rank calibration metrics before vs. after temperature scaling ( 7 ) . Entries are mean (std across seeds), to three decimals. Baseline is model-invariant and shown once.

\downarrow

indicates improvement (lower is better), and

\uparrow

indicates degradation. Bold marks the best (minimum) within each tool-LLM block for each metric column.

Biodiversity Monitoring

See all 23 papers

Improved Confidence Estimates for Black-Box Large Language Models
Aug 19, 2026Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-GanemConfidence Estimation in Language Models Confidence Estimation
Layer 6 AI · TD Insurance
POOL: Propagated Uncertainty Over Lookalikes
Aug 24, 2026Rounak Sharma, Ananya B. Sai, Soumyabrata PalConfidence Estimation in Language Models LLM Uncertainty Estimation
Adobe Research, India
A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models
Jun 18, 2026Jiayi Wang, Xu-Yao ZhangLLM Evaluation Uncertainty Quantification
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Aug 19, 2026J/K move · Enter open · S save

Improved Confidence Estimates for Black-Box Large Language Models

Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-Ganem

Layer 6 AI · TD Insurance

Confidence Estimation in Language Models Confidence Estimation LLM Uncertainty Estimation Language Model Calibration

Open paper AskPDF arXiv

Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.

Abstract

Biodiversity Monitoring

Explore similar work

Improved Confidence Estimates for Black-Box Large Language Models

POOL: Propagated Uncertainty Over Lookalikes

A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models